Legal
AI-generated responses can be wrong
The agents in this Service are built on large language models. These models generate each response by predicting likely text based on patterns learned during training - they don't look up facts in a database and confirm them before answering. Most of the time this produces accurate, useful answers. Sometimes it produces a fluent, confident answer that is simply wrong: a fabricated detail, a misattributed fact, or a plausible sounding conclusion drawn from an incomplete picture. This is usually called "hallucination," and no model available today is immune to it.
What to do about it
Treat an agent's response as a well-informed first draft, not a verified conclusion. Before acting on anything that matters - a compliance judgment, a number that feeds a filing, a claim about a specific person or company - check it against the underlying source (the "Sources" panel in a conversation shows what the agent drew on) or another reliable reference. This matters more, not less, in the kind of due-diligence and compliance work this Service is often used for, where a wrong conclusion can have real consequences.
This is not a theoretical risk
Courts are actively deciding cases about exactly this failure mode - an AI system stating something false about a real person or business with enough confidence that someone relied on it. A few examples:
A fabricated regulatory claim about a real company
A Minnesota solar installer, Wolf River Electric, sued Google after its AI Overview search feature stated that the state's Attorney General had sued the company for deceptive practices - a claim the company says was fabricated and unsupported by any of the sources the AI feature cited. Customers reportedly cancelled contracts after seeing it. The company is seeking over $100 million in damages; the case was still pending as of early 2026.
A German court held an AI feature directly liable
In 2026, the Regional Court of Munich ruled against Google after its AI Overview feature wrongly associated two publishers with scams and other misconduct. The court's reasoning is the notable part: it treated the AI-generated summary as Google's own speech, not a neutral link to someone else's content, because the feature synthesizes a new answer rather than just pointing to sources - so the usual legal protections for search results didn't apply. Google was ordered to stop repeating the claims.
A company held responsible for its own chatbot's mistake
In 2024, Air Canada's support chatbot told a customer he could apply for a bereavement fare discount after the fact, contradicting the airline's actual policy. When the airline refused to honor it, a Canadian tribunal ruled that Air Canada was bound by what its own chatbot told the customer, regardless of whether the chatbot's answer was accurate. The airline had to pay.