In June 2023, a New York federal judge sanctioned two attorneys after they filed a brief citing six court decisions that did not exist. ChatGPT had invented them, complete with plausible names, docket numbers, and quotations. The lawyer using the tool had even asked the chatbot whether the cases were real. It said yes.
If trained lawyers can be fooled that badly, an ordinary consumer pasting a lease or a service agreement into a chatbot is running the same risk without the training to catch the miss. AI still has a place in reading a contract. The useful question is where a probability machine belongs and where you need a hard rule instead.
Two Different Machines Are Being Sold as One
A consumer-grade chatbot and a specialized legal AI look similar from the outside. Both take your document. Both spit back a fluent summary. Both sound sure of themselves. Underneath, they are doing very different work.
A general large language model predicts the next word based on patterns in its training data. It is a probability engine dressed up as an assistant.
A purpose-built contract system layers hard rules on top of that prediction. Checks that must pass, clauses that must be present, thresholds that cannot be crossed. The model's guess gets filtered before it reaches you. One tool is fast and confident. The other is slower, narrower, and designed to fail loudly rather than silently.
Fluent Answers Are Not the Same as Correct Ones
Fluency is the trap. A language model can summarize your non-compete in prose that reads beautifully and misstate the enforceable term by a year, and you will not hear a hitch in the sentence to warn you.
The scale of the problem is documented. A Stanford study of popular general-purpose LLMs found they invent false legal content on a majority of legal questions, and, worse for a consumer, the models tend to be confident about the wrong answers. A tool built specifically for contract review can still be wrong. It is engineered to know what it doesn't know and to escalate rather than improvise.
Each Approach Wins in Different Places
Neither tool is useless. They win in different places, and mixing that up is where consumers get hurt.
Confidentiality Is the Problem Nobody Reads About
Accuracy is not the only thing to weigh. When you paste a contract into a free chatbot, you are handing that document (names, dollar amounts, terms the other side considers confidential) to a third party whose data practices you probably haven't read.
This is the same reading problem the contracts themselves have.
The chatbot's own terms fall into that same unread pile. Purpose-built legal AI is generally deployed under agreements that keep your documents out of training data and inside a controlled environment. For anything sensitive, that difference matters more than the quality of the summary.
Use AI Without Getting Burned
You do not have to swear off chatbots to stay safe. Match the tool to the task and keep a few habits in place before you paste anything in.
- Strip the sensitive details. Redact names, addresses, dollar figures, and anything that identifies the other party before a general chatbot ever sees the document.
- Use it to translate, not to decide. Ask a general model to define jargon or restate a clause in plain English. Do not ask it whether a clause is enforceable or fair.
- Verify every specific it gives you. If the summary names a number, a deadline, or a statute, open the contract or the source and confirm it word for word.
- Match the tool to the stakes. A quick read of a gym waiver is a different job from reviewing a lease or a severance agreement. The second one deserves a purpose-built tool or a human lawyer.
- Read the chatbot's own terms. Know whether your inputs are used for training and whether the vendor retains them, before you hand over anything you would not post publicly.
The Architecture Question Sits Behind All of This
The deeper story, and the one professionals are having right now, is about how contract AI should be built in the first place. A model that predicts is not the same as a model that enforces. Pretending the first can do the second is how you end up with fabricated cases in a federal filing.
For readers who want to hear that argument made in full, the the Law.co discussion of why Contract AI Needs Hard Rules, Not Just Smart Guesses discussion of why Contract AI Needs Hard Rules, Not Just Smart Guesses walks through the design choices that separate a fluent guess from a reliable answer. The short version for consumers: when the stakes are real, ask what rules the tool is enforcing, not how smart it sounds.
