Reviewing contracts with AI: what the tools can and cannot do

One contract, five minutes, 25 cents

Ten procurement contracts, the same review questions, three types of reviewer: experienced lawyers set the correct answers, then junior lawyers, an outsourcing provider and several language models competed against each other. The result of this 2024 study by software provider Onit still causes unease today. When it came to spotting legal issues, the best model was on a par with the humans. It took just under five minutes per contract instead of almost an hour, and cost 25 cents instead of a good 74 dollars.

Anyone reading only those figures might think contract review is a solved problem. A second look at the same study, and at more recent research, paints a different picture: the tools find clauses quickly and cheaply. What a clause means for the client is beyond them. Review remains lawyers' work; it simply starts at a different point.

What AI contract review delivers

The strength of these models lies in reading large volumes of text. An 80-page contract costs a language model seconds and a lawyer an afternoon. The tools find notice periods, liability caps, non-compete clauses or jurisdiction clauses and compile them into an overview. They compare a draft against a template and flag deviations. They summarise what a contract provides for and answer questions about it.

That is precisely what the Onit study measures: is there a problem in the contract in relation to a particular question, yes or no? On that task, the best model matched the hit rate of the junior lawyers. For a small law firm this means that the first pass, the sifting and sorting, can largely be handed over. The lawyer starts with the second pass.

Where the tools miss clauses

The first weakness showed up in that same study. As soon as the models had to do more than report that a problem existed and instead identify the relevant passage in the contract, they fell behind the outsourcing provider. The authors describe the models as cautious: they preferred to overlook existing problems rather than report ones that were not there.

A 2025 benchmark called ContractEval tested 19 current models against 102 commercial contracts and 41 clause types. Even the best models only reached a score of around 0.64 on a scale up to 1. The models were weak precisely on rare and expensive clauses, such as unlimited liability or joint intellectual property. Some smaller models answered "no matching clause" in 30 per cent of cases even though the contract contained one. That is the most dangerous kind of error: a report with something missing looks just as clean as a complete one.

On top of that, both pieces of research used English-language contracts governed by American or New Zealand law. They say nothing about how well the models handle German standard terms control or formal requirements.

Invented law remains a risk

The second limit concerns everything that goes beyond reading. Whether a clause is valid depends on statute and case law. Here the tools fall back on their training or on connected databases, and both can mislead. Researchers at Stanford University tested the legal research assistants of LexisNexis and Thomson Reuters in 2024. The systems delivered wrong or wrongly supported answers in 17 to 34 per cent of queries. General-purpose chatbots were between 58 and 82 per cent on legal questions.

The researchers distinguish two types of error: either the statement about the law is wrong, or it is correct but the cited source does not support it. The second type is more treacherous because it looks authoritative. If you have to verify every assertion and every citation, you lose a good part of the promised time saving again. For contract review, this leads to a clear division of labour. The tool shows what the contract says. Whether that is permissible, customary or dangerous is for the lawyer to decide, on the basis of their own knowledge.

What professional rules require

At the end of 2024 the German Federal Bar (BRAK) published guidance on the use of AI in law firms. It confirms what the studies suggest: AI output must be checked, because the systems can invent or distort content. Anyone who adopts a result unchecked breaches their duty of care and is liable for the loss. There is also the duty of confidentiality. A contract uploaded into a tool will almost always contain client secrets. Where the provider is based, what it does with the data and whether it uses it for training must be clarified before the first upload.

The EU AI Act comes on top of that. When using such tools, law firms count as deployers and, since February 2025, must ensure that their staff have sufficient AI literacy. For a firm with three lawyers and two legal assistants, that does not mean a training department, but it does mean a documented briefing: what the tool can do, what it cannot do and what must never be uploaded into it.