Who reviews the code the AI writes?

AI-generated code under security testing

A language model writes a working function in seconds. Whether it is also secure is another matter entirely. In 2025, the security provider Veracode tested more than 100 language models with 80 programming tasks in Java, Python, C# and JavaScript. In 45 per cent of cases, the code produced contained a vulnerability from the OWASP Top 10, the best-known list of typical security flaws in web applications. For Java, the failure rate was 72 per cent. Anyone who adopts AI-generated code without checking it is therefore quite likely to adopt a vulnerability along with it. The question of who checks this code consequently belongs on the management's agenda.

The models performed particularly badly on two classics. With cross-site scripting, the injection of foreign scripts into a web page, they failed in 86 per cent of the relevant cases, and with manipulable log entries in 88 per cent. Another finding is sobering: newer and larger models did write functionally better code, but not more secure code. The security rate remained flat across the model generations.

The German Federal Office for Information Security (BSI) cites similar figures in its recommendations. A study by New York University found security flaws in around 40 per cent of the programmes generated by an assistant. A study involving 16 models showed that just under 20 per cent of the software packages suggested did not exist at all. Attackers can register such invented package names and fill them with malicious code. Anyone who then follows the recommendation invites the attacker in.

The reviewer trusts too much

The vulnerabilities would be only half as bad if someone found them reliably. That is precisely where things fall short. Researchers at Stanford University had 47 participants solve security-relevant programming tasks, some with an AI assistant, some without. The group with an assistant wrote insecure code and at the same time considered it secure more often than the comparison group. The BSI calls this phenomenon automation bias: the code reads fluently and convincingly, so it seems correct.

Developers themselves are more sceptical than one might think. In the 2025 Stack Overflow survey, with around 49,000 responses, 46 per cent distrusted the accuracy of AI tools and only 3 per cent trusted it strongly. Two thirds were annoyed by solutions that are "almost right". That is the most dangerous kind of error: code that looks finished and is not. In a survey of technology leaders by the software provider New Relic, 62 per cent said at the same time that they put AI code into operation without reviewing it line by line. 82 per cent had experienced at least one outage within six months that was traceable to such code.

More code, more copies, more disruptions

Security is the hardest test, but not the only one. The analytics company GitClear examined 211 million changed lines of code from the years 2020 to 2024. The share of copied code blocks rose from 8.3 to 12.3 per cent. The share of tidying-up changes, meaning the restructuring of existing code without any new functionality, fell from 25 to under 10 per cent. In 2024, for the first time, more code was copied than moved. Copied code runs, but it has to be maintained in several places, and sooner or later someone forgets one of them.

The 2025 DORA report, for which Google surveyed almost 5,000 technology professionals, shows the consequence in operation. 90 per cent use AI at work, and output is rising. The stability of the software delivered, however, continues to correlate negatively with AI use. The authors describe AI as an amplifier: teams with good automated tests and fast feedback loops get better, while teams without such controls produce disruptions faster.

What the BSI and OWASP recommend

In 2024, the BSI published recommendations for AI coding assistants together with the French authority ANSSI. The first key recommendation is, in essence: AI assistants do not replace experienced developers, and their unchecked use can have serious security consequences. Generated code should as a matter of principle be reviewed and understood by the developer. Anyone producing code faster must expand quality assurance to the same degree. The authorities also advise marking AI-generated code blocks and documenting the tools used, for instance for later security audits. And they warn of a creeping effect: anyone who relies too heavily on the assistant unlearns how to assess someone else's code at all.

OWASP, the global organisation for web application security, is more specific in its cheat sheet on secure coding with AI. Every AI-generated change gets a human owner who answers for correctness, security and maintenance. Nobody approves a change just because its description sounds good. And tests written by the same AI prove little: OWASP describes the case of an AI agent making tests pass by weakening or deleting them. Tests for authentication, access rights and encryption should therefore be written by humans.

Responsibility stays with the company

Who bears responsibility internally can therefore be settled by rules. Externally, the question does not even arise, because the law grants no AI discount. The EU's Cyber Resilience Act imposes obligations on manufacturers of products with digital elements, and according to the BSI this includes pure software products such as accounting programmes or apps. These manufacturers must design their products to be secure from the outset, remedy vulnerabilities throughout the entire lifecycle and report actively exploited flaws. The reporting obligation has applied since 11 September 2026; from 11 December 2027 all requirements apply.

On top of this comes product liability. The new EU directive expressly treats software as a product, irrespective of how it is supplied or used. Germany must transpose it by 9 December 2026; the government draft has been available since December 2025. Anyone who places defective software on the market will then be liable for damage without anyone having to prove fault. Whether the defective line comes from an employee, a service provider or a language model makes no difference. The BSI holds the model providers responsible only for attacks on the model itself, for example via manipulated training data. For the individual code suggestion, responsibility rests with whoever adopts it.