When Shadow AI Becomes Shadow High-Risk Under the EU AI Act

When the Nieuwsuur team looked into how Dutch organizations are coping with AI, they asked me to weigh in. The example that anchored their report is hard to forget: in the space of a single month, employees at the municipality of Eindhoven uploaded more than a thousand documents containing personal data to external AI tools. Not test files. BSN numbers, social-care records under the Wmo, information about addiction sensitivity, financially sensitive data. Nobody set out to cause harm. People were just trying to get their work done.

What I told Nieuwsuur was deliberately plain: "Unseen can flag to you, for example, when an employee puts confidential data into an AI system." That is the immediate problem, and it is worth solving. But sitting with the report afterwards, I kept coming back to the part we did not have time to get into on camera. Some of this usage is not only a data-protection problem. It quietly drags the organization into the most tightly regulated corner of the EU AI Act, and most of the organizations it applies to have no idea it is happening.

It is not a hypothetical

The Eindhoven numbers are not an outlier, they are just unusually well measured. Most organizations have never counted, which is its own answer. The same dynamic has played out at far larger companies.

Two of the experts in the piece framed the mechanics well. Remco van der Schoot, an AI researcher at Hogeschool Utrecht, put the data point simply: if we put data into an AI model, that data can be used to train the model. And ethical hacker Jan van der Put located the real risk where I think it belongs, with "employees who want to work with AI but do not know how to do it safely." The instinct to respond with an outright ban is understandable, and in my experience it mostly just moves the activity somewhere you can no longer see it.

Most AI use is not regulated. Some of it suddenly is.

This is the distinction I wish more compliance conversations started with. The AI Act's record-keeping rule, Article 12, does not apply to all AI use. It applies to high-risk AI systems, defined in Annex III of the Act. An employee using a chatbot to tidy up an email or debug a function is low risk, and Article 12 simply does not enter into it. (The GDPR still applies to any personal data, but that is a separate conversation.)

The catch is in how "high-risk" is decided. It depends on what the system is used for, not on which product it is. A general-purpose chatbot is low risk on Monday and high-risk on Tuesday if, in between, someone starts using it to filter job applicants, assess someone's creditworthiness, or support a decision about access to an essential service. The tool did not change. The legal status did.

The shadow high-risk problem

Put those two facts together and you get the scenario that keeps me up at night. An HR team pastes a stack of CVs into a chatbot and asks it to rank them. A back-office colleague uses an AI assistant to sanity-check a lending decision. Each of those is an Annex III high-risk use, carried out in an unsanctioned tool, with no oversight and no record.

At that moment the organization has, without deciding to, become the deployer of a high-risk AI system. Under Article 26 of the Act, deployers carry real obligations, including the duty to keep the logs that system produces, to the extent they are under the deployer's control, for at least six months. The organization now owes an account of a deployment it never knew it had.

This is the gap I do not see either side of the market closing. A discovery tool can tell you an employee is using a chatbot, but it does not produce a record you could defend to a regulator. A governance platform wired into your approved applications keeps tidy logs, but is blind to the unsanctioned tool where the high-risk task is actually happening. You can look compliant on paper and be exposed in practice.

What the Act actually asks of you

For a high-risk deployment, "we have some logs somewhere" is not the bar. Reading Articles 12 and 26 together, three things matter:

1. The system has to automatically record events across its lifetime (Article 12), not be reconstructed from memory after something goes wrong.
2. As the deployer, you have to retain those logs for at least six months, to the extent they are under your control (Article 26(6)).
3. The logs have to be good enough to actually account for what the system did, supporting oversight and post-market monitoring rather than just existing.

There is a fourth point the text does not spell out but every auditor will arrive at: a record you could have quietly altered last week is a claim, not evidence. Accountability is the whole purpose of the requirement, and it collapses if the log can be edited without trace. Failing to produce compliant records here falls into a penalty tier that reaches €15 million or 3% of worldwide annual turnover, whichever is higher. For most organizations the discovery moment will not be a calm internal review. It will be an audit, or an incident.

Where I think this lands

You cannot keep logs of something you cannot see, and you cannot stand behind a log you could have changed. To me that means the honest answer has to live at the point the traffic already passes through, the gateway between your people and the AI they are using. That is where you can notice an unsanctioned tool being pointed at a high-risk task, record the interaction into a tamper-evident trail that stays inside your own perimeter, and give the employee a safe, sanctioned route so the work happens somewhere you can actually govern.

None of this is an argument for banning AI, and I said as much to Nieuwsuur. The organizations that come through the August deadline in good shape will not be the ones that said no. They will be the ones that can sit across from an auditor, or a journalist, and answer one question without flinching: can you show what your AI actually did?

This piece draws on the Nieuwsuur report "Waarschuwing voor personeel dat met AI aan de slag gaat: ‘Kans op datalek groot’", in which I was interviewed. You can watch and read the original (in Dutch) at NOS.