Table of content

• What does “high-risk” mean for healthtech AI under the EU AI Act?
• How do the EU AI Act and MDR work together for healthtech products?
• Why does the AI Act make training data a regulatory artefact?
• What does meaningful human oversight mean under the EU AI Act?
• How can healthtech teams build AI Act compliance into the roadmap?
• What logging does the EU AI Act require for high-risk AI systems?
• Conclusion

For many companies, conversations about the EU AI Act quickly turn into long legal briefings that are dense, organisation-wide, and difficult to translate into actual product decisions. 

Those in tech roles often find themselves looking at the product roadmap with a sense of impending dread. Without practical context, it might feel like the next two quarters of feature development will turn into a product rebuild and an urgent compliance sprint.

However, the AI Act can prove to be a much lesser legal challenge than it’s framed, and be more of a product quality opportunity. 

As an organisation in the healthtech space, you are likely already halfway to the finish line by virtue of your existing standards. The companies that will struggle are those that will leave the EU AI act as a last-minute audit checkbox.

Here, we bridge the conversation on “what the AI Act compliance for healthtech requires” to  “how our systems must work”.

What does “high-risk” mean for healthtech AI under the EU AI Act?

In terms of the AI Act, “high-risk” is a technical classification, not a warning label. If your AI forms part of a medical device or supports its function, such as diagnostic software, clinical decision support, imaging analysis or triage tools, it almost certainly qualifies as high-risk under Article 6 and Annex III.

Yet, in healthtech, high-risk is often the default state. The new EU-wide legislation requires your AI system to meet a higher standard of proof. You must demonstrate that your algorithms are accurate, your training data is sound, and human intervention remains possible.

We recommend thinking of it like achieving ISO 27001. It raises the bar and validates a company’s professional standing, rather than acting as a wall that keeps it out of the market.

Key takeaway for tech leads:

Accept “High-Risk” as your primary architectural constraint rather than a mark of failure. Your strategic focus must shift from rapid prototyping to a “deliberate engineering” model where safety and evidence are baked into the initial sprint. Effectively managing this classification now ensures you avoid a roadmap-crippling technical debt when the 2027 deadlines arrive.

How do the EU AI Act and MDR work together for healthtech products?

The AI Act does not replace the Medical Device Regulation (MDR) or the In Vitro Diagnostic Regulation (IVDR). It is designed to sit alongside them and, in practice, to tighten the expectations around software-driven systems.

The EU is not creating a parallel regulatory universe. It is extending existing medical device oversight into areas that traditional frameworks didn’t fully cover, particularly around adaptive, data-driven systems.

Many of the companies we speak to at Holisticon Connect express their belief that this sounds like twice the work. In reality, however, it’s more of an extension of what they already possess.

Most healthtech companies already operate a Quality Management System based on ISO 13485. That system already covers risk management, validation, documentation, and post-market surveillance.

The AI Act builds on those mechanisms rather than replace them.

So instead of creating a second compliance track, teams extend the one they already have. Specifically:

• risk management files now include algorithmic risks (e.g. model drift, data bias)
• validation plans expand to cover model performance across subpopulations
• post-market surveillance incorporates continuous model monitoring
• technical documentation includes data lineage and model lifecycle details

The structure stays the same, but the content becomes more detailed. Guidance like MDCG 2025-6 makes this direction clear. The expectation is that Notified Bodies will increasingly assess MDR/IVDR and AI Act requirements together, as a combined evaluation of whether the system is safe, effective, and controlled across its lifecycle.

What new ground does the AI Act bring? 

There are a few things that MDR did differently from this latest legislation, because the former was not built for the capabilities of today’s technology. These are: 

• Explainability. Not just “what” the system outputs, but also sufficient context to understand ”why”.
• Bias management. This must be built into an organisation’s workflow as something continuously monitored and mitigated, not a one-off or irregular initiative.
• Continuous monitoring. Performance must be tracked in real-world conditions. Doing so in the pre-market phase will no longer suffice.
• User transparency. Clinicians need to have clarity on system limitations and confidence levels to carry out their work.

Key takeaway for tech leads:

Treat the AI Act and MDR as a unified development track rather than two separate silos. This requires evolving your existing QMS to include AI-specific controls, ensuring that a single audit can cover both clinical safety and algorithmic trust. Success here means your product remains eligible for CE marking without doubling your regulatory overhead.

Why does the AI Act make training data a regulatory artefact?

Article 10 of the AI Act exposes many healthtech teams to the greatest compliance risk. It mandates that datasets used to train, validate, and test high-risk AI must be “relevant, representative, and to the best extent possible, free of errors and complete.” Your training data now becomes a formal, auditable part of your technical file.

The term “representative” in this part of the Act sets an exceptionally high bar. It requires you to account for age, sex, ethnicity and the specific clinical environments where you plan to use your product. If you train a model predominantly on data from one demographic, the model will fail to meet the Act’s scrutiny.

For example, if a team trains and validates a skin cancer detection AI primarily on lighter skin tones, the product would not meet the Act’s compliance requirements.

Arguably, this strict criterion emerged out of necessity after countless cases of AI producing inaccurate and biased results. Just a few years ago, the medical community raised significant concerns after researchers discovered a massive geographic data imbalance.

Research showed that since many clinical AI researchers operated in US, prestigious coastal hubs, over 70% of the training data for many innovations (with potentially global reach) originated from just three US states – California, Massachusetts, and New York. This meant that patients not only beyond the country, but in rural areas or different clinical settings were effectively invisible to the algorithms.

Key takeaway for tech leads:

This a shift in how you value your data pipeline. Decisions made during early-stage model development are now either future assets or potential liabilities. Retrospectively cleaning or re-validating a biased dataset is exponentially more expensive than building rigorous governance into your pipeline today. To support this transition, you can refer to the European Health Data Space (EHDS). It’s emerging as a critical resource, designed to provide companies with access to the diverse, high-quality, and representative datasets required to meet these new standards.

What does meaningful human oversight mean under the EU AI Act?

The EU AI Act requires “meaningful” human oversight for high-risk systems under Article 14. In practice, this requirement is significantly more demanding than many teams assume. 

What meaningful Human-in-the-Loop (HITL) looks like in a medical context involves surfacing the AI’s confidence level alongside its recommendation and making it easy for a clinician to see why the system reached a specific conclusion.

If an UI simply presents a finding with a “confirm” button, a busy clinician may instinctively click through without critical evaluation. To meet the Act’s standards, your product must move toward a model of productive friction. This involves surfacing the AI’s confidence levels and “local explainability”, for example, highlighting the specific pixels in an image or the specific patient vitals that triggered a recommendation, so the doctor can validate the logic behind the output. 

Now, the question is – what happens if a clinician disagrees with an AI-powered finding?

For an Innovation Lead or CTO, this means treating the “override” not as a critical clinical data point. The system must be designed to log every instance where a human disagrees with the AI, documenting the clinician’s reasoning. 

Products that master this level of transparency will be compliant, but that’s secondary to the fact that they’ll also earn clinician trust and reduce institutional liability. It’s our belief that, as hospital procurement teams become more sophisticated in AI use, evidence of these oversight mechanisms will also become their prerequisite for purchase.

Key takeaway for tech leads:

You must lead your engineering and UX/UI teams to build systems and interfaces that provide clear explainability and confidence scores, allowing clinicians to make informed overrides. From a leadership perspective, this is about reducing your company’s liability by ensuring the AI remains a tool for doctors, not a replacement for human staff.

How can healthtech teams build AI Act compliance into the roadmap?

Arguably, the greatest long-term risk of ignoring EU AI Act regulation is finding oneself in the process of an audit only to realise the product cannot pass without a total rebuild. 

Requirements like immutable logging, bias testing, and explainability are architectural decisions. If they’re undervalued for years, they won’t be at the stage where the organisation will be able to “bolt” them on as an afterthought.

Compliance by Design means your sprint acceptance criteria must reflect these needs. Bias testing should be part of your standard validation, and risk classification should be considered the moment a new feature is scoped.

For the board, this approach is the ultimate risk reduction strategy. It ensures a faster time-to-market by eliminating last-minute rework and acts as a trust signal to the enterprise buyers you are courting.

Key takeaway for tech leads:

Introduce compliance into the daily CI/CD pipeline. You must integrate bias testing and risk classification directly into your definition of “Done” for every feature. This architectural foresight prevents the “compliance bottleneck” at launch, ensuring that your time-to-market is dictated by your team’s speed, not by regulatory rework.

If you’re finding yourself at a crossroads of understanding how to build compliant software that keeps up with your technological requirements, that’s something we’re able to advise you on at Holisticon.

What logging does the EU AI Act require for high-risk AI systems?

Technical documentation under the AI Act is extensive, ongoing, and strictly auditable. Companies must treat logging as a primary product feature, so that end users of the systems have the ability to trace their steps, overruns, or considerations in a retraceable manner.

For live systems, every inference should generate an immutable log containing the model version, the input data, the output, the confidence score, and the human action that followed. If a model issue or clinical error is discovered later, these logs are how you identify affected patients and demonstrate due diligence to regulators.

Important note: From August 2026, high-risk AI systems must be registered in a public EU database. Your product’s purpose, accuracy levels, and known limitations will be publicly visible. Robust logging will be the foundation of your post-market performance analytics and your future model improvement cycles.

Key takeaway for tech leads:

This requirement transforms your company’s backend telemetry into a strategic asset for reputation management. Because your system’s performance metrics will be publicly visible to competitors and procurement teams, your logging infrastructure must move beyond error catching to become a “flight recorder” for clinical validation. 

Conclusion

The AI Act raises the bar for everyone. While the transition requires significant engineering effort in a relatively short time frame, that work compounds into a lasting asset. Companies that incorporate the “trustworthy AI” approach into their code will be harder to unseat in clinical procurement and more resilient to the tightening product liability reality.

The transition is real work, but it is the work of building a market leader. At Holisticon Connect, we can help you build the bridge of what the law requires from your healthtech organisation and how your code works. If you’re looking for a partner who’ll ease your transition to the AI Act compliance and make it feel like a natural evolution of your product,  reach out. Let’s talk about your project.

Passion And Execution

Who We Are

At Holisticon Connect, our core values of Passion and Execution drive us toward a Promising Future. We are a hands-on tech company that places people at the centre of everything we do. Specializing in Custom Software Development, Cloud and Operations, Bespoke Data Visualisations, Engineering & Embedded services, we build trust through our promise to deliver and a no-drama approach. We are committed to delivering reliable and effective solutions, ensuring our clients can count on us to meet their needs with integrity and excellence.

more about Holisticon Connect