Author: Dr Louise Metcalf, Creator and Co-Founder, Gheorg
Every major AI safety framework in existence today was built with an adult in mind. That’s true of the guidelines coming out of the big labs, the emerging regulatory standards, even most of the academic literature on AI ethics. They cover misinformation, bias, privacy, inappropriate content. All of it matters, but none of it was designed for a six-year-old.
That gap is the reason I started gheorg, and it’s the problem I want to talk about here, because I think it’s about to become one of the most consequential blind spots in children’s technology.
The scale of the problem
Anxiety and depression in children are not rare or fringe conditions. Global estimates put childhood anxiety around 6.5% and depression around 2.6% (Polanczyk et al., 2015), with national data from Australia and the US in a similar range. On top of clinically diagnosed conditions, self-reported surveys from Save the Children and UNICEF during the pandemic found that as many as half of children reported experiencing significant emotional distress symptoms at the time, a figure that speaks to a broader mental health strain even where it doesn’t meet the threshold of a diagnosed disorder, and none of these figures are yet to go down.
Whatever measure you use, the workforce to meet this need doesn’t exist. Waitlists for child psychology services commonly run 12 to 18 months in Australia, the UK, and the US. Families in rural and regional areas often drive hours for a single appointment. Teachers absorb the burden in classrooms without training or tools. Technology is the only lever that can close a gap of this size, but only if it’s built the right way.
Why “adult AI, made cuter” doesn’t work
Most digital mental health tools aimed at children are adult products with a friendlier skin. A meditation app built for stressed professionals doesn’t become a children’s clinical tool because you add a cartoon character. The problem isn’t aesthetic, it’s structural. Children process language, emotion, and social cues differently at different developmental stages. A child disclosing fear or distress to an AI system needs a fundamentally different response architecture than an adult doing the same thing. And children are more susceptible to influence from authoritative-sounding sources, including AI characters, which creates real risk of unhealthy dependency or emotional attachment if the system isn’t explicitly designed to guard against it.
This is the thinking behind VERA-MH-P (Validated Ethical Responsive AI for Mental Health, Paediatric), the framework we’ve built at gheorg specifically to govern how AI interacts with children aged 4 to 12 in a mental health context. The VERA-MH-P is built around seven principles: developmental appropriateness, clinical safety governance, emotional safety, safeguarding and crisis response, transparency, dependency prevention, and data ethics.
The dependency prevention principle is one I think about the most. Our engagement data shows high daily usage and very low monthly churn, and the honest question to ask about numbers like that is whether they reflect genuine skill-building or a manipulative engagement loop. The design answer has to be structural: interaction patterns that actively strengthen a child’s real-world relationships and support networks, rather than substituting for them. That’s a very different design goal from maximising time on screen, and it needs to be measured and governed, not just claimed in a pitch deck.
Why this matters beyond one company
I don’t think the answer to this problem is one company doing it well. I think it’s an industry accepting that children’s mental health AI needs its own standard, the same way paediatric medicine has its own clinical trial requirements distinct from adult medicine. Regulators are starting to move in this direction. The EU AI Act classifies AI systems used in mental health contexts as high-risk, requiring documented safety frameworks and ongoing monitoring. School districts, health systems, and governments are increasingly unwilling to procure AI tools for children without evidence of a credible safety standard behind them. That’s a good thing. It raises the floor for everyone.
We’ve been fortunate to have this work engage well with serious institutional partners and our current work through that engagement is helping validate the clinical rigour behind our approach as we build toward a formal partnership. We’re also working through Master Services Agreement negotiations with a major US health system, and our first population-level outcomes dataset, from a school district deployment, is due to publish this year. None of that is a finish line. It’s evidence that the standard we set out to build is holding up under real scrutiny, which is the only kind of validation that actually counts in paediatric health.
What I’d ask other builders to take from this
If you’re building anything that puts AI in front of children, especially in a mental health or wellbeing context, the question worth sitting with isn’t “how do we keep this safe enough.” It’s “what would a safety framework look like if it were designed for a child’s mind from the ground up, instead of retrofitted from an adult one.” That’s a harder design problem than most teams want to take on. It’s also the only one worth solving.
Dr Louise Metcalf is Creator and Co-Founder of gheorg, an AI-powered mental health platform for children aged 4 to 12, deployed through schools, health systems, and governments in 78 countries.
