Building a Clinical Speech-Therapy App With a Real SLP: 4 Lessons From PhoenixSteps

How this started

My son has a speech sound disorder. Specifically, rotacismo — he struggles with /r/ and /rr/, which in Spanish are foundational phonemes that show up in roughly one in every six words. His speech-language pathologist is Stefania. We've been seeing her weekly for over a year and the progress has been real, but inconsistent: he'd nail a sound during a session and lose it by mid-week.

The gap was obvious to both of us. He'd do exercises with Stefa for forty minutes, then we'd go home and the exercises mostly stopped, because:

• The "drill at home" sheet Stefa sent had no feedback loop. My kid would say "ratón" five times and have no idea if any of them were correct.
• Existing pediatric speech-therapy apps in Spanish are either commercially mediocre (gamified versions of basic flashcards) or clinically rigid (built for adult speech rehab, not children).
• The market for tools that actually run the clinical exercises a Spanish-speaking SLP would prescribe — with audio feedback, automatic scoring, and progress tracking the therapist can read — basically did not exist for a private practice working with a 4-year-old.

I asked Stefa if she'd want to co-design something. She said yes. That's how PhoenixSteps started — and the four lessons below are the ones I wish I'd known going in.

Lesson 1: A clinical co-creator changes everything about what you ship

I had built consumer iOS apps before. I had not built a clinical tool. The thing I underestimated was how much of the actual product is the protocol, not the software.

Stefa works from named, published clinical protocols — Borrás, Bosch, the AELFA articulation drills. When she prescribes an exercise, she's pulling from a tradition that has decades of consensus on order, dosage, and progression. "Lengua a la nariz" isn't a cute idea — it's Borrás Exercise 29, with specific instructions about duration, repetitions per day, and what to do if the child can't sustain the position.

Before working with Stefa, I would have built a "speech therapy app" that was basically a glorified flashcard deck with cute animations. With Stefa, the exercise catalog became:

• Orofacial praxias — 7 exercises pulled directly from her clinical sheet, in the order she actually prescribes them.
• R-group syllable warmups — "ra ra ra," "rrrr-on" — building muscle memory before tackling words.
• R simple words — rosa, ratón, mira, perro — graded by Stefa for difficulty.
• R-cluster words (sinfones) — bra, cra, dra, fra, gra, pra, tra. The hard ones.
• Minimal pairs — R/RR, R/L, D/R, T/D. Auditory discrimination drills.
• Carrier phrases — embedding the target sound in real sentences.
• "Tren de la Risa" — a karaoke song Stefa wrote that hits every R context across 8 verses. The kid sings along; we score the R productions.

None of that comes out of an engineer's imagination. It comes out of a working SLP's notebook.

Lesson 2: Apple won't give you what your patient needs. Build it yourself.

This is the technical story, and it's the one I'm most proud of.

One of the most prescribed praxias for kids working on /r/ is "lengua a la nariz" — extending the tongue tip toward the nose. The exercise builds the lingual elevation needed for the alveolar trill. Stefa wants the app to automatically verify the kid did the exercise correctly: tongue out, pointed up, sustained for 10 seconds. No therapist in the loop, no parent grading by eye.

This sounds like a job for ARKit. Apple has had face tracking with the TrueDepth camera since the iPhone X. ARFaceAnchor.blendShapes includes things like jawOpen, mouthSmileLeft, cheekPuff — and yes, tongueOut. So I plug in tongueOut and watch the value spike when the kid sticks out his tongue. Great.

Except: tongueOut is a scalar. It's 0 when the tongue is in, and 1 when it's out. Apple does not tell you where the tongue is pointing. Up, down, left, right — they all read identical. For the alveolar-trill praxia, "lengua a la nariz," I need direction. Apple refuses to model it.

I read the documentation three times to make sure. Then I read every WWDC session on Vision and ARKit going back to 2018. Then I emailed Apple developer support. The answer was: no, the tongue is not modeled as 3D geometry, and there's no API to detect tongue direction. The reason given was that tongue tracking is inherently unstable because the tongue gets occluded by teeth and lips depending on position, and Apple chose not to ship something they couldn't validate at the precision Face ID requires.

Stefa and I had a choice: drop the exercise, or build the detector ourselves. We chose the second.

The pipeline we ended up with

The detector uses standard iOS frameworks layered into a custom pipeline. Nothing about it requires private APIs or jailbreaks — it's all stuff Apple gives you, just in a configuration nobody outside of clinical applications would bother with.

1. ARKit captures the camera frame on the TrueDepth camera at 60 fps. ARKit already has the face anchor and the blendshape values. tongueOut tells us the tongue is somewhere outside the mouth, but not where.
2. We grab the raw frame.capturedImage — the YUV pixel buffer from the camera, that ARKit hands you for free in every didUpdate callback.
3. Vision detects face landmarks on that pixel buffer: VNDetectFaceLandmarksRequest gives us back outerLips, innerLips, and nose as 2D polygons in normalized image coordinates.
4. We define three Regions of Interest (ROIs) outside the lip polygon:
   • UP ROI — the rectangle between the top of the upper lip and the bottom of the nose.
   • LEFT ROI — extending leftward (in pixel space) from the left corner of the lips.
   • RIGHT ROI — same, mirrored to the right.
5. We count pink/red pixels inside each ROI. The lip-skin transition is at Cr ≈ 18; the tongue itself is at Cr ≈ 25-50. We threshold Cr > 25 to filter out facial skin and pale lips, then count pixels that pass both luminance (Y > 0.20) and chroma thresholds.
6. If a ROI has > 400 "tongue-colored" pixels, the tongue is projecting in that direction. Cross-check with ARKit's tongueOut blendshape to confirm the tongue is actually out. Mirror-compensate for the front-facing camera (pixel-left = child's right).

The detector reports one of up, down, left, right, center, or notVisible at 20Hz with a confidence score. The first time I showed Stefa the demo — me sticking my tongue toward my nose and watching the screen say "ARRIBA conf 100% pix 3,974" — she didn't believe it was real until I sent her the source code.

Lesson 3: Audio quality is a feature, not a detail

PhoenixSteps ships with about 325 pre-recorded voice prompts, all generated using OpenAI's gpt-4o-mini-tts with the "nova" voice. These cover every word, syllable, phrase, and instruction the kid hears during a drill. Why pre-recorded TTS instead of letting iOS synthesize on the fly?

• Pediatric voice consistency. Kids learn faster when the audio prompt sounds the same every time. iOS's stock voice synthesizer drifts in pitch and pacing between sessions. Pre-recorded with a single voice/instructions prompt locks the audio to one consistent timbre.
• Speed and articulation. Stefa wanted slower-than-normal pronunciation for warmup drills, regular pace for practice, and a specific cadence for the "Tren de la Risa" song. Generating these with TTS instructions ("habla en español neutro latinoamericano, ritmo lento y articulado, énfasis infantil sin caricaturizar") gets us the exact register a real SLP would use.
• Reliability. Pre-recorded audio works offline, doesn't depend on a phone's TTS pipeline being up, doesn't get interrupted by Siri.

We did learn the hard way that the OpenAI API will occasionally return a truncated mp3 (we caught three files at 0.36s when they should have been 1.2s — the API didn't error, just silently returned a fragment). The fix was a post-generation validation step: every newly generated mp3 has to pass a minimum-duration check before it lands in the bundle. Boring infrastructure, but the difference between "the kid hears 'ratón'" and "the kid hears 'ra...' and gives up" is exactly this validation.

Lesson 4: HIPAA-equivalent privacy isn't optional, it's the foundation

The users of PhoenixSteps are children. Their voice recordings and progress data are protected health information under HIPAA, GDPR, the Uruguayan personal data law, and basically every jurisdiction's pediatric privacy regulation.

This shaped the architecture from day one:

• Speech recognition is on-device. WhisperKit, the same library we use in ClearCaps. The kid's voice never leaves the iPhone.
• Face tracking is on-device. ARKit + Vision, everything stays on the chip.
• Progress data is in SwiftData, syncing to the family's private iCloud — Apple's iCloud syncing uses end-to-end encryption for protected categories. The therapist sees data through an export the parent shares, not through a server we operate.
• No analytics, no third-party SDKs, no Crashlytics, no Facebook Pixel. The app's privacy manifest is empty by design.
• AI features are gated by parental consent. An on-device LLM (Apple Foundation Models) can summarize progress for the therapist, but it only runs after the parent affirmatively opts in.

The architectural payoff: PhoenixSteps will never have a data breach involving children's voice samples, because there's no centralized data to breach. That's the only acceptable design for this category.

Where PhoenixSteps is right now

Honest snapshot:

• Not in the App Store yet. Build 28 as of writing. We're finishing the clinical pilot with Stefa and a small group of her patients (including my son) before submitting.
• Spanish-first. The catalog of exercises, audio prompts, and instructional voiceovers is built for Spanish-speaking children. Localization to English is on the roadmap once the clinical content is validated by an English-speaking SLP.
• Free for parents, with an optional Pro tier for clinicians. The basic drill catalog will be free; the clinician dashboard (progress reports, custom exercise prescriptions, multi-patient management) is the paid tier. Pricing TBD pending the pilot.
• Stefa drives the clinical content. The exercise catalog, the protocols, the cadence, the success criteria — all hers. The app exists because she said yes to co-designing it.

What I want the rest of the SLP world to know: this app exists because one therapist agreed to co-design it. If you're an SLP working with pediatric patients in Spanish and you'd want a tool tuned to the way you work, please write. We're going to add more clinical advisors as the product matures, and the goal isn't to ship a generic app — it's to ship something that respects the practice as it actually is.

— Bruno Aiub, founder, AstroLexis LLC. Clinical content by Stefania, SLP. contact@astrolexis.space.