How AI form tracking works
Published 30 September 2026 by TRACK AI Coach
TRACK AI uses nothing but your phone’s camera: no wearables, no sensors, no special equipment. Here is what happens between you starting a set and the coach telling you to sit deeper.
1. Finding your body
Each camera frame goes through a body-tracking model, Google’s MediaPipe Pose Landmarker, that finds 33 points on your body: shoulders, elbows, wrists, hips, knees, ankles, heels, toes and more. It estimates where each point is in the picture and in 3D, up to about thirty times a second on most phones.
2. Smoothing and calibration
Raw tracking jitters slightly from frame to frame, so the points are smoothed with a filter that stays steady when you’re still but keeps up when you move fast. When you hold still before your first rep, the coach also learns which way is up (in case your phone is tilted) and what your start position looks like.
3. Measuring the movement
From those points the coach measures what matters for each exercise: the angle of your knees and hips in a squat, how far your hips travel back in a Romanian deadlift, your elbow angle in a curl, or the line from your shoulders to your ankles in a plank. It also works out whether it’s seeing you from the side, the front or an angle, and only checks what that view can reliably show.
4. Counting reps
A rep counts when you move from your start position to the target depth and back again. Half reps don’t count, and small wobbles at the top or bottom can’t trigger a double count. That’s why the numbers match what a strict coach would count.
5. Checking your form
Each exercise has its own form rules: knees caving in, back rounding, hips sagging, swinging the weight, leaning back in a press, and more. The coach only flags a fault when it sees it consistently, not on a single noisy frame, so it corrects you without nagging.
6. Coaching you out loud
Cues are spoken by your device’s own voice: short corrections like “Chest up!”, rep counts, and encouragement when you fix something. Important corrections jump the queue; routine counting waits its turn.
7. Scoring the set
Every rep starts with a score of 100. A minor fault takes a few points off and a major one takes off a lot more. Your form score for the set is the average of your reps, so one messy rep doesn’t ruin a good set, but a habit shows up clearly. For a plank, the score is the share of the hold you spent without a major fault. After the set, the summary tells you the one thing to work on next time.
Why the camera angle matters
A single camera sees in two dimensions, and depth is hard to judge. From the side, the angles of your hips and knees are clear, but your knees could cave in without the picture changing much. From the front, it’s the other way round. That’s why each exercise has a recommended view, and why the coach only checks what the current view can show. How to set up your phone lists the best view for each exercise.
Why your video stays private
All of this runs inside your browser on your own device. The camera video is never uploaded, recorded or stored, and you don’t need an account. Only your settings and workout history are saved, in your browser. The app saves itself on your first visit, and the tracking model the first time you start a set, so after that it works without a connection.
What it can’t do
A single camera can be fooled by poor light, baggy clothes or a body part that’s out of view, and it can’t feel what you feel. Use it as a second pair of eyes, and see a professional for pain or injury.
Get the best tracking with the tips in how to set up your phone.