On-device rep counting via torso axis vector tracking
tflite_flutter, ensuring zero latency and strict user privacy by avoiding server-side video transmission. The primary engineering challenge was not the model inference itself, but the inherent noise and perspective distortion present in monocular 2D projections.Initial iterations relied on joint-angle thresholds, assuming that a specific degree measurement at the knee or elbow would trigger a repetition event. However, this approach failed in production scenarios. Because mobile cameras capture a 2D projection of a 3D space, the perceived angle in the image plane is highly sensitive to the camera's pitch and yaw. Even incorporating Z-axis data from the model proved unreliable due to the high jitter associated with monocular depth estimation. I attempted to mitigate this by implementing a collinearity check—a "body-line" gate—to verify the alignment of the shoulder, hip, and heel during push-ups. This also succumbed to perspective distortion; when a user positioned their device at an oblique angle, the perceived line appeared bent, causing the logic to fail and the user experience to degrade.
The technical pivot occurred when I reframed the problem: I stopped treating the human body as a rigid geometric object and started treating the movement as a dynamic signal. I transitioned from absolute angle detection to tracking the orientation of the torso axis vector. This vector-based approach is significantly more resilient to arbitrary camera placement.
To validate these heuristics, I developed an internal debug switch within the Flutter/Riverpod/Drift stack. This allowed for real-time hot-swapping of signal candidates—comparing torso displacement against normalized limb travel—while observing the live feed. The final implementation prioritizes relative displacement patterns that remain invariant to the camera's perspective. For anyone building mobile ML pipelines, the takeaway is clear: absolute spatial coordinates are deceptive. You must engineer features that survive the chaos of real-world spatial transformations.
https://pub.dev/packages/tflite_flutter