Building an AI-Native Fitness Platform with an Autonomous Command Center
How I designed and built Fitly AI, a full-stack AI coaching platform with intelligent backend agents, real-time monitoring, and a self-improving system. Built with Claude Code and Codex.
More Than a Chatbot. An AI That Actually Does Things.
The fitness app market is saturated with calorie counters and workout loggers that bolt on a chatbot as an afterthought. I wanted Fitly AI to be different: an AI-native platform where every interaction happens through natural conversation. Users talk to their coach; the AI logs meals from photos, generates personalized workouts, tracks macros, and adapts over time.
But here's the real challenge: when an AI is making thousands of decisions per day (estimating calories from a blurry food photo, choosing the right exercises for a bad knee, deciding when to escalate to a more powerful model) you need visibility and control. You need a command center.
"How do you monitor, manage, and continuously improve an AI system that's autonomously coaching hundreds of users without hiring a team of people to babysit it?"
A Three-Layer AI System
A conversational AI coach on the front end, an intelligent command center in the middle, and autonomous agents working behind the scenes.
The AI Coach
A chat-first mobile experience where users interact through natural conversation instead of tapping through forms. The AI understands context, remembers preferences, and takes real actions.
- Vision-powered meal logging. Snap a photo, AI identifies food and estimates macros with clarifying questions before committing
- 30+ specialized AI tools. Structured actions for logging, querying, planning, and updating fitness data
- Intelligent model routing. Lightweight model handles simple requests; complex ones auto-escalate to Claude Opus
- Real-time streaming. SSE-based responses with typewriter effect for a natural coaching feel
- Personalized workout & meal plans. Generated from user goals, injuries, dietary preferences, and available equipment
The Command Center
A real-time operational dashboard that gives full visibility into what the AI is doing, how users are responding, and where the system needs attention. Every conversation reviewable. Every decision traceable.
- Conversation browser. Search, filter, and review any user conversation with full AI response and tool call details
- Feedback analysis. Aggregate user signals (thumbs up/down) to identify where the AI struggles and prioritize improvements
- AI cost tracking. Per-model usage and spend analytics so you always know what your AI is costing
- Content management. Built-in CMS for SEO content with a publishing pipeline and approval workflows
16 Autonomous Agents
A framework of specialized bots and AI agents that handle backend operations without human intervention, with a human-in-the-loop approval system for sensitive actions. Each agent has its own tools, schedule, and domain of responsibility.
Key Technical Decisions
The architecture patterns that make Fitly AI reliable, cost-effective, and ready to scale.
Provider-Agnostic AI
An abstraction layer lets us swap between Claude and OpenAI without changing business logic. Adapters handle each provider's quirks. Future-proof against model changes and price shifts.
Intelligent Context Caching
User context split into stable (profile, goals) and volatile (today's meals) layers, cached in Redis with surgical invalidation. Reduced prompt overhead by 90%.
Structured AI Output
Every AI response validated against a TypeScript schema before rendering. Actions are type-safe and database-ready. Display components are structured data, not raw text.
Real-Time Streaming
SSE-based chat with token-level events. API stays under 200ms. Heavy work dispatched to async job queues. Users see responses building in real time.
Self-Improving System
Passive collection of conversations, user corrections, and feedback signals. The AI gets smarter from usage. No manual labeling, no user friction.
Enterprise-Grade Security
Encryption at rest and in transit. Full audit trail on every data mutation. Pre-signed URLs for media. JWT auth with refresh tokens. Role-based admin access.
What I Delivered
Need an AI Command Center for Your Product?
Every AI product needs more than a model and a prompt. You need observability, feedback loops, autonomous agents, and a system that improves itself over time. I build these systems. See my full service offerings or get in touch.