One Conversation. A User's Entire Fitness Life.
How I built Fitly, a chat-first fitness app where a single agent logs meals from a photo, builds workouts around an injury, sets macros, and tracks progress. Behind the chat is a ten-layer orchestration engine that does the thinking before the model is ever allowed to speak. Built with Claude Code and Codex.
More Than a Chatbot. An Agent That Actually Does Things.
The fitness market is full of calorie counters and workout loggers that bolt a chatbot on as an afterthought. Fitly is the opposite: the conversation is the product. You talk to your coach, and it logs meals from a photo, builds a workout around a bad knee, recalculates your macros, and remembers what changed.
That only works if the agent is right. It takes real, irreversible actions on a user's health data, so a hallucinated "logged" meal or a plan that ignores an injury isn't a rough edge, it's a broken product. The hard part was never the chat. It's everything that has to happen, correctly, before and after the model is allowed to speak.
"The model is the last thing we trust. It is one component of the orchestrator, not the other way around."
One Agent, the Job of Five Apps
Meal tracker, workout planner, macro calculator, progress dashboard, and the coach you'd normally text. Fitly folds all of it into a single conversation, and because it's one brain, every part knows about the others.
Log it however you want. The numbers take care of themselves.
A meal, a glass of water, a workout, and a weigh-in, logged from a single sentence. No forms, no database search, no switching apps. Twenty seconds, done.
- Point, shoot, logged. A photo of a plate becomes a meal with macros. Nutrition labels, barcodes, and even a treadmill screen just work.
- You never enter a number by hand. Macro targets populate from the conversation, from weight, goals, and activity level.
- Programs that know your body. Workouts generate around injuries and the equipment you have, and adapt as you progress.
- Voice is first-class. Log a meal or a set hands-free while cooking, driving, or mid-workout.
- It remembers. Say you hate cilantro once, and no meal plan ever includes it again. Log a brutal leg day, and dinner advice accounts for what you burned.
Nine Systems Run Before the Model Is Allowed to Speak
The chat is the easy part. Every fitness AI in the store has the same models we do; the difference is what runs around the model. Ten layers execute on every message: context assembled fresh, tools scoped to the moment, the right model chosen for the job, and guards that verify what actually happened before a user ever sees a reply.
A few background jobs handle health checks and data-quality sweeps on a small internal agent framework. But the product users feel is the conversation, and the orchestration engine is what makes it trustworthy.
Key Technical Decisions
The patterns that make Fitly reliable, cost-effective, and ready to scale.
Provider-Agnostic, Multi-Model
Claude (the default) and OpenAI sit behind one interface, with automatic cross-provider fallback. I can swap models or ride out an outage without touching business logic. Future-proof against model changes and price shifts.
Cached Context Document
User context splits into a stable layer (profile, goals, injuries) and a session layer (today's macros, weight trend). Both live in Redis, so the hot path rarely touches the database. Prompt overhead dropped ~90%.
Structured, Validated Output
Every tool call is validated server-side before it runs, and every response is checked against a TypeScript schema. The model can't corrupt data or invent a save that never happened.
Real-Time Streaming
SSE chat with token-level events keeps the API responsive while heavy work is dispatched to async job queues. Users watch the response build in real time.
Open by Protocol (MCP)
Fitly runs a Model Context Protocol server, so Claude, ChatGPT, or any agent can read and update fitness data with scoped, token-based access. The data is usable without opening the app.
Evaluated & Secured
A personalization eval harness checks prompt, tool, and model changes against real cases before they ship. Health data is encrypted in transit and at rest, with an audit trail on every mutation and pre-signed media URLs.
What I Delivered
Want Production AI in Your Product?
Fitly is the playbook I bring to client work: real AI features that take actions on real data, wired into a product people use every day, with the orchestration and guardrails that make them trustworthy. That's the AI Integration Sprint. Or get in touch.