Systems · Personal AI
Designing Faye 🦋
Notes on building a personal AI that doesn't just chat — it remembers, delegates, checks its own work, and quietly runs things in the background.
Most “AI assistants” are a text box wired to a model. You ask, it answers, and the moment the window closes the whole relationship resets. Faye 🦋 is my attempt at the opposite: an assistant that persists, acts on its own between messages, and treats a single chat as the least important part of the system. This is a tour of how it's built — and, just as much, of the specific ways it broke and what each failure forced me to design.
Four ideas do most of the work: many front doors, one brain; the brain is a coordinator, not a genius; memory is the product; and the whole thing tunes itself, carefully. Everything else is detail.
Many front doors, one brain
Faye 🦋 answers from wherever I happen to be — a chat channel on my phone, or a native voice app I can just talk to. Those are only interfaces. They all route to the same brain, and the brain's actual state doesn't live in any conversation. It lives in a set of plain files on disk.
This split matters more than it looks. Because no channel is Faye — they all just talk to it — I can lose a conversation, restart a process, or switch from typing to speaking mid-thought, and the assistant is still the same one that knew what I was doing an hour ago. The interface is disposable. The brain and its memory are not.
The brain is a coordinator, not a genius
The single most useful design decision was to stop asking one model to do everything. Faye 🦋 the brain is a coordinator. For real work — writing a multi-file feature, refactoring, digging through a large codebase, searching the live web, reading a long PDF — it routes the job to a specialized worker and manages the result. It doesn't try to be the best at every task; it tries to pick the right tool and be a good reviewer.
Three rules keep this honest, and all three came from getting burned:
- Workers start blind.A worker can't see the coordinator's conversation, so every delegation has to be a fully self-contained brief — the goal, what's already ruled out, and what “done” looks like. “Based on your findings, fix it” is a bug; the coordinator has to do the synthesis first.
- Never trust the self-report.A worker saying “done, tests pass” is a claim, not evidence. The coordinator re-runs the tests, spot-checks the output, and writes its own verdict. A delegation isn't finished until it's been independently verified.
- Long work goes to the background. Anything slow is fired off detached, with a hard time cap and an automatic notifier, so the conversation can end immediately instead of sitting there frozen while something churns.
The coordinator's job isn't to be the smartest agent in the room. It's to route well and refuse to be fooled — including by itself.
Memory is the product
If the brain is a coordinator, memory is what makes it mine. Faye 🦋 keeps a small set of structured files instead of one giant transcript: a live board of what's active today, a status dashboard plus an append-only history of everything done, and a set of deep-context files for each area of life. Recall is a lookup into those files, never a guess.
The subtle failure here is drift. Early on, finishing a task meant editing a couple of these files by hand — and inevitably one got updated and another didn't, so the “done” list and the “now” list disagreed and the whole memory quietly lost trust. The fix was to make closing a task a single atomic command that writes every affected layer together. You either move all of them or none of them. Consistency isn't a habit you rely on; it's a property the tool enforces.
The rule I'd underline: the human should never have to maintain the assistant's memory. If keeping it accurate takes discipline, it will rot. Make the correct update the only update.
The whole thing tunes itself — carefully
The last piece is that Faye 🦋 is meant to get better without me hand-editing it forever. Every real conversation is logged; reactions and implicit signals (did I have to rephrase? correct it?) become labels; a separate judge scores traces. That data feeds a bounded optimization loop over the assistant's own instructions.
The discipline is what makes this safe rather than a slow-motion disaster. Nothing ships from a one-shot prompt: you write the harness spec first, run it, grade it against explicit checks, and only revise if there's real headroom — capped at a few rounds. And the judge is graded against a locked, held-out set the loop can never see. Point an optimizer at a score it's allowed to peek at and it'll learn to game the score instead of doing the job. I've written separately about why that evaluator, not the model, is the real bottleneck.
What broke, and what it taught me
None of these rules are theory. Each is a scar. The most transferable lessons from building an assistant that runs unattended:
| What broke | The lesson |
|---|---|
| Claimed done, wasn't | The worst failure isn't a wrong answer — it's a confident false one. Never report an action as done without evidence from that same turn. |
| Partial migration | Change one shared dependency and something scheduled breaks at 3am. When you move a path or a name, sweep every consumer in the same change and test it under the real environment, not the convenient one. |
| Script edited itself | A long-running program that rewrites its own file mid-run corrupts itself. Stage the new version and swap it in atomically; never hot-edit something that's live. |
| Blocked on a wait | A rule that's right for an interactive session can freeze a one-shot run. Every automation rule needs an explicit answer for the headless case. |
| Restarted mid-turn | Never bounce the server while a turn is in flight. Know what's running before you touch anything shared. |
There's a single thread running through all of them: the danger in an autonomous assistant isn't that a model isn't smart enough. It's the seams — handoffs, shared state, background work, self-modification — where things silently fall out of sync and nobody's watching. Most of the engineering is making those seams loud, atomic, and grounded.
Where this is going
Faye 🦋 today is reactive: it answers, delegates, remembers, improves. The horizon I'm building toward is something closer to Samantha from Her— an assistant that initiates instead of waits, has taste rather than just competence, and models a person over years, not turns. Many front doors, a coordinator brain, durable memory, and a careful self-tuning loop aren't the whole of that. But they're the foundation you need before proactivity is a feature instead of a liability.