Supply Genie

Personal demo of an AI worker for supply-chain exceptions: playbooks that run themselves, a live phone agent, and a person on every customer-facing step.

A personal demo, built while applying to HappyRobot and inspired by how HappyRobot approaches the problem: an AI worker for two supply-chain exceptions, a shipment delay and a worker absence. Webhooks carry an Idempotency-Key, so a retried delivery never opens a second incident; Python playbooks then post to Slack, place Twilio calls and amend the purchase order, and the customer email waits for an operator's approval. A Gemini 2.5 Flash chat and a Gemini Live phone agent request actions through the same command list and approval rule. Not deployed; no users or outcome metrics.

Problem: Supply-chain exceptions such as a late shipment or a missing worker are coordinated by hand across phone, email, Slack and order systems. Handing that work to an AI agent is only safe if a retried webhook cannot open a second incident, a timed-out call is never placed twice by an automatic retry, a purchase order is never silently overwritten, and a customer email never goes out without a person's approval.

How it was built: FastAPI and PostgreSQL backend. Webhooks require an Idempotency-Key, each incident becomes ordered action runs that plain Python executes, customer-facing actions wait as needs_approval rows until an operator decides, and purchase-order writes use a version check. Slack, Twilio and Resend calls run behind per-connector timeouts and circuit breakers, with full-jitter retries, a dead-letter list, manual requeue and an optional PostgreSQL queue worker (SELECT … FOR UPDATE SKIP LOCKED). Gemini 2.5 Flash drives the operator chat through three function tools, and Pipecat bridges Twilio Media Streams to Gemini Live for phone calls, with Redis checkpoints and a per-call command bus. Next.js and React operator workspace.

Stack: FastAPI, Gemini Live, Twilio, PostgreSQL, Pipecat, Gemini 2.5 Flash, Redis, Python, SQLAlchemy, Next.js, React, Slack SDK, Resend, OpenTelemetry

Tags: AI Agent Case Study, Voice AI, Tool Calling, Human-in-the-Loop, Failure Handling

GitHub: https://github.com/RohithReddy20/supply-genie

Overview

Supply Genie is a personal demo of an AI worker for two supply-chain exceptions, a shipment delay and a worker absence. A webhook opens an incident, a fixed playbook turns the incident into ordered actions (for a delay: a Slack alert, a one-way Twilio call that reads the new ETA to the supplier, a purchase-order amendment and a customer email), and an operator workspace shows every step. The customer email waits for an operator's approval. Operators can ask a Gemini 2.5 Flash chat about the incident or tell the chat to run a step, and a separate phone agent streams live calls into Gemini Live. The project was built while applying to HappyRobot and is a conceptual demo inspired by how HappyRobot approaches the problem, not a production system: no users, no deployment and no outcome metrics, and the repo's prompts and UI labels still carry the HappyRobot name. The demo is about the engineering around the model: retry-safe intake, approvals stored as data, versioned purchase-order writes, and failure handling that refuses to auto-retry a Slack post or a phone call when the outcome of the first attempt is unknown.

Architecture

Operator workspace (Next.js): Next.js 16, React 19, Tailwind 4 and shadcn/ui. The incident inbox polls every five seconds and groups incidents into needs-action, escalated and resolved; a playbook strip marks the gated step with a lock; approval cards sit in the center column; an action history shows each step's retries, errors and approval decision; and a chat panel relays questions and commands to Gemini through the API. Every status on screen comes from a database row.

API (FastAPI): Python and FastAPI. The delay and absence webhooks require an Idempotency-Key header; other routes handle approvals, retries, the dead-letter list and requeue, chat, and the Twilio voice webhooks plus the media-stream WebSocket. Middleware stamps a correlation ID on every request and response and stops non-voice requests after 90 seconds (default). The readiness probe reports database health, circuit-breaker states and active calls.

Agents (Gemini): The operator chat calls Gemini 2.5 Flash with three function tools (incident status, shipment lookup, run a command) and loops for up to five tool rounds per message. The phone agent runs Gemini Live with native audio inside a Pipecat pipeline and has five tools, adding call-progress tracking and hang-up. Neither model talks to Slack, Twilio or Resend directly: both agents request incident actions through the same command list, executor and approval rule. After each call, Gemini 2.5 Flash turns the transcript into a structured summary that is saved with the call and posted to Slack.

State (PostgreSQL and Redis): PostgreSQL, through SQLAlchemy and Alembic, holds suppliers, shipments, purchase orders, incidents, action runs, approvals, voice sessions and transcripts. Each step's status, attempt count, request and response payloads and last error live on a single action_runs row, and in queued mode the same table is the work queue. Redis holds only short-lived call state: a checkpoint per active call, with a one-hour expiry by default, and a command list per call.

Connectors and failure handling: Slack (slack-sdk), Twilio (REST calls and Media Streams) and Resend are real integrations; the purchase-order system is a PostgreSQL table with a version column, and the labor (HRIS) system is a mock. Slack, Twilio and Resend calls each run behind a per-connector timeout and circuit breaker. Failed actions retry with exponential backoff and full jitter, then move to a dead-letter list that can be requeued through the API.

Core Engineering

Retry-safe intake and versioned purchase orders: The delay and absence webhooks reject a request without an Idempotency-Key header (422). A known key returns the existing incident with 200 instead of 201, so a sender that retries never opens a second incident or runs the playbook twice, for example calling the contractor agency twice for a single absence. When two requests race with the same new key, a unique constraint lets a single insert through; the losing request rolls back and returns the winning incident. Purchase-order amendments use a version compare-and-swap instead of a lock: the UPDATE bumps the version and only matches while the version is unchanged since the read, so a concurrent edit produces a conflict error rather than an overwrite.

Playbooks run by plain Python: A playbook is a Python list, so a delay always becomes four action runs (Slack alert, supplier call, PO amendment, customer email) and an absence three (labor-system update, contractor call, site-manager alert in Slack). A deterministic executor walks the runs in sequence order, sends each pending action to the matching connector, saves the request and response payloads on the row, and writes every action in a separate database transaction, so a later failure cannot roll back an earlier PO change. An incident resolves once every run is completed or skipped, and escalates when any run is dead-lettered. No model decides what a playbook does next.

Approval gates stored as rows: A customer-facing action, here the customer email, is created as needs_approval next to a pending row in the approvals table, under a setting that is on by default. The gate follows the action type, not a model's sense of risk, and neither agent has a tool that approves. A decision is a single UPDATE that only matches while the approval is still pending, so a double click or a racing request gets 409 instead of a second email. Approving moves the action to pending and dispatches the action; rejecting marks the action skipped. Asking either agent to email the customer produces a gated approval instead of an email.

Failure handling per connector: Slack, Twilio and Resend calls each run behind a dedicated timeout (defaults: 10 s, 15 s and 10 s) and a circuit breaker that opens after five straight failures, lets calls through again after 60 seconds and closes after two successes (defaults). A failed action retries with exponential backoff and full jitter (200 ms base, 10 s cap) and is dead-lettered after the third failed attempt, with a fallback note stored on the row. Timeouts get stricter treatment: a timed-out Slack post, Twilio call or manager alert may still have gone through, and no idempotency key reaches Slack or Twilio, so the action goes straight to the dead-letter list for a person to check. Email stays retriable because every attempt sends Resend the same idempotency key. Dead-lettered actions can be listed and requeued through the API.

A PostgreSQL queue, and no durable engine yet: By default, actions run inline, inside the API request that created or approved the actions. With queued mode switched on in configuration, the actions are marked queued instead, and a background worker claims the oldest queued action with SELECT … FOR UPDATE SKIP LOCKED, runs that incident's queued actions, and requeues failed actions once the backoff window has passed. A queue-status endpoint reports queue depth, retriable and dead-lettered counts, and worker health. A written evaluation in the repo compares the queue with a durable workflow engine such as Temporal and recommends keeping the queue until defined adoption gates are met; selecting the durable mode of the workflow-engine interface raises an error on purpose.

Live phone agent: Twilio, Pipecat, Gemini Live: Inbound calls, and outbound calls started through the voice API, get TwiML that opens a Twilio Media Stream to a WebSocket on the backend. A Pipecat pipeline converts the 8 kHz telephone audio and streams the audio to and from Gemini Live, with Gemini's voice-activity detection tuned (high start and end sensitivity, 260 ms of silence to end a turn; defaults). When the caller talks over the agent, a custom Pipecat processor sends Twilio a clear event so buffered agent audio stops playing. The agent marks four call objectives (cause, new ETA, mitigation, risk) through a tool and closes the call after a brief closing statement once all four are confirmed; the transcript is saved and summarized into Slack. The playbook's supplier and contractor calls are different: one-way Twilio calls that read a fixed message.

Operator chat with Gemini tool calling: The chat panel sends the operator's message, the conversation so far and a snapshot of the incident to Gemini 2.5 Flash with three function tools: get_incident_status, list_active_shipments and execute_command. The backend runs each tool call and returns the result to the model, for up to five rounds per message. The execute_command tool accepts only a fixed list of commands: internal steps run immediately through the same executor as the playbook and report success or the connector's error, while a request to email the customer becomes an approval card instead of an email. Status answers are grounded in the action rows, because get_incident_status returns each step's status and last error.

Call control from any server: Groundwork for running several backend servers, built but never deployed to more than a single server. Each active call records the owning server's ID and a heartbeat, and checkpoints state to Redis every five seconds and on each progress or end-call tool call: objective flags, close state and the transcript tail, with an expiry. A control command (only end_call is supported) runs locally when the call lives on the receiving server; otherwise the command goes onto a per-call Redis list that the owning server polls every second. Routing fails closed: no Redis bus returns 503, no checkpoint returns 404, and a checkpoint older than 20 seconds returns 409 (timings are defaults). Kubernetes and NGINX sticky-routing templates and a multi-pod smoke-test runbook are in the repo; the smoke test has not been run.

Execution Flow

Webhook opens an incident: An upstream system posts a shipment delay with an Idempotency-Key. A repeated key returns the existing incident with 200. A new key creates the incident and four action runs in one transaction, with the customer email stored as needs_approval next to a pending approval.

Internal steps run in order: The executor, inline by default or in the queue worker, posts the Slack alert, places the one-way supplier call and amends the purchase order with the version check. Each step is saved in a separate transaction and recorded as an OpenTelemetry span.

Failures stay contained: A failed step stores the error and the next retry time. Retries use jittered backoff and come from the queue worker or the Retry failed button in the workspace. A timeout on Slack or Twilio dead-letters the step at once; a third failure dead-letters the step and moves the incident to Escalated in the inbox. Dead-lettered steps are requeued through the API.

The gate waits for a person: The workspace marks the email step as awaiting approval and shows an approval card in the center column; the operator can ask the chat for a status summary first. Approve sends the email through Resend with the action's idempotency key; Reject marks the step skipped. A second decision on the same approval gets 409.

A live call, when needed: Separately from the playbook, a call into the Twilio number, or an outbound call started through the voice API, reaches the Gemini Live agent. The agent reads incident status, can request a step through the same approval rule, and leaves a structured summary in Slack after hang-up.

The incident resolves: Once every action is completed or skipped, the incident is marked resolved and moves to Recently resolved in the inbox. Each step's payloads, errors and approval decision stay on the action and approval rows.

Key Learnings

Keep the model out of the failure path. Gemini picks from a fixed tool list, and plain Python owns retries, breakers, timeouts and state changes. A bad model turn can request the wrong step, but cannot approve a customer email or change how failures are retried.

Idempotency belongs at the edge, before any side effect. A required header plus a unique constraint stop a retried webhook from opening a second incident and calling the same contractor agency twice.

An approval is a row, not a prompt. A needs_approval status plus a compare-and-swap on the decision means a double click returns 409 and the customer gets a single email; a prompt telling the model to ask first would guarantee neither outcome.

A timeout means unknown, not failed. A Twilio call that timed out may still ring, so a non-idempotent connector should stop and hand the action to a person; email can retry because every attempt sends Resend the same idempotency key.

Adopting a durable workflow engine is a decision with entry criteria. The repo's evaluation keeps the PostgreSQL queue until at least one adoption gate holds: workflows that regularly run past 30 minutes with timers, compensation across more than three external systems, worker restarts that keep needing manual fixes, or audits that need replayable history.

Back to Archive

Multi-Agent Orchestration

NOV 2026 · PROTOTYPE

Paradev Office

A city of your projects where every running coding agent has a desk: see who needs you, answer their questions, seat agents from different repos together, and get work back only after a second agent has checked it.

Artifacts

Repo

Stack

Multi-Agent OrchestrationDeveloper ToolsElectron + PhaserAgent GuardrailsHuman-in-the-Loop

Synopsis

Paradev is an open-source desktop app for running coding agents: each task gets a git worktree, a terminal and whichever CLI agent you pick. Once several agents run at once, the work turns into a row of terminals. You check each tab to find the one stuck on a question, agents in different repositories can't ask each other anything, and the only sign a task is finished is the agent saying so.

In November 2026 I built the City and the Office on a fork of the app. Every project is a building, every workspace a room and every running agent a named teammate at a desk, drawn from what the app already tracks. From the City a person can see who is working, finished or waiting, answer an agent's on-screen question by name, seat agents from different repositories in one room so they talk by @mention, hand work to another workspace, and give an agent a task that a check command and a second agent verify before it comes back. Agents can talk to each other, but guards outside the model bound every exchange, and a person can stop any of it.

It is a prototype on my fork, built and tested in the running app against small demo repositories. The screenshots on this page are from those test runs.

The Product

Screenshots · GitHub

Every coding agent you run, at a desk you can walk up to, with a person deciding what is done.

The City view in the Paradev desktop app: four pixel-art buildings on a dark grid, the Office (3 idle) and three demo projects, city-demo-api (1 done), city-demo-web and city-demo-ops (nobody in). Lit windows mark agents. A 'Needs you' bar under the header shows one item: 'Done, take a look', Bolt, Add health check, 22 minutes ago. The sidebar lists the projects and their workspaces. — Every project is a building and every running agent a lit window. The bar on top lists what is waiting for you, most urgent first.

Product features

Watch · Talk · Answer · Check · Reconnect

Watch

Inside a building, each workspace is a room and each agent sits at a desk with its name and status. Its panel shows the conversation, where it works and a link to its terminal.

My partI built the City and the floors: buildings drawn from the app's own projects, workspaces and agents, window and bubble colours from each agent's status, names that follow an agent to a new terminal, a Map and List view, and the 'Needs you' bar.

Inside the city-demo-api building: an agent named Bolt sits at a desk with a speech bubble saying 'Done, take a look', next to the 'Add health check' room sign. Bolt's panel on the right shows its kind and @bolt handle, the workspace it works in, a Done status, where it sits, 'Hand its work to another workspace…', and its conversation, in which it introduces itself as Bolt.

Talk

Agents from different repositories sit in one meeting room and talk by @mention. Each keeps its own workspace and branch; only the messages move between terminals.

My partI built seating and room talk: an answer that mentions a roommate is typed into that roommate's terminal with who it is from, an answer of 'no reply' is never passed on, and a room pauses after 8 messages without a person. In one test, two agents in separate repositories wrote and tested a server and its client in about 20 seconds, with 4 turns in all.

The meeting room panel with two agents seated, one from the Check health workspace and one from Plan release, both Done. The person asks @claude-plan-release to ask @claude-check-health which files are in its folder; the first asks the question, the second answers that its folder in city-demo-api holds README.md, count.txt and health.txt, and the first reports the answer and stops without mentioning anyone. The office floor on the left shows both agents in the meeting room.

Answer

When an agent stops at its own on-screen menu, such as 'Trust this folder?', the City spots it, raises it in the bar, and lets you answer with a named button.

My partI built the screen reading: no event fires while a CLI waits at its own menu, so the City reads the terminal's screen every 2.5 seconds, finds the menu and turns each option into a button that sends the right arrow keys and Enter. A blind Enter would have quit one CLI, whose trust menu starts on 'No'.

The city-demo-ops floor with a Codex agent named Milo at a desk in the Rotate logs room. The 'Needs you' bar shows 'Asks to trust this folder, Milo'. Milo's panel shows 'Milo asks: Trust this folder?' with two buttons, 'Trust and continue' and 'Quit', above a copy of its terminal screen (the folder path is blurred) and the raw arrow, Enter and Esc keys.

Check

Give an agent a task with what done means and a check command. When it reports done, the check runs and a second agent reviews the diff before the task reaches you.

My partI built checked tasks: the brief typed into the agent's terminal, the commit it started from, the check command, a checker agent that changes nothing and ends with PASS or FAIL, up to 3 rounds back to the agent, and close, send back or ask for a pull request from the card. Here the checker passed the task in about 10 seconds.

Iris's panel with task C-1, Add HEALTH.md, marked 'Checked, ready for you'. It shows the agent's report (it created HEALTH.md containing 'ok' and committed it on main with no other changes), the check command passing, and the checker Otto's verdict that the commit adds only HEALTH.md with the single line 'ok', above the buttons 'Ask for a pull request', 'Close' and 'Send back…'. The 'Needs you' bar reads 'Checked, ready to close · C-1 Add HEALTH.md'.

Reconnect

If an agent's terminal stops, the agent stays at its desk marked disconnected. Reconnect resumes the same conversation in a new terminal with its name, seat and task.

My partI built the terminal watch: a dead terminal is flagged in about 20 seconds instead of the roughly 10 minutes the app took to notice, Reconnect resumes it through the app's own resume path, and an open task gets one 'your terminal restarted, carry on' message once the agent is back.

The city-demo-api floor with three disconnected agents. The 'Needs you' bar lists Disconnected chips for Iris and Nova. Nova's panel explains that its terminal stopped, maybe because the app restarted, and that Reconnect continues the same conversation in a new terminal with its name, seat and task, above Reconnect and Remove buttons and its open task C-4, Add WAIT.md.

Architecture

System Layers

City and Office floors

Phaser scenes in the desktop renderer. The City draws one building per project, one floor per workspace and one window per agent, lit by status. A building opens on its office floor: a map built in code from walls, doorways and floor materials, with rooms, desks, a camera that follows you, and DOM overlays for name tags, room signs and panels. A Map and List switch and the 'Needs you' bar sit on top.

Iglobe

City rooms (host service)

Seats, names, room talk and hand-overs, kept in the app's home folder and served over tRPC. It listens to the app's agent lifecycle events; when an agent stops, it reads the agent's last answer from its own transcript and types it into the terminal of every roommate it mentioned, with who it is from and how to answer.

IIserver

Office teammates over ACP and MCP

The Office's own teammates are long-lived agent sessions driven over the Agent Client Protocol, with office tools served over MCP: say, set_status, who_is_here, read_thread and the task tools. Presence comes from the session stream, and CLI agents seated in the Office join its rooms as guests.

IIIcpu

Delivery guards

Every message from one agent to another is a hand-off record with a hop depth from the person who started it. Past 3 hops, 6 agent posts in a thread without a person, 3 teammates pulled in by one turn or 60 turns a teammate a day, the message is held with a visible reason. Repeated asks are dropped, a paused teammate refuses, and a person writing 'stop' cancels the thread.

IVlayers

Tasks and the terminal watch

Tasks live next to the City's state with the commit each started from, a check command and a checker. A terminal watch compares the app's agent records with the terminal daemon's live sessions, so an agent whose terminal died shows as disconnected instead of idle.

Vdatabase

Execution Flow

5 steps

Step 01
A person gives a task
From an agent's panel: a title, what done looks like, an optional check command and which agent checks it. The commit checked out at that moment is kept as the base, and the brief is typed into the agent's terminal.
Step 02
The agent works and reports
It works in its own workspace and ends with TASK DONE or TASK BLOCKED. When it stops, the City reads that last answer from its transcript.
Step 03
The check runs
The check command runs in the workspace. A failure goes back to the agent with its output as the next round.
Step 04
A second agent reviews
A checker starts in the same workspace with the base commit, the report and the check result, and answers PASS or FAIL. A FAIL goes back to the agent; after 3 rounds the task is blocked.
Step 05
It reaches the person
A passed task shows as 'Checked, ready to close' in the bar. The person closes it, sends it back with a note, or asks for a pull request, and the card records the link.
Done

Core Engineering

Field Notes

One view built from what the app already tracks

The City joins the app's projects, this machine's workspaces and the agents running in their terminals, and takes each agent's status from the same rules as the sidebar, so watching adds nothing that runs on the host.

Buildings, floors and windows follow from that join; opening a finished agent marks it seen, the same as focusing its terminal.

Guards outside the model

Agents are told the rules, but the office enforces them: hop depth from the person, agent-only rounds per thread, teammates pulled in per turn, a daily turn budget, dropped repeats and a person's 'stop'.

City rooms pause after 8 passed messages without a person until someone speaks or resumes them.

Answering on-screen questions by name

A CLI waiting at its own menu, such as 'trust this folder?' or a permission prompt, fires no event and looked idle.

The City now reads the screens of agents that haven't finished a turn every 2.5 seconds, recognises a menu by its cursor and footer, and raises it as 'Needs your OK'. Each option becomes a button that sends the arrow keys from the cursor to that option, then Enter.

Agents whose terminal stopped

After a restart or a crash an agent's terminal could be gone while the City still drew it as idle; the app only marked it ended after two missing checks five minutes apart.

A terminal watch now compares the daemon's live sessions every 10 seconds and flags a terminal missing twice in a row, so a dead agent shows as disconnected in about 20 seconds. Reconnect resumes it through the app's own resume path with its name, seat and task, and an open task gets one message to carry on.

Time to flag an agent whose terminal died

Before~10 minAfter~20 s

Messages move, work stays put

An agent can sit in any room in the City, but its workspace, folder, branch and terminal never change by sitting.

A newcomer gets one note: the room, its own @handle, who else is there and where they work, and to write only 'no reply' when it has nothing to add. Reading the agents' records after the first test showed a welcome message and a polite closing each costing a turn; after the fix, two agents in separate repositories wrote and tested a server and its client in about 20 seconds with two messages passed and 4 turns in all.

Hand-overs only when a person says so

Moving work to another workspace is separate from sitting.

A hand-over asks the agent for a brief (what was asked, what is done, what is left, the files and decisions that matter), starts a new agent in the target workspace through the app's own launch path with that brief, and seats it where the old one sat. An agent can ask for a hand-over by writing a request line; it waits in the bar until the person approves or declines it.

Checked tasks

A task types a brief into the agent's terminal: what to do, what done means, its folder and branch, what will check it, and to end with TASK DONE or TASK BLOCKED.

On TASK DONE the check command runs in the workspace, then a checker agent in the same workspace reads the diff from the starting commit and answers PASS or FAIL without changing anything. A failure goes back as the next round, up to 3; a pass reaches the person, who can close it, send it back with a note, or ask the agent to open a pull request.

Names and identity

Every agent gets a short name (Ada, Milo, Iris) and an @handle that is the same in every room.

A name follows its agent into a new terminal and is kept for a day after it stops, and the first thing the City types into an agent tells it who it is and where it works, so agents answer to their names. Renames that clash are refused.

Reflections

Outcomes

“Read the screen when there is no event. A CLI waiting at its own menu fires nothing, so it looked idle; reading its terminal was the only way to know it needed a person.
“Keep the work where it is. Letting agents sit anywhere while their folder and branch never move made cross-repository talk safe; moving work became a separate hand-over a person approves.
“Turns are the cost. A welcome message and a polite closing each cost a turn; a note on arrival and never passing on 'no reply' removed both.
“Guards belong outside the model. Hop limits, agent-only rounds and a person's stop are enforced by the office, not asked of the agents.
“Done should mean checked. A check command plus a second agent that cannot change anything turns 'the agent says it's done' into something a person can close in one click.

Next Steps

Token and money budgets per task, not only rounds and daily turns.
Remote and cloud hosts in the City; today it shows only this machine's workspaces.
Hand-overs into a new workspace, not only an existing one.
Move the floor scene and overlays the City borrows from the Office into one shared place.

One Last Thing

The core challenge: Running several coding agents at once means a row of terminals. You find out an agent is stuck on a 'trust this folder?' prompt by opening each tab, agents in different repositories can't ask each other anything, and 'done' means whatever the agent says. Letting agents talk to each other is only safe if a loop between two of them stops on its own, a person can stop any of it, and finished work is checked by someone other than its author before a person spends time on it.

View Source

Paradev Office

Synopsis

The Product

Watch

Talk

Answer

Check

Reconnect

Architecture

City and Office floors

City rooms (host service)

Office teammates over ACP and MCP

Delivery guards

Tasks and the terminal watch

Execution Flow

A person gives a task

The agent works and reports

The check runs

A second agent reviews

It reaches the person

Core Engineering

One view built from what the app already tracks

Guards outside the model

Answering on-screen questions by name

Agents whose terminal stopped

Messages move, work stays put

Hand-overs only when a person says so

Checked tasks

Names and identity

One Last Thing

More from the Archive

Supply Genie

Aegis Compliance Agent

Vestra Assets