Building for Reliability: Notes from Operating Hermes

I've been running Hermes Agent — an open-source AI agent platform — as my primary assistant for several months now. It lives on a NUC in my house, talks to me over WhatsApp, browses the web, runs terminal commands, and generally tries to be useful without burning down my homelab.

Operating an agentic AI system 24/7 has taught me things about reliability that no amount of theory would have conveyed. Here are the ones that matter most.

1. Agent Loops Amplify Every Failure

A traditional web service has a request-response cycle measured in milliseconds. An agent loop runs for minutes or hours, making dozens of tool calls, each of which can fail. A transient network error that a web service handles with a retry becomes a corrupted state in an agent.

The solution isn't more retries. It's idempotency and state snapshots. Every significant operation should be able to resume from the last known good state.

2. Model Fallbacks Are Not Optional

Every provider has outages. Every model degrades. If your agent depends on a single model, your agent is as reliable as that model — which is to say, not very. I run a multi-provider fallback chain: primary → fast-fallback → cheap-fallback. The agent should never stop because Anthropic or OpenAI had a blip.

3. Rate Limits Will Find You

Agent loops are chatty. A single task might burn 50 rapid-fire API calls. Without careful throttling, you'll hit limits within minutes. The fix is a token-bucket rate limiter that's aware of the agent's current task depth — background tasks can wait, foreground interactions can't.

4. Observability Is the Killer Feature

When an agent goes off the rails — hallucinates a tool call, gets stuck in a loop, makes six API calls to do something that should take one — you need to know. I log every tool call, every completion token, every state transition. Without this data, debugging is hopeless.

5. The Human-in-the-Loop Contract

Agents make mistakes. The key design question is: how expensive are those mistakes? For reading operations, let the agent run free. For writes to production, require confirmation. For anything involving payment or identity, the agent should be physically incapable of proceeding without a human nod.

This sounds obvious. It's frightening how many agent frameworks get it wrong.