All articles
forge ai-agents reliability 5 min read

Your AI agent got stuck for five hours. Nobody noticed.

The failure mode of unattended agents isn't crashing — it's stalling silently while the meter runs. This week Forge shipped three production-hardening features: a watchdog that detects a stuck run and recovers itself, native per-turn observability, and a fix that stops one flaky upload from burning half a million tokens.

by Vitaly Nikitin

A few days ago, one of our own agent runs spun for more than five hours. It re-leased its active turn fifteen times. It made no progress the entire time. And nobody noticed — because the default failure mode of an unattended agent isn’t a crash. It’s a quiet stall while the meter keeps running.

That’s the part nobody warns you about when you start running coding agents 24/7. We’ve written before about the loop that ships its own features end-to-end. Letting a loop like that run unattended is exactly when reliability stops being a nice-to-have. This week we shipped three things that turn “the agents can ship” into “the agents can ship safely while you’re asleep”: a watchdog that notices a stuck run and fixes it itself, native per-turn observability so you can actually see what an agent did, and a payload fix that stops one bad upload from torching half a million tokens.

Here’s each, and the current discourse it speaks to.

1. A watchdog that recovers the run itself — not just pings you

The recurring horror story of unattended agents right now isn’t bad code. It’s the silent stall. In a Show HN this week , a developer running multi-agent workflows named the exact pain: agents “taking unanticipated/destructive actions when it gets stuck.” The foundational thread everyone keeps citing puts it more plainly — the hard problem is “knowing when an agent should hand control back to a human.”

The industry’s answer so far has been: notify a human. Wire up Telegram, wire up Slack, hope you’re looking. That’s better than nothing, but it still puts a person on the critical path for detecting the stall.

Forge went a step further. We shipped a run-level wall-clock liveness watchdog that scans every running agent workflow on a two-minute beat. It fires when a run has made no forward progress for 45 minutes, or when its active turn has been re-leased three or more times — the exact signature of a stuck run. Then it applies a three-tier recovery: first a silent redispatch (most stalls clear themselves on a clean re-attempt); if that doesn’t take, it escalates to operator chat with one-click Retry / Resume / Terminate buttons; and only as a last resort does it fail the run with an alert so nothing hangs forever burning budget.

This closed the third distinct stuck-run failure mode we’d seen in 24 hours — including the five-hour, fifteen-re-lease run that opened this post. The difference is subtle but it’s the whole point: the platform notices and recovers on its own before a human has to. Notification is the fallback, not the mechanism.

2. See what your agent actually did — natively, no instrumentation

The single hottest sub-theme in AI-dev discourse this week is observability, and the complaint is always the same. The author of Orchid , a new record-and-replay debugger, built it because he was “tired of debugging agent failures by grepping through logs.” Vendors are racing to define the category — multiplayer.app on curating agent telemetry, execlave on governance-vs-observability — and the through-line is that you can’t see inside a run without bolting on a proxy or an SDK.

Forge’s Logs tab now streams the agent’s own lifecycle, per turn — start, each iteration, every tool call, the model’s responses, finish and flush — natively, with zero instrumentation on your side. Previously a cloud-exec turn showed one or two worker lifecycle lines because the interesting part (the agent’s own stdout) went through a stream relay and never landed in the turn log. Now a typical three-to-five-minute developer turn surfaces ten or more distinct lifecycle lines, so “what is this agent doing right now” is a tab, not a forensic exercise.

One honest caveat: most of the new debuggers in this wave lean local-first. Forge’s logs are cloud-hosted. The win here isn’t “local” — it’s native and zero-instrumentation: nothing to wire up, no third-party proxy in the path, no SDK to keep in sync.

3. One timeout shouldn’t cost you half a million tokens

Token economics are suddenly loud. The “AI Conundrum” thread this week argued we’re living in a subsidised moment that won’t last, and practitioners now count tokens out loud . So this one stings in a very 2026 way.

Our agent workers post their turn results back to the conductor when a turn finishes. On a flaky connection, a large result (around 165KB) would time out — and every retry re-ran the LLM from scratch. One run burned roughly 558,000 input tokens across four cycles before it landed. A single network hiccup, cascading into half a million wasted tokens.

The fix: large turn payloads now go direct to object storage via a presigned upload, and the worker sends a few-hundred-byte envelope to the conductor, which fetches the body server-to-server. It’s size-gated at 32KB, so small turns keep the fast inline path, and it falls back to inline if storage is briefly unavailable. The flaky leg is out of the LLM’s retry loop entirely. One timeout costs you one retry now — not another full model run.

The takeaway

“The agents can ship features” and “you can leave the agents running unattended” are different claims, and the gap between them is reliability engineering — the unglamorous part. A stall detector that self-recovers, observability you don’t have to instrument, and a payload path that doesn’t re-bill you for a dropped packet: none of these are headline features. They’re the load-bearing plumbing that makes the headline features safe to actually leave alone.

That’s the work this week. Forge runs your agent teams unattended — and now it watches itself, recovers itself, and shows you exactly what it did. Come see how .