Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions changelog.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,28 @@ export const STAGE_SELF_MANAGED_M66 = "$0.0001";
No API shapes, endpoints, or error codes changed.
</Update>

<Update label="Jul 10, 2026" tags={["API", "Bug fixes"]}>
## Per-fleet spend ceilings are enforced

`daily_dollars` is required in every `TRIGGER.md` and has been documented as a hard spend ceiling since it shipped. It was read by nothing: a fleet stuck in a tool-call loop drew on the tenant credit pool until the pool emptied or the lease's runtime cap fired. The ceiling now holds.

## What changed

- **`daily_dollars` and `monthly_dollars` now stop a fleet** — a fleet that has spent its allowance is refused before its next run opens, recording `gate_blocked` with a `budget_breach` label. The refused event is **not** charged. A fleet that crosses its ceiling mid-run is stopped at its next lease renewal (`UZ-RUN-015`) and records `fleet_error` with the same label.
- **`budget_breach` is a distinct label** — previously every mid-run stop reported `renewal_terminate`, so a fleet hitting its own ceiling looked identical to a tenant running out of credit. The two are now distinguishable on the event row.
- **Spend windows** — `daily_dollars` is a rolling 24 hours, `monthly_dollars` the UTC calendar month. Spend counts what you were actually charged. `monthly_dollars` remains optional; omit it and only the daily ceiling applies.
- **Lowering a ceiling takes effect immediately**, including on a run already in flight.
- **`/v1/admin/models` is documented** — the platform-admin model catalogue routes (`GET`/`POST` on the collection, `PATCH`/`DELETE` on `{uid}`) have been served for a milestone without appearing in the API reference. A continuous-integration gate now fails whenever a served route is missing from the specification.

## Documentation corrections

- **Continuation chains never existed.** The context-lifecycle page described the runtime re-queuing an exhausted run as a continuation, a chain capping at 10, and an 11th attempt labelled `chunk_chain_escalate_human`. None of that was ever built. The runtime observes how full the context window is and writes a log line; it cannot interrupt a fleet mid-loop. A fleet that runs out of context wraps up, the run ends `processed`, and nothing re-queues it. To carry work forward, trigger the fleet again — it recovers its state from memory.
- **`actor=continuation:<original_actor>` is not a value the runtime writes.** It has been removed from the activity-stream and `--actor` filter documentation.
- **`budget_breach` was documented before it existed.** It exists now, and the pages describing it are accurate.

<Note>Every run is charged `$0` during the free-trial window, so no fleet accrues spend and no budget can be reached. The per-fleet ceiling and the tenant credit gate both begin to bite when the window closes.</Note>
</Update>

<Update label="Jul 10, 2026" tags={["Bug fixes", "UI"]}>
## Fleet timeline recovers frames lost to a connection blip

Expand Down
2 changes: 1 addition & 1 deletion cli/agentsfleet.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@ agentsfleet events 0198a7b2-9e1f-7c3a-8b25-6d4f0a9e2c71 --json | jq 'select(.err

| Flag | Default | Purpose |
|------|---------|---------|
| `--actor <glob>` | *(none)* | Filter by actor — `webhook:*`, `cron:*`, `steer:*`, `continuation:*`. |
| `--actor <glob>` | *(none)* | Filter by actor — `webhook:*`, `cron:*`, `steer:*`. |
| `--since <duration>` | `24h` | Time window — Go-style duration (`2h`, `7d`) or RFC 3339 timestamp. |
| `--cursor <cursor>` | *(none)* | Pagination cursor from a previous response. |
| `--limit <n>` | `50` | Max events per page. |
Expand Down
50 changes: 36 additions & 14 deletions concepts/context-lifecycle.mdx
Original file line number Diff line number Diff line change
@@ -1,37 +1,55 @@
---
title: "How long does a run take?"
description: "When a run opens, when it stops, and the three knobs you'd ever tune."
description: "When a run opens, when it stops, and the knobs you'd ever tune."
---

A single trigger — a webhook, a cron fire, a `steer` message — opens one **run**. The fleet reasons, calls tools, and the run closes when it's done or when it would otherwise overflow the model's context window.
A single trigger — a webhook, a cron fire, a `steer` message — opens one **run**. The fleet reasons, calls tools, and the run closes when it's done or when something stops it.

Most fleets finish a run in seconds to a couple of minutes and never need any of the controls on this page. Read on if you have a long incident that issues dozens of tool calls and want to understand what stops it.

## The three things that end a run
## The three ways a run ends

| What | When it fires | What happens |
| What | When it fires | What the event records |
|---|---|---|
| **Run finished** | Fleet decides it's done | Run closes cleanly. Diagnosis posted, state checkpointed. |
| **Context full** | Prompt approaches the model's context cap | Fleet ends the run with a checkpoint, the runtime opens a fresh run on the same chain, the new run `memory_recall`s where it left off. |
| **Budget breach** | Daily or monthly dollar cap hit | Run terminates, `budget_breach` recorded. Future runs in the next billing window are unaffected. |
| **Run finished** | Fleet decides it's done | `processed`. Diagnosis posted, session state checkpointed. |
| **The fleet is stopped** | Wall-clock deadline, sandbox limits, or a control-plane policy stop | `fleet_error`, with a label naming the cause. |
| **Refused before it starts** | A gate — budget, credit, or approval — says no | `gate_blocked`, with a label naming the gate. |

## Continuation chains — when one run isn't enough
## What happens when the context fills up

If the fleet can't finish in one run (a forty-tool-call incident on a 200K-token model, for example), it ends the current run with `exit_ok: false` and a `checkpoint_id`. The runtime re-enqueues the same event as a continuation; the next run opens with a fresh prompt and pulls the prior findings via `memory_recall(checkpoint_id)`.
The runtime watches how full the model's context window is. After every model round-trip it compares the prompt size against `context_cap_tokens`, and once the ratio crosses `stage_chunk_threshold` it writes a `chunk_threshold_breached` line to the run log.

A chain caps at **10 continuations**. The 11th attempt is refused, the originating event is labelled `chunk_chain_escalate_human`, and an operator picks it up. The fleet itself stays alive — only that one chain forfeits.
**That is all it does.** The runtime cannot interrupt a fleet mid-loop, so it cannot force one to wrap up — it can only tell you the window is filling. The fleet itself decides what to do, guided by your `SKILL.md` prose: consolidate the findings into memory, then finish the run.

When the fleet does that, the run ends normally: the event is recorded `processed` and the session state is checkpointed. **Nothing is automatically re-queued.** There is no continuation chain, no chunk counter, and no cap on how many times a fleet may pick a problem back up. To continue the work, send another trigger — a fresh `steer`, a webhook, or a cron fire — and the next run pulls the prior findings out of [memory](/memory).

<Note>
Because the runtime only observes, a fleet that ignores its `SKILL.md` guidance and keeps calling tools will run until one of the real ceilings below stops it. The context threshold is a signal, not a brake.
</Note>

## Knobs you can tune

Set these under `x-agentsfleet.context` in `TRIGGER.md`. Defaults work for almost every workload — leave them alone unless you have a specific reason.

| Knob | Default | When to touch it |
|---|---|---|
| `tool_window` | 20 | Raise if your fleet routinely runs 30+ tool calls and you want fewer continuations. Lower if you want it to compact its findings sooner. |
| `tool_window` | 20 | Raise if your fleet routinely runs 30+ tool calls before consolidating. Lower to make it compact its findings sooner. |
| `memory_checkpoint_every` | 5 | Cadence at which the fleet is prompted to write a memory snapshot during a long run. Rarely worth changing. |
| `context_cap_tokens` | model-dependent | Set explicitly if your model has no published context cap or differs from the default. |
| `stage_chunk_threshold` | disabled | Fraction of the context cap (`0.0`–`1.0`) at which the fleet is told to snapshot and wrap up. `0` turns the check off. |
| `context_cap_tokens` | model-dependent | Set explicitly if your model has no published context cap, or differs from the default. |

## What actually stops a run

Every ceiling below is enforced by the platform, not by the fleet's own good behaviour.

If you've never seen a `chunk_chain_escalate_human` failure label, you don't need to change any of these.
| Ceiling | Where it comes from | What the event records |
|---|---|---|
| **Wall-clock deadline** | the run's lease | `fleet_error` / `timeout_kill` |
| **Per-fleet budget** | `daily_dollars` / `monthly_dollars` in `TRIGGER.md` | `gate_blocked` / `budget_breach` before the run; `fleet_error` / `budget_breach` when a running fleet crosses its ceiling |
| **Tenant credit** | your credit balance | `gate_blocked` / `balance_exhausted` before the run; `fleet_error` / `renewal_terminate` mid-run |
| **Memory and process limits** | the sandbox | `fleet_error` / `oom_kill` or `resource_kill` |

A crash or a dropped connection also ends a run, labelled `runner_crash` or `transport_loss`.

## Common questions

Expand All @@ -41,7 +59,11 @@ Yes. Runs can be seconds (a single webhook → Slack post) to a few minutes (a m

**Will it run forever?**

No. Three independent ceilings stop it: the run's own context cap (the fleet voluntarily ends and continues), the continuation cap (10 chunks max), and the budget cap (`daily_dollars` / `monthly_dollars` in `TRIGGER.md`).
No. The wall-clock deadline ends it regardless of what it is doing, and the sandbox's memory and process limits kill it before it can exhaust the host. Independently, your fleet's `daily_dollars` bounds what it can spend in a rolling day — see [Budgets](/billing/budgets).

**Will it re-run itself?**

No. A run ends and stays ended. Only a new trigger opens a new run. Long work is picked back up by triggering the fleet again; it recovers what it learned from [memory](/memory).

**Do I need to tune anything?**

Expand Down
14 changes: 10 additions & 4 deletions fleets/authoring.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ triggers:

The App installation belongs to the workspace. The `repositories` list belongs to this fleet. It narrows the repositories selected during App installation and prevents unrelated fleets in the same workspace from waking.

<Note>**Long-run continuations are not a trigger type.** When a run exhausts its context, the runtime re-enqueues the same event with `actor=continuation:<original_actor>` automatically. You don't declare this in `TRIGGER.md`; it happens regardless of which trigger type woke the original run. See [Context lifecycle](/concepts/context-lifecycle).</Note>
<Note>**A run is never continued automatically.** When a fleet runs out of context it wraps up and the run ends; nothing re-queues it. To carry long work forward, trigger the fleet again — the next run recovers its state from [memory](/memory). See [Context lifecycle](/concepts/context-lifecycle).</Note>

### Tools

Expand All @@ -184,10 +184,16 @@ Each entry under `credentials:` is a **bare vault key name** referencing a JSON-

| Field | Purpose |
|---|---|
| `daily_dollars` | Rolling 24-hour dollar ceiling for this fleet. |
| `monthly_dollars` | Calendar-month dollar ceiling. |
| `daily_dollars` | Rolling 24-hour dollar ceiling for this fleet. Required; must be greater than `0` and at most `1000`. |
| `monthly_dollars` | Calendar-month (UTC) dollar ceiling. Optional — omit it and only the daily ceiling applies. At most `10000`. |

A breach terminates the in-flight event cleanly and records a `budget_breach` entry. Subsequent events in a later billing window are unaffected. Workspace-wide budgets live in [Billing](/billing/budgets).
This is your fleet's blast-radius guard, and it is separate from your tenant's credit balance: a runaway fleet stops at its own `daily_dollars` long before it can drain the pool every other fleet is drawing from.

The ceiling is checked twice. Before a run starts, a fleet that has already spent its allowance is refused — the event is recorded `gate_blocked` with a `budget_breach` label, and it is **not** charged. If a fleet crosses its ceiling while a run is in flight, the run is stopped at its next renewal and the event is recorded `fleet_error` with the same `budget_breach` label. Spend counts what you were actually charged, over a rolling 24 hours for `daily_dollars` and over the current UTC calendar month for `monthly_dollars`.

<Note>During the free-trial window every run is charged `$0`, so no fleet accrues spend and no budget can be reached. Both this ceiling and the tenant credit gate begin to bite when the window closes.</Note>

Workspace-wide budgets live in [Billing](/billing/budgets).

## Common authoring mistakes

Expand Down
2 changes: 1 addition & 1 deletion fleets/running.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ agentsfleet status --json

## `agentsfleet logs <fleet_id>`

Print the fleet's recent activity stream. Every webhook receipt, cron fire, steer message, tool call, and run exit lands here, tagged with what triggered it (`webhook:github`, `cron:*`, `steer:<user>`, `continuation:<original_actor>`).
Print the fleet's recent activity stream. Every webhook receipt, cron fire, steer message, tool call, and run exit lands here, tagged with what triggered it (`webhook:github`, `cron:*`, `steer:<user>`).

```bash
agentsfleet logs 0198a7b2-9e1f-7c3a-8b25-6d4f0a9e2c71
Expand Down
17 changes: 12 additions & 5 deletions fleets/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ A fleet that's misbehaving falls into one of six failure shapes. Find the sympto
| Upstream says it sent the webhook, fleet didn't wake | Signature mismatch or wrong URL | [§2 Webhook delivered but fleet silent](#2-webhook-delivered-but-fleet-silent) |
| Fleet woke, run failed before producing output | Secret, network, or tool error | [§3 Run opens then fails](#3-run-opens-then-fails) |
| Run produces a result but it's wrong | `SKILL.md` prose or memory issue | [§4 Run completes but output is wrong](#4-run-completes-but-output-is-wrong) |
| Run terminates with `budget_breach` | `daily_dollars` / `monthly_dollars` ceiling hit | [§5 Budget breach](#5-budget-breach) |
| Run refused or terminated with `budget_breach` | `daily_dollars` / `monthly_dollars` ceiling hit | [§5 Budget breach](#5-budget-breach) |
| `agentsfleet` itself is broken (auth, network) | Local config or platform reachability | [§6 agentsfleet can't reach the platform](#6-agentsfleet-cant-reach-the-platform) |

## 1. Frontmatter rejected
Expand Down Expand Up @@ -142,16 +142,23 @@ Edit `SKILL.md`, `agentsfleet fleet update <fleet_id> --from .`, re-steer. The n

## 5. Budget breach

The run terminated on a budget breach — the `daily_dollars` or `monthly_dollars` ceiling in `TRIGGER.md` was hit. The activity stream records the terminated run (there's no separate wire code).
The fleet hit the `daily_dollars` or `monthly_dollars` ceiling in its `TRIGGER.md`. The event carries a `budget_breach` label, in one of two shapes:

| Event status | What happened |
|---|---|
| `gate_blocked` | The fleet had already spent its allowance, so the run never started. **The event was not charged.** |
| `fleet_error` | The fleet crossed its ceiling mid-run; the run was stopped at its next lease renewal (wire code `UZ-RUN-015`). |
Comment thread
greptile-apps[bot] marked this conversation as resolved.

```bash
agentsfleet status --json | jq '.[] | select(.fleet_id == "<id>") | {state, budget}'
```

Two layers of cap exist independently:
Two caps exist independently, and both must pass:

1. **Per-fleet** (`TRIGGER.md` `budget.daily_dollars` / `monthly_dollars`) — refuses this fleet's runs until the rolling 24-hour window slides past the spend, or until the UTC calendar month turns over. Raise the value in `TRIGGER.md`, then push it with `agentsfleet fleet update <fleet_id> --from .`. A lowered ceiling takes effect immediately, including on a run already in flight.
2. **Tenant wallet** — every workspace's runs debit the same pool. When it hits zero, all runs across all workspaces stop, labelled `balance_exhausted` (before a run) or `renewal_terminate` (mid-run). Top up at `app.agentsfleet.net/billing`. See [Billing → Budgets](/billing/budgets).

1. **Per-fleet** (`TRIGGER.md` `budget.daily_dollars` / `monthly_dollars`) — terminates this fleet's runs until the rolling window resets. Raise the value in `TRIGGER.md`, then push it with `agentsfleet fleet update <fleet_id> --from .`.
2. **Tenant wallet** — every workspace's runs debit the same pool. When it hits zero, all runs across all workspaces stop. Top up at `app.agentsfleet.net/billing`. See [Billing → Budgets](/billing/budgets).
The labels are distinct on purpose: `budget_breach` means your own ceiling held, `balance_exhausted` and `renewal_terminate` mean the tenant ran out of credit.

If you're hitting the per-fleet cap repeatedly on legitimate traffic, the fleet's per-run cost is higher than you sized for. Audit the activity stream for unusually long runs (high `tokens` or `wall_ms` fields), and consider tightening `SKILL.md` or splitting the work across multiple smaller fleets.

Expand Down
Loading