✨ Soft (metered, non-gating) limits

[sodre]

Problem or Use Case

While landing #455 (consume is the declared scope of a lease), the question came up whether a caller needs a way to declare a limit as adjustable from the lease without it gating admission — e.g. consume={"tpm": None} meaning "I will reconcile tpm post-hoc, but do not reject this call if the tpm bucket is in debt".

Decision for #455: no. After #455 a configured limit is either:

• declared in consume → gates on non-debt via the fast-path tk >= consumed condition, and is adjustable from the lease; or
• omitted → neither.

A per-call third state was rejected because it just shifts the rejection onto the next caller that declares the limit properly — the debt is still there, and whoever names the limit next eats the RateLimitExceeded.

What this issue proposes instead: model "meter but never block" as a property of the limit, not of the call. A soft limit records consumption and debt, emits a signal when overdrawn, but never causes RateLimitExceeded.

Use cases:

• Billing-style token accounting (tpm) alongside a hard rpm gate — count every token, never refuse a request because of the count.
• Shadow-mode rollout of a new limit: configure it soft, watch how often it would have tripped, then flip it hard.
• Per-tenant overage alerting: let the tenant run over, but surface the overage to ops.

Proposed Solution

A tri-state-free boolean soft on a limit's stored config (default false = hard, today's behaviour). Configurable at every level of the ADR-100 hierarchy (system / resource / entity) and via the declarative manifest.

# Sketch — exact API is a design question below
await repo.set_resource_defaults(
    "gpt-4",
    limits=[
        Limit.per_minute("rpm", 500),                 # hard: gates admission
        Limit.per_minute("tpm", 50_000, soft=True),   # soft: metered, never rejects
    ],
)

async with limiter.acquire("user-1", "gpt-4", consume={"rpm": 1, "tpm": 1200}) as lease:
    ...                                    # never raises on tpm, even if tpm bucket is in debt
    await lease.adjust(tpm=actual_tokens)  # still persisted, bucket may go further negative

# Declarative manifest (Issue #405)
resources:
  gpt-4:
    limits:
      rpm:
        capacity: 500
      tpm:
        capacity: 50000
        soft: true

Design questions to settle

1. Where the flag lives. Composite config items already carry l_{name}_cp / l_{name}_ra / l_{name}_rp (ADR-114). Is l_{name}_soft the right shape, and does the bucket item need a denormalized copy (as cascade / parent_id / disabled are) so the speculative path can honour it without a config read?
2. Fast-path condition. Can speculative_consume() simply omit soft limits from the tk >= consumed condition while still ADDing their consumption? If so the feature is 0 extra RCU / 0 extra WCU on the fast path. The wcu infrastructure limit must stay hard.
3. Slow path. try_consume() / _commit_initial() must skip soft limits when deciding admission but still include them in the write set. Confirm build_composite_retry's tk >= consumed condition is only applied to hard limits.
4. Reporting. RateLimitExceeded.passed / LimitStatus — how does a soft limit in debt appear? Options: exceeded=False plus a new soft=True field on LimitStatus; or a separate overdrawn list on the exception / lease. The retry_after_seconds bottleneck computation must ignore soft limits.
5. Overdraw signal. Which is the primary observable: an AuditEvent (new AuditAction), a CloudWatch metric emitted by the aggregator from the stream, a flag on the usage snapshot, or several? The aggregator already sees every bucket MODIFY; a tk < 0 transition on a soft limit is cheap to detect there.
6. Cascade parents. Is soft resolved per (entity, resource) like disabled (ADR-125), so a child's tpm can be soft while the parent's tpm is hard? Or must the flag agree along the cascade chain? Decide and document, mirroring the ADR-125 note that carve-outs do not extend to parents.
7. Config resolution. Precedence follows the existing walk (entity(resource) > entity(_default_) > resource > system). Does the flag resolve independently per level like disabled, or travel with the limit definition it is attached to?

Cost expectations

Path	Today (hard)	Soft limit target
Speculative success	0 RCU + 1 WCU	0 RCU + 1 WCU (condition shrinks, ADD set unchanged)
Speculative fast rejection on a soft limit	0 RCU + 0 WCU	n/a — must not reject
Slow path	1 RCU + 1 WCU	unchanged
Overdraw signal	—	aggregator-side, 0 client cost

Out of Scope

• Per-call None in consume (rejected in 🐛 Make consume the declared scope of a lease; stop silently dropping adjust() keys #455 — see above).
• System-level disabled-style semantics; this is about gating, not availability.

Alternatives Considered

• Per-call consume={"tpm": None} — rejected in 🐛 Make consume the declared scope of a lease; stop silently dropping adjust() keys #455: the debt still lands on the next properly-declared call, so it is not "non-gating", it is "gate someone else".
• Configure tpm with a huge capacity — works today but loses the overdraw signal entirely and makes RateLimitExceeded.passed meaningless for that limit.
• Track tpm outside the limiter — duplicates the bucket write for billing and forfeits usage snapshots / audit integration.

Acceptance Criteria

• A limit can be marked soft at system, resource, and entity level through the existing set_system_defaults / set_resource_defaults / set_limits APIs and their CLI counterparts (-l flag or equivalent)
• A limit can be marked soft in the declarative YAML manifest and round-trips through the generated Custom::ZaeLimiterLimits CloudFormation resource
• acquire() never raises RateLimitExceeded because of a soft limit, on both the speculative and slow paths, and on cascade parents — covered by unit tests in tests/unit/test_limiter.py (sync counterpart generated)
• Initial consumption and lease.adjust() / consume() / release() against a soft limit are persisted to the bucket item and appear in usage snapshots — covered by an integration test
• Speculative-path success on a soft limit costs 0 RCU + 1 WCU (no additional round trip) — verified with capacity_counter in tests/benchmark/test_capacity.py
• An overdrawn soft limit (bucket tk < 0) is observable via at least one of: AuditEvent, CloudWatch metric, or usage-snapshot field — with a test asserting the signal is emitted
• RateLimitExceeded / LimitStatus expose a soft limit's in-debt state without listing it in violations, and retry_after_seconds ignores soft limits — unit test
• docs/guide/basic-usage.md contains a "Hard vs soft limits" section; docs/api/ and docs/cli.md document the flag; CLAUDE.md config section updated
• ADR written for the flag's storage location and fast-path semantics, referencing ADR-100, ADR-114, and ADR-125

Related

• 🐛 Make consume the declared scope of a lease; stop silently dropping adjust() keys #455 — consume is the declared scope of a lease (motivating discussion)
• 🐛 Speculative fast path drops LeaseEntry for limits estimated at 0 #453 — speculative fast path drops LeaseEntry for zero estimates
• ADR-100 (centralized config), ADR-114 (composite config items), ADR-125 (resource disable — precedent for a per-level flag beside limits)