MCP database server circuit breakers: shed load before retries become an outage
A slow database rarely fails alone.
Queries take longer, connection pools fill, agent requests hit deadlines, clients retry, and the retry traffic consumes the capacity the database needs to recover. The system turns one degraded dependency into a self-sustaining outage.
An MCP database server needs admission control, bulkheads, circuit breakers, and explicit load shedding before that loop begins.
Protect the database at admission time
The cheapest query is the one the server rejects before it acquires a database connection. Check authenticated identity, tenant, tool permission, request size, time window, query class, per-user and per-tenant budget, and global capacity before work enters the pool.
Use separate limits for interactive questions, scheduled reports, background indexing, and administrative workflows. One large tenant or automation loop should not consume every connection available to unrelated users.
Rate limits control arrival volume. They should complement the rate-limit model for MCP database servers, not replace database-aware capacity checks.
Bound every queue
An unbounded queue hides overload while increasing memory use and user latency. By the time queued work reaches the database, the caller may already have timed out and retried.
Set a maximum queue length and maximum queue age for each workload class. Reject expired work before execution. Propagate the caller deadline so a query cannot continue consuming capacity after its result is no longer useful.
Expose queue depth, oldest age, admitted and shed requests, pool wait, active queries, cancellation latency, and retry volume as separate metrics.
Use bulkheads for different failure domains
A single shared pool lets one expensive workflow starve everything else. Partition capacity by risk and purpose:
- small interactive reads;
- bounded analytical reads;
- scheduled jobs;
- metadata and health operations;
- mutations, when the product supports them.
Reserve enough capacity for health checks, cancellation, reconciliation, and operator access. A system that uses every connection for user work may be unable to diagnose or recover itself.
For pool math and backpressure, see MCP database connection pool sizing.
Define what opens the circuit
A circuit breaker stops sending a class of work to a dependency after evidence shows that more attempts are harmful. Do not open it on every error. Validation and authorization failures say nothing about database health.
Useful signals include:
- pool acquisition timeout rate;
- database connection and failover errors;
- statement timeout and cancellation backlog;
- replica lag beyond the workflow's freshness contract;
- p95 or p99 latency above a sustained threshold;
- resource saturation from the database or gateway;
- a rapid increase in retry-amplified attempts per operation.
Evaluate signals per dependency, database, tenant, and workload class where practical. A reporting replica failure should not automatically disable a separate metadata operation against another service.
Move through closed, open, and half-open deliberately
In the closed state, requests flow under normal admission policy. In the open state, matching work fails quickly or receives a safe fallback without touching the dependency. After a cooldown, the half-open state admits a small number of controlled probes.
Half-open traffic should be bounded and representative. Do not release the entire retry backlog as the recovery probe. Require several successful observations across a minimum interval before closing the circuit, and reopen quickly if they fail.
Add jitter to cooldowns across instances so every server does not probe the recovering database at once.
Shed work by value and safety
Load shedding needs an explicit order. Drop expired requests first, then duplicate or speculative work, low-priority background refreshes, and broad analytical queries. Preserve bounded interactive reads, cancellation, reconciliation, and operator health operations according to the incident policy.
Do not return stale cached data without saying so. If a fallback answer is allowed, include source, observed time, freshness window, policy decision, and whether the result is partial or stale. Some workflows should fail rather than use old data.
Return an error the agent cannot misread
A shed request should produce a stable machine-readable response: error code, category, retryable flag, retry-after guidance, outcome certainty, operation ID, and trace ID. The model should not infer from prose whether immediate retry is safe.
For example, CAPACITY_SHED may be retryable after a server-controlled delay for an idempotent read, while DEADLINE_EXPIRED should ask the user to narrow or rerun deliberately. An unknown mutation outcome must not be placed into the same retry bucket.
Use the MCP database tool error contract for the wider taxonomy.
Stop retry amplification
Count logical operations separately from attempts. Scope retry budgets across the complete request path so the MCP client, gateway, tool wrapper, database driver, and job scheduler cannot each perform independent retries.
Honor server retry guidance, use exponential backoff with jitter, cap attempts inside the user deadline, and avoid retrying non-idempotent work without an execution receipt. A circuit breaker that rejects quickly but triggers an immediate client loop has not reduced load.
Concrete example: reporting replica degradation
A reporting replica slows during a storage incident. Pool acquisition p95 rises, queue age exceeds the interactive deadline, and clients begin retrying.
The server stops admitting broad analytical queries, opens the circuit for scheduled reports, and preserves a small pool for bounded interactive reads and operator probes. Shed responses include a randomized retry window. Expired queued work is removed instead of reaching the replica late.
Half-open probes use one approved lightweight query with a strict timeout. The circuit closes only after replica lag, acquisition latency, and query success remain inside policy for several observations. The retry backlog is released gradually.
Test the breaker before production needs it
- Slow and exhaust the pool while sending mixed workload classes.
- Verify queues remain bounded and expired work never executes.
- Trigger one tenant's budget without affecting another tenant.
- Open the circuit and confirm no matching database calls escape.
- Run half-open probes from several server instances with jitter.
- Simulate client, gateway, and driver retries together.
- Check fallback freshness and partial-result labels.
- Confirm traces distinguish user operations, attempts, shed work, and probes.
Where Conexor fits
Conexor provides MCP infrastructure for connecting AI clients to databases and APIs through governed tools. Production access needs bounded admission, workload isolation, typed failures, and evidence that helps operators reduce pressure before retries turn degradation into an outage.
For the broader control model, see Conexor security and trust.