Health scoring¶

Each provider gets a score in [0.0, 1.0] recomputed every probe cycle. The router selects providers based on these scores.

Score formula¶

score = w_latency * latency_score
      + w_error   * (1 − error_rate)
      + w_slot    * slot_freshness_score
      + w_success * recent_success_rate

Weights are normalised before use — they don't need to sum to 1.

Component	Default weight	What it measures
`latency_score`	0.4	Round-trip time. ~20ms → near 1.0; ~500ms → near 0.0.
`1 − error_rate`	0.3	Fraction of probes that succeeded in the last `window_secs` (default 60s).
`slot_freshness_score`	0.2	1.0 when at the network tip; decays as drift grows past `slot_drift_threshold` (default 10). Uses the worse of the `processed`/`confirmed` drift (see below).
`recent_success_rate`	0.1	Fraction of the last N probes that succeeded. Fast-reaction signal.

Background probes¶

Health probe (default: every 1s)¶

Each cycle sends three concurrent getSlot requests — one per commitment (processed, confirmed, finalized). It measures round-trip latency (from the processed call), tracks success/failure, updates the rolling error rate and consecutive-failure counter, records the per-commitment slot heights, and triggers circuit breaker state transitions.

Individual requests are used rather than a JSON-RPC batch so the probe stays on the same connection-pool path as live traffic and doesn't depend on provider-side batch handling. On metered plans this is 3 getSlot calls/provider/second (~260k/provider/day) — raise interval_ms to reduce it.

Commitment isolation¶

Health is tracked per commitment, not from processed alone. A provider whose optimistic-confirmation pipeline stalls keeps advancing processed while confirmed freezes — a single processed probe would see a perfectly fresh node while every confirmed read served increasingly stale state.

Drift is measured against a per-commitment network tip (the max slot across providers at that commitment). This matters because finalized trails processed by ~32 slots by design; comparing it to the processed tip would flag every provider.
The slot component of the score uses the worse of processed/confirmed drift, so a stalled confirmed pipeline demotes the provider even when processed looks fresh.
finalized drift is exported but not scored — its lag is expected, so it informs dashboards (getTransaction/getBlock served off a lagging ledger tier) without penalising the score.

Per-commitment slot and drift are surfaced on the /health endpoint (commitments object) and as the rpc_plane_provider_slot_height_commitment / rpc_plane_slot_drift_commitment metrics.

External slot reference¶

The network tip is the max slot across your configured providers — which has a blind spot: if every provider stalls together (a shared upstream incident, a cluster-wide slowdown), the tip slides down with them and drift stays at 0, so the stall is invisible. A single-provider or own-node deploy has the same gap by definition: the one node is the tip, so it can never show drift against itself.

The optional [health] reference_url closes that gap. Set it to an external endpoint (e.g. a public cluster RPC) and RPC Plane probes it for its slot on the same cadence as provider health probes, then folds that slot into the network tip. Now an all-providers-stale stall shows up as every provider drifting behind the reference, and a lone node finally has a checkpoint to measure against.

[health]
reference_url = "https://api.mainnet-beta.solana.com"

Probe-only, never routed. The reference is never added to the routing pool — no client request is ever sent to it. It contributes a slot number and nothing else.
Default-off. Unset by default, so the proxy keeps its promise of zero outbound connections beyond your configured providers. Enabling it is an explicit opt-in to one extra getSlot probe stream.
Safe when it lags or fails. The fold is a per-commitment max, so a reference that falls behind (or stops answering) simply stops pinning the tip — it can never raise a provider's drift falsely. The only misconfiguration that reports false drift is pointing it at the wrong cluster: use a devnet/testnet reference for a devnet/testnet provider set, mainnet for mainnet.
Observability. The reference slot appears on /health as the top-level reference object, and its probe outcomes are counted under rpc_plane_probe_requests_total{type="reference"}.

Circuit breaker¶

Closed ──(N failures or error_rate > threshold)──▶ Open
   ▲                                                │
   └───── probe succeeds ──── HalfOpen ◄────────────┘
                                  (after cooldown_secs)

Closed — normal routing. All requests are eligible.

Open — provider excluded from routing. Opens when either:

Consecutive probe failures ≥ circuit_open_failures (default 5), or
Rolling error rate ≥ circuit_error_threshold (default 0.5)

HalfOpen — after circuit_cooldown_secs (default 30s), one probe is sent. Success closes the circuit; failure reopens it and resets the cooldown.

Note

When all providers have open circuits, the proxy routes to all of them anyway — degraded behaviour is better than refusing to attempt.

What counts as an error

Transport failures, 5xx responses, retryable JSON-RPC errors, and auth failures (401/403 — a revoked or wrong key makes the provider genuinely unusable) all count against health and can open the circuit.

Two categories are treated differently:

Rate limits (429) — the request fails over to the next provider, and the throttled provider's score is demoted so best-score/weighted routing sheds its traffic to peers with headroom. But a 429 is a load signal, not a fault, so it is excluded from the circuit breaker (both the consecutive-failure and error-rate triggers). A rate-limited provider stays eligible and reabsorbs traffic the moment the burst passes — opening its circuit would exclude it for the full cooldown and cascade its load onto everyone else, exactly when you can least afford it.
Client-attributable 4xx (400, 404, 405, 413, 415, 422) — a malformed or unsupported request is the caller's fault, so it is passed through to the client and recorded as client_error without touching provider health at all. This stops one buggy client loop from serially opening every provider's circuit.

Tuning for latency-sensitive workloads¶

For trading bots and latency-critical paths, tighten the probe interval and weight slot freshness heavily:

[health]
interval_ms            = 500    # probe every 500ms
window_secs            = 30     # react faster to errors
slot_drift_threshold   = 3      # aggressively deprioritise stale nodes
circuit_open_failures  = 3      # fail fast
circuit_error_threshold = 0.3   # open at 30% error rate
circuit_cooldown_secs  = 10     # recover quickly

w_latency = 0.3
w_error   = 0.2
w_slot    = 0.4   # slot freshness is critical for transaction landing
w_success = 0.1

See the trading-bot example in the repo.

Checking scores live¶

See Observability — Live status for rpc-plane status output and the /health JSON endpoint.