Observability¶
RPC Plane exposes Prometheus metrics and a health endpoint out of the box. No configuration needed.
Prometheus metrics¶
Metrics are served on :9401/metrics (configurable via server.metrics_listen).
Metrics reference¶
| Metric | Type | Labels | Description |
|---|---|---|---|
rpc_plane_requests_total |
Counter | method, provider, status |
Total JSON-RPC calls handled. A batch request increments by its call count, so a 1000-call batch counts as 1000. |
rpc_plane_request_duration_seconds |
Histogram | method, provider |
Request latency. One observation per request (a batch records its single round-trip latency once). |
rpc_plane_provider_health_score |
Gauge | provider |
Current health score (0.0–1.0) |
rpc_plane_provider_slot_height |
Gauge | provider |
Last observed processed slot height |
rpc_plane_slot_drift |
Gauge | provider |
Slots behind the network tip (worst of processed/confirmed) |
rpc_plane_provider_slot_height_commitment |
Gauge | provider, commitment |
Slot height per commitment (processed/confirmed/finalized) |
rpc_plane_slot_drift_commitment |
Gauge | provider, commitment |
Slots behind the per-commitment tip |
rpc_plane_circuit_breaker_state |
Gauge | provider |
0=closed, 1=half-open, 2=open |
rpc_plane_failover_total |
Counter | from_provider, to_provider |
Failover events |
rpc_plane_rate_limited_total |
Counter | provider |
Requests shed from a provider because its max_rps token bucket was empty. A rising rate means the provider is regularly at its configured cap. |
Batch request labels
For a JSON-RPC batch, the method label is normalized to keep cardinality bounded: distinct method names are deduplicated and capped (up to 5, with a +N suffix for the rest). A homogeneous batch collapses to the bare method name (e.g. a batch of 1000 getTransaction is labeled getTransaction), so it groups with single calls — while the counter still reflects all 1000 calls.
The status label
rpc_plane_requests_total uses four status values:
ok— 2xx response with no JSON-RPC error body.error— a provider-attributable failure: a transport error, a5xx, a retryable JSON-RPC error, or an auth failure (401/403). These count against the provider's health score and circuit breaker.rate_limited— an HTTP429. The request fails over to the next provider, and the throttled provider's score is demoted (so traffic sheds to peers) but its circuit is not opened — a rate limit means the provider is capped, not broken. See health scoring.client_error— a client-attributable 4xx (400,404,405,413,415,422): a malformed body, unknown route, or unsupported payload. The status is passed through to the caller, but it is not counted against provider health — a buggy client loop can't open every provider's circuit.
Only error feeds provider error-rate panels; rate_limited and client_error are tracked separately so a throttle or a caller bug doesn't read as a provider fault. (rpc_plane_probe_requests_total gains a matching rate_limited status when a health probe is itself throttled.)
Prometheus scrape config¶
# prometheus.yml
scrape_configs:
- job_name: rpc-plane
static_configs:
- targets: ["localhost:9401"]
For Prometheus Operator (Kubernetes):
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: rpc-plane
spec:
endpoints:
- port: metrics
interval: 15s
selector:
matchLabels:
app: rpc-plane
Health endpoint¶
Returns a JSON snapshot of each provider's current state: health score, slot height, slot drift, circuit state, latency, availability, plus a per-commitment commitments object:
{
"name": "provider-a",
"score": 0.912,
"slot": 341892471,
"slot_drift": 0,
"is_drifting": false,
"circuit": "closed",
"available": true,
"rate_limited": false,
"commitments": {
"processed": { "slot": 341892471, "drift": 0, "is_drifting": false },
"confirmed": { "slot": 341892469, "drift": 2, "is_drifting": false },
"finalized": { "slot": 341892439, "drift": 0, "is_drifting": false }
}
}
finalized drift is measured against the finalized tip (not processed), so a healthy provider shows ~0 finalized drift despite trailing the processed tip by ~32 slots.
available is false when the provider is out of rotation — either its circuit is open (broken) or it is currently at its max_rps cap. rate_limited distinguishes the second case: true means the provider is healthy but at its configured per-second cap, so new requests are shedding to peers.
When an external slot reference is configured ([health] reference_url), the response also carries a top-level reference object with the reference's last observed slot per commitment — the checkpoint the tips are pinned to. It is null when no reference is configured. Reference probe outcomes are counted under rpc_plane_probe_requests_total{type="reference"}, so you can alert on the reference itself going unreachable.
Live status CLI¶
rpc-plane status
# NAME SCORE SLOT DRIFT LATENCY CIRCUIT
# ---------- ------- ------------ ------ ---------- -------
# provider-a 0.912 341892471 0 23.4ms closed
# provider-b 0.841 341892469 2 31.1ms closed
# provider-c 0.724 341892468 3 38.7ms closed
The proxy must be running. rpc-plane status reads from the health endpoint.
Grafana dashboard¶
The repository ships grafana/dashboard.json — import it directly into any Grafana instance.
Import steps¶
- In Grafana, go to Dashboards → Import.
- Click Upload JSON file and select
grafana/dashboard.jsonfrom the release archive or the repo. - Select your Prometheus datasource.
- Click Import.
Dashboard panels¶
| Panel | Description |
|---|---|
| Provider health score | Score over time per provider (multi-line chart) |
| Slot height | Absolute slot height per provider |
| Slot drift | Slots behind network tip — gauge and line chart |
| Circuit breaker state | Current state per provider (table) |
| Request rate by method | Top 10 methods by request rate |
| Latency p50/p95/p99 | Percentile breakdown over time |
| Failovers | Failover events as timeline annotations |
| Error rate | Per-provider error rate |
| Period totals | Stat cards — today / 7d / 30d (follows dashboard time picker) |