Primary runtime
Active
Disaster Recovery and Resilience - bp Sphere
Failure scenarios, degraded-mode operations, RTO/RPO posture, replay-assisted recovery, and dependency fallback.
ResilienceRTO / RPOReplay-assisted recovery
Disaster Recovery and Resilience Center
This surface answers what happens when runtime, identity, model, source-system, or data-platform dependencies fail. The operating rule is safe degradation: preserve evidence, stop unsafe write-back, keep humans informed, and recover from replayable state.
Secondary runtime
Warm standby
RTO
10 min design
RPO
Replay ledger
Resilience design
Resilience is not a dashboard claim; it is a set of runtime behaviors that keep financial decisions safe under partial failure.
| Runtime failure | Pause autonomous execution, route open decisions to human queue, restore from replay and event ledger. |
|---|---|
| Identity failure | Deny new privileged actions, preserve read-only sessions where safe, require break-glass workflow. |
| Source-system failure | Use last-good context with freshness warning; block write-back until adapter recovers. |
| Databricks failure | Fallback to compact profile / cached aggregates; mark data confidence and escalation required. |
| Model failure | Route to fallback model or deterministic policy path; require human review for high-impact decisions. |
| Evidence failure | Mark evidence pack incomplete; do not approve action until required evidence is restored or waived. |
| Policy service failure | Fail closed for write-back; allow read-only explanation with explicit policy-unavailable status. |
| Security event | Downgrade autonomy, quarantine suspect run, notify operator, preserve replay package. |
Recovery metrics and controls
These are the resilience metrics and controls a technology reviewer should expect to inspect during the workshop.
| RTO | Target recovery time by dependency: model 2 min, adapter 10 min, runtime 10 min, data plane 30 min. |
|---|---|
| RPO | Decision replay ledger and source snapshots define the maximum acceptable recovery point. |
| Failover | Primary-to-secondary runtime design with warm standby and replay rehydration. |
| Backlog handling | Decision queue, approval queue, DLQ, retry count, and SLA pressure remain visible. |
| Kill switch | Autonomy downgrade and connector pause controls stop unsafe execution without hiding state. |
| Recovery replay | Rebuild signal, context, evidence, policy, agent outputs, human action, and outcome. |
| Incident ownership | Named runtime owner, data owner, policy owner, and business supervisor path. |
| Post-incident learning | Lessons feed resilience certification, policy tuning, and promotion gates. |
Failure scenario matrix
| Failure | Detection | Safe behavior | Recovery proof |
|---|---|---|---|
| Runtime pod fails | Health probe + queue heartbeat | Shift to standby; pause write-back | Replay ledger rehydrates open decisions |
| ERP adapter timeout | Adapter latency + stale read | Use compact replica for context; block external action | Adapter drawer shows last sync and retry path |
| Model unavailable | Model router failure | Fallback model or deterministic rule; human approval required | Model routing center records fallback reason |
| Evidence store degraded | Evidence completeness drops | Decision can be reviewed, but action cannot execute | Evidence pack marks missing sources |
| Security incident | SOC event / injection guard | Quarantine run and downgrade autonomy | Replay captures blocked tool call |
Workshop proof sequence
- Open Enterprise Resilience to show kill switch, failure simulations, dependency map, and survivability controls.
- Open Runtime Command Center to show queues, failures, throughput, and recovery posture.
- Open Replay Center to prove a decision can be reconstructed after recovery.
- Open Integration Fabric to show source-system fallback behavior and write-back hold.