Spanda

Recovery Playbooks

Recovery playbooks are versioned, reusable multi-step recovery workflows managed by the Recovery Orchestrator.

Built-in playbooks

Playbook	Trigger	Steps
`battery_low`	battery	Navigate to charger → assign backup → transfer mission → resume
`sensor_failure`	sensor	Retry → reinitialize → switch redundant sensor → degraded mode
`connectivity_loss`	connectivity	Retry → reconnect → switch gateway
`fleet_member_loss`	fleet	Detect → redistribute tasks → reassign missions
`mission_transfer`	mission	Pause → select backup → takeover with state transfer

TOML configuration

Add custom playbooks in spanda.toml:

[recovery.playbooks]
[[recovery.playbooks]]
name = "custom_overheat"
version = "1.0.0"
description = "Thermal emergency playbook"
trigger = "temperature"
entity_kinds = ["robot"]

[[recovery.playbooks.steps]]
order = 1
description = "Reduce workload"
strategy = "graceful_degradation"
escalation_level = 1
timeout_secs = 30

[[recovery.playbooks.steps]]
order = 2
description = "Safe shutdown if temperature critical"
strategy = "safe_shutdown"
escalation_level = 8
timeout_secs = 15
requires_validation = true

CLI

spanda recovery playbooks --config examples/packages/basic_project
spanda recovery plan rover.sd --failure battery --playbook battery_low

REST API

GET /v1/recovery/playbooks

SDK

client.list_recovery_playbooks()?;

client.list_recovery_playbooks()

await client.listRecoveryPlaybooks();

Plugin playbooks

Plugins declare recovery extensions in spanda.plugin.toml:

[[recovery.extensions]]
kind = "playbook"
name = "custom_sensor_reset"
description = "Reset custom sensor stack"
trigger = "sensor_failure"

[[recovery.extensions]]
kind = "strategy"
name = "vendor_retry"
description = "Vendor-specific retry loop"

Enabled plugins with on_recovery_completed hooks receive payloads after orchestrator execute. Example plugin: examples/plugins/recovery-plugin/ (playbook, strategy, validator extensions with trigger and strategy fields). See plugin-api.md.