Skip to content
Computese home
Menu
Financial services07Cloud10AI & automation

Automating a financial firm’s disaster recovery drills

A financial firm proved its recovery with a long manual checklist run over phone calls and spreadsheets. We automated the drill end to end on Jenkins and our AI automation platform, kept a named person approving every step, and made the audit evidence assemble itself.

Industry
Financial services
Engagement
Pilot and reference architecture
Scope
One critical application, one disaster scenario
AI
Advises, never acts
Automation
Jenkins and our AI automation platform
Mode
Simulation, no production impact

Where it started.

Regulated financial firms must show regularly that critical applications can be recovered at a second site. Here that meant dozens of manual steps across database, application, infrastructure and business teams, under strict technology-risk regulation.

  • Warning signs were spread across several monitoring tools, so declaring a disaster was slow.
  • Coordination ran on calls, chat and spreadsheets, with no live view of progress for the people making go or no-go decisions.
  • Results depended on who was on shift, so recovery time was hard to predict.
  • Logs and evidence were scattered, and the audit bundle was assembled by hand for days after every drill.

The shape of the work.

Drawn new for this page. No screens or data from the client's systems.

The drill runs in five stages: prepare, declare, restore, validate and close. Before each stage starts, a named person approves it. An AI advisor reads every result and explains it but cannot act. Each stage writes evidence, and the drill ends with a sealed evidence bundle.

Fig. 1 The drill as the platform runs it, simplified. Drawn new; no client systems shown.

How it ran.

  1. 01

    Discover

    Walked through the current drill with the client’s teams and agreed the gaps and a baseline for recovery time and effort.

  2. 02

    Design

    A blueprint (architecture, deployment, network access, portal, workflow, the runbook step by step and the drill script) and architecture decision records, signed off before any build.

  3. 03

    Mirror

    A lab mirroring the DR stack (database, application server and web server) with our control and data planes beside it, so every command was rehearsed before it touched a client system. Backup commands ran through a stub with the same grammar, ready to swap for the real ones.

  4. 04

    Build

    A Jenkins pipeline per drill step that executes, verifies and pushes its evidence; screen and log automation on our AI automation platform; a war-room portal with a step tracker, chat and an evidence viewer; a gateway that accepts only signed, approved commands; and an AI advisor that reads every result but cannot act. A versioned JSON schema for step reports let the automation and the AI work move in parallel.

  5. 05

    Onboard

    Firewall requests, forward-proxy and agent connectivity, privileged service accounts through the client’s access management, and single sign-on, each verified from the real hosts.

  6. 06

    Govern

    A register of every command to be run on the client’s systems, each classified as read-only, state-changing or data-overwriting and approved by the client in advance, and a plain-language guide for the non-technical DR roles.

  7. 07

    Rehearse

    A database restore from backup media on the test environment first, with our review of the restore procedure turned into a hardened runbook; then a full dress rehearsal with every role in the portal.

  8. 08

    Drill

    The official drill, closing with the evidence bundle and the continuity report, then a handover and a plan to extend to more applications and scenarios.

Four planes, one gate.

Our control and data planes outside the client’s network, execution agents inside it, and its DR stack reached only through approved commands.

Four planes. Our control plane holds the war-room portal, the gateway, the AI advisor and the orchestrators. Only approved, signed commands pass the gateway to the execution plane: agents inside the client’s network that reach the DR systems over SSH through the client’s privileged access management. The DR target is the client’s own web server, application server, Oracle database and enterprise backup. Every step writes its state and evidence to our data plane: PostgreSQL with pgvector, and write-once object storage with a SHA-256 manifest per step.

Fig. 2 The four planes of the platform, simplified. Drawn new; no client systems shown.

The stack, by layer.

Every product and practice in this work, named so an engineer can check it.

01Orchestration
  • Jenkins declarative pipelines
  • Our AI automation platform
  • Groovy
  • Bash
  • Python
02War-room portal
  • React
  • FastAPI
  • NGINX
  • OpenID Connect single sign-on
  • Multi-factor authentication
  • Role-based access
  • Maker-checker approvals
03Security
  • Mutual TLS
  • HMAC-signed command catalogue
  • Privileged access management
  • Scoped sudo
  • PII redaction
04AI advisor
  • Rule engine
  • Large language model
  • Retrieval-augmented generation
  • pgvector
  • Multi-signal alert correlation
05Evidence and state
  • PostgreSQL
  • MinIO
  • S3-compatible object storage
  • Object Lock (WORM)
  • JSON Schema
  • SHA-256 manifests
06The client’s DR stack
  • Oracle Database
  • Oracle RMAN
  • Veritas NetBackup
  • JBoss EAP
  • Apache HTTP Server
  • SQL*Plus
07Runtime
  • Red Hat Enterprise Linux
  • Windows Server
  • Docker
  • Docker Compose

Constraints we worked within.

AI must not act
The AI has no dispatch rights, and a deterministic rule layer runs under it. The gateway refuses any command without a recorded human approval.
No production impact
Simulation only: no DNS change, no real failover, and one read-only query on production for the baseline.
Every command approved in advance
Each command marked read-only, state-changing or data-overwriting and signed off before its first run. No ad-hoc commands.
Privileged access through the client only
The client exposed only the execution agents. Least-privilege accounts brokered by its own access management, and no database or backup clients installed by us.
The backup stack stays as it is
The database edition has no native standby, so restoring from backup media is the DR path. We orchestrate Oracle RMAN and the enterprise backup already in place.
Evidence kept, personal data removed
An evidence store designed for write-once retention, and personal data redacted from logs before the AI reads them, under a data processing agreement.

What changed.

  • Every drill step runs as a pipeline, and the next one starts only when the responsible person approves it in the war-room portal.
  • Every command traces back to a named approver; the gateway refuses anything else.
  • Evidence, sealed with SHA-256 manifests, is designed to be ready on the day of the drill instead of days later.
  • The restore procedure itself was reviewed, and every finding closed in a hardened runbook.
  • A cloud-agnostic reference architecture, ready to extend to more applications and scenarios.

What we learned.

  1. 01Orchestrate, don’t rebuild. The client’s backup stack worked; the value was in automating the runbook on top of it.
  2. 02In financial services, governance artefacts count as much as code. A command-by-command approval register built trust faster than demos.
  3. 03Test connectivity from the real host and read the proxy’s actual reply. “Blocked” had several different causes.
  4. 04Across trust boundaries, prefer protocols that pass a proxy: a shared file system does not, S3-compatible storage does.
  5. 05Restores fail on details, not on media: list every datafile before remapping, bound recovery to a known point, and preview before moving data.
  6. 06Retries must know the state: transient errors retry, failed gates stop, and destructive steps never retry on their own.

Questions this work answers.

Should AI run a disaster recovery drill?

It should advise, not act. Here the AI correlates alerts, reads every step’s result, explains it and drafts the report, but it has no right to dispatch anything: the gateway refuses any command without a recorded human approval.

How can a DR drill be automated without touching production?

By running in simulation: no DNS change, no real failover, and a single read-only query on production for the baseline. Everything else runs on the DR side, and every command is rehearsed first in a lab that mirrors it.

What evidence does an automated DR drill produce?

Each step writes a machine-readable report and its artefacts, sealed with a SHA-256 manifest, into an object store designed for write-once retention. The drill closes with the continuity report, the completed checklist and a post-incident review draft.

The services behind this work

All work →

Start with a conversation.

Tell us what you run and what is getting in the way. You get a reply within 24 hours.

Talk to an engineer
Email
support@computese.com
Hours
Mon–Fri, 9:00–17:00 ET
Closed on statutory holidays
Office
110 Place d'Orléans Dr
Ottawa, ON K1C 2L9