Step 7 · Day 4 · 07

Full-Stack DR Setup & Operations Guide

1. DR Strategy Decision and Inputs

1.1 Questionnaire inputs consumed (BCP & DR sheet)

1.2 The DR strategy decision record

1.3 DR tiers (bound to the DRTier tag)

1.4 Tag every resource

Sweep the resource register (Doc 6): every PRD resource carries a DRTier tag value agreed in §1.2 — no blanks, no defaults left unexamined.

The DRTier tag is the contract: §3–§5 build exactly what the tags promise, the DR exercise (§6) tests exactly what the tags promise, and the quarterly audit (Doc 5) flags mismatches.

2. DR Region Foundation

2.1 Prerequisites

2.2 Activate the DR compartment

The DR compartment exists empty since Doc 3 §3.2. Update its description with the approved strategy reference and owner (Infrastructure Operations).

Add tag defaults: Environment=DR (locked), CostCenter required.

Create budget BUD-DR-01 scoped to the DR compartment with 70/90/100 alert rules — steady-state DR spend is a standing line the customer approved in §1.2; alert on drift from it.

2.3 Identity and access in the DR region

3. Data Protection Setup (the RPO layer)

3.1 ATP

3.1.1 DR2 path — cross-region backup copy

Console path: Oracle Database > Autonomous Database > <instance> > Manage backups > Edit backup settings

For each PRD ATP: enable cross-region backup copy to <SECONDARY-REGION>; confirm long-term retention schedule (Doc 4 §4.1) also replicates per policy.

Record first successful cross-region backup in the build log; verify it appears under the secondary region’s backup list (switch region → Autonomous Database > Backups).

3.1.2 DR1 path — Autonomous Data Guard

Console path: Oracle Database > Autonomous Database > <instance> > Disaster recovery > Add peer database

Add a cross-region peer: type Autonomous Data Guard; region <SECONDARY-REGION>; the peer needs a private endpoint — this requires the DR VCN (§4) to exist FIRST; sequence accordingly.

After provisioning, note lag metrics (RPO indicator) and set an alarm on data-lag > threshold (§6 register).

Quarterly: perform a switchover test in a maintenance window (switchover is non-destructive) — record duration against RTO.

3.2 Object Storage

For each governed bucket whose DRTier demands it (TFSTATE/evidence, LOGARCHIVE, FBDI-IN, file archives): create a replication policy to a same-named bucket in <SECONDARY-REGION> (Storage > Buckets > <bucket> > Replication Policy > Create). Destination buckets: versioning on, private, same encryption posture.

Replication is one-way and continuous; the destination is read-only while the policy exists. Verify: upload a test object, confirm arrival in the secondary bucket, delete test object, log it.

3.3 Vault and secrets

Virtual private vaults support cross-region replication: enable replica in <SECONDARY-REGION> for PRD-PLATFORM-VAULT-CORE-01 (Vault > Create replica) — keys and secrets become readable in the secondary region.

If the vault type in use does not support replication, maintain the sealed-procedure fallback: documented secret inventory (names/owners, never values) + regeneration procedure per secret (Doc 6 handover pack §7). Decide and record which path applies.

3.4 OIC and configuration state

4. DR Region Infrastructure (the RTO layer)

4.1 What gets pre-built vs rebuilt

4.2 Build the DR VCN — Option A (manual)

Execute Doc 3 §5.2–§5.3 in <SECONDARY-REGION> with the reserved CIDR (10.50.0.0/16 pattern — MUST differ from all primary CIDRs so replication traffic and eventual dual-running never collide): VCN DR-PLATFORM-VCN-CORE-01 in the DR compartment (or Shared-Services:Networking per design — record which), subnets, SGW, NAT (if justified), route table, private DNS zone dr.<customer>.oci, NSG set mirroring PRD rules.

Hybrid: if on-prem connectivity is required in DR (usually yes for SFTP counterparties and users), extend the DRG design: either a second DRG in <SECONDARY-REGION> with its own VPN/FastConnect, or remote peering to the primary DRG — per the network design; document tunnel parameters with the customer network team NOW, not at declaration.

Security: replicate the Security Zone on the DR compartment’s production-data scope, Cloud Guard already covers the region (Doc 2 monitored ALL); verify the region appears in Cloud Guard targets.

4.3 Build the DR VCN — Option B (Terraform/ORM)

The stacks are region-parameterized (provider region variable). Create DR-scoped ORM stacks: STK-DR-STACK2-* (network), STK-DR-STACK10-* (flow logs), STK-DR-STACK9-* (alarms) sourcing the SAME repo, working directories unchanged, with DR tfvars (region=<SECONDARY-REGION>, DR CIDRs, DR names).

Keep DR tfvars in the repo beside primary tfvars (e.g., terraform.dr.tfvars committed per stack) so a declaration applies reviewed, version-controlled values — never hand-typed values under pressure.

Apply the pilot-light set through the normal PR → plan → approve → apply loop; the DR network is a standing, drift-checked part of the estate from then on.

5. Failover and Failback Runbooks

5.1 Declaration and roles

5.2 Failover sequence (DR2 pilot-light reference; adapt per tier)

Declare (per above); open the incident record; start the comms clock; freeze all non-DR changes (ChangeWindow=Frozen).

Network first: verify DR VCN health; bring up hybrid connectivity (tunnels/FastConnect) with the customer network team; verify on-prem reachability.

Data: ATP — DR1: initiate Data Guard failover (role transition); DR2: restore latest cross-region backup to a new instance on the DR private endpoint; capture the restore timestamp = achieved RPO. Buckets: replicated buckets are already present (destination becomes writable once the replication policy is removed — do so deliberately, log it).

Secrets: vault replica readable in-region (or execute the regeneration procedure); verify one secret retrieval before dependent services start.

Compute: launch SFTP/agent VMs from copied custom images; apply the hardening checklist deltas; register OIC connectivity agents against the DR OIC.

Integration: provision/start OIC (pilot light: provision now — this is usually the RTO long pole; warm standby: already provisioned); import latest design-time export with the DR config plan; re-key connections from the vault; activate integrations in dependency order (foundation → transactional).

[SAAS] Fusion is unaffected by a customer-region outage (Oracle-managed, different infrastructure); integrations reconnect from the DR OIC to the same Fusion pods. If the event is an ORACLE SaaS outage instead, this runbook does not apply — track Oracle’s SaaS recovery, keep the error-hospital draining, replay queued files after restoration.

DNS/endpoints: cut counterparty-facing endpoints (LB IPs, SFTP host) to DR — pre-lowered TTLs help; notify counterparties per the comms plan (new IPs / host keys if they changed — better: preserve host keys via the vault so they do not change).

Validate: run the workload smoke set (Doc 4 §10 tests 1, 2, 4, 5) against DR; record achieved RPO and RTO per workload vs the tags’ promise.

Steady-state in DR: alarms live (§4.3 stack9-DR or manual equivalents), budgets watched, error hospital monitored; document everything — the failback needs it.

5.3 Failback

Failback is a planned migration, not an emergency: schedule it, change-control it, and rehearse the data direction carefully.

Data: DR1 — Data Guard switchover back after the primary is re-established and caught up. DR2 — reverse-path: backup/export the DR ATP, restore to the rebuilt primary; reconcile deltas written during DR (the reconciliation file pattern from the migration machinery applies).

Re-point integrations back with the primary config plan; re-establish bucket replication primary→secondary (recreate policies); DNS back; counterparties notified.

Post-failback: full Doc 5 validation pass on the primary + CD3 extract + a post-incident review that updates this runbook with everything learned.

6. DR Operations (standing)

6.1 Backup verification cadence

6.2 DR exercise program

Semi-annual tabletop: walk §5.2 end-to-end with all role-holders; inject two surprises (e.g., vault replica unavailable; a counterparty unreachable); update the runbook with every gap found.

Annual live exercise: execute §5.2 for at least one production workload into <SECONDARY-REGION> in a maintenance window (or as an isolated parallel run that never takes production traffic — decide the model with the Board); measure achieved RPO/RTO against the DRTier promise; the DR-exercise success rate is a standing governance KPI (Doc 1 §14).

Every exercise ends with: findings register (Doc 5 format), runbook updates, and Board-visible report.

6.3 DR drift control

6.4 [SAAS] Fusion business-continuity validation

7. DR Readiness Gate

Revision History