Skip to content
Computese home
Banking07Cloud02Hosting

Migrating a bank from OpenShift 3 to OpenShift 4

A bank’s digital channels ran on OpenShift 3, past its end of life. We moved production to OpenShift 4 inside one short change window, rebuilt disaster recovery across two sites, then ran the platform and its DevSecOps toolchain around the clock.

Industry
Banking
Our role
Lead DevOps engineering
Engagement
Delivery, then a year of managed service
Platform
Red Hat OpenShift on VMware vSphere
Environments
Production, DR and non-production
Support
24x7 for critical incidents

Where it started.

The bank’s digital channels, from onboarding and eKYC to lending, wealth, insurance, securities and anti-money-laundering, run as dozens of microservices on one container platform, behind an API gateway and a service mesh that reach core banking, next to traditional Java middleware.

  • Production still ran on OpenShift 3, on operating system and container runtime releases past their support, so every month added security and operational risk.
  • No single partner ran the platform and its DevSecOps toolchain; the bank wanted one team accountable around the clock.
  • Releases could not be load-tested, and the environment is air-gapped, so standard installs did not work.

The shape of the work.

Drawn new for this page. No screens or data from the client's systems.

Production moves from OpenShift 3, past its end of life, to OpenShift 4 at the primary site. A second cluster at the DR site receives continuous replication. The cutover runs as pre-sync, delta sync, final sync and go-live, with the last two inside one change window. Afterwards the platform is run 24x7 across production, DR and non-production.

Fig. 1 Shape of the engagement: upgrade, protect across two sites, then run. Drawn new; no client systems shown.

What moved, layer by layer.

OpenShift 3 has no in-place upgrade to OpenShift 4, so this was a migration to a new platform, not an update.

Container platform
Before: Red Hat OpenShift 3
After: Red Hat OpenShift 4
Node operating system
Before: Red Hat Enterprise Linux
After: Red Hat Enterprise Linux CoreOS
Container runtime
Before: Docker
After: CRI-O
Persistent storage
Before: NFS, with volumes shared between services
After: OpenShift Data Foundation, one volume per service
Disaster recovery
Before: A previous design, replaced in full
After: Regional DR: ODF volume mirroring, Red Hat ACM, Submariner
Logs
Before: A large log store
After: A fresh logging stack; old logs not copied across
Load testing
Before: None before release
After: Tricentis NeoLoad, installed offline
Operations
Before: No single accountable partner
After: One team, 24x7 for critical incidents, SLAs per priority

How it ran.

  1. 01

    Assess

    Inventoried every persistent volume and service before planning any downtime. Real usage was well above what had been provisioned and several services shared one NFS volume, so the plan was resized and shared volumes split before the move.

  2. 02

    Sync

    Pre-synced the data to OpenShift 4 through the bastion with rsync, then ran delta syncs verified by checksum, so the cutover had little left to move.

  3. 03

    Cut over

    Stopped the workloads on OpenShift 3, ran the final sync, exported and imported the remaining data, deployed and verified on OpenShift 4, then went live, all inside one short change window.

  4. 04

    Protect

    Built the DR cluster on the regional DR pattern: OpenShift Data Foundation mirrors block volumes to the DR site, Red Hat Advanced Cluster Management places applications and fails them over, and Submariner links the cluster networks. A full initial sync, then continuous replication, then rehearsed runbooks for switching production to DR and back.

  5. 05

    Test

    Froze a NeoLoad release, mirrored its images and Helm charts into the internal registry, and deployed it on the platform fully offline, working around gaps in the vendor’s own install guide, then smoke-tested it.

  6. 06

    Run

    Transition, stabilize, operate: the platform, the middleware and the whole toolchain (Jenkins, Argo CD, SonarQube, Nexus, Trivy, Prometheus and Grafana) across production, DR and non-production, with SLAs per incident priority.

The platform, inside the air gap.

How a change reaches production with no internet access, and how production reaches the DR site.

Vendor images, operators and charts come in from outside through an approved, checksummed transfer into the internal registry; nothing inside the air gap reaches the internet. Inside, a change moves from Git through a Jenkins build, a Kaniko image build, the SonarQube quality gate and scans by Trivy and Red Hat Advanced Cluster Security into Sonatype Nexus, and Argo CD syncs it to OpenShift 4. Red Hat Advanced Cluster Management places applications on the primary and DR sites and fails them over; OpenShift Data Foundation mirrors volumes from the primary site to the DR site, and Submariner links the cluster networks. Both sites run on VMware vSphere. Prometheus, Thanos, Grafana and Loki observe the platform.

Fig. 2 The delivery chain and the two sites, simplified. Drawn new; no client systems shown.

The stack, by layer.

Every product and practice in this work, named so an engineer can check it.

01Container platform
  • Red Hat OpenShift 4
  • Red Hat Enterprise Linux CoreOS
  • CRI-O
  • Kubernetes
  • Rancher
  • VMware vSphere
  • Red Hat Enterprise Linux
02Storage and DR
  • OpenShift Data Foundation
  • Ceph RBD mirroring
  • Red Hat Advanced Cluster Management
  • OpenShift DR operators
  • Submariner
03CI/CD and GitOps
  • Jenkins
  • Argo CD
  • Kaniko
  • Sonatype Nexus
  • SonarQube
  • Helm
04Security
  • Red Hat Advanced Cluster Security
  • Trivy
  • Single sign-on
  • Cluster hardening
  • Bastion access with SSH keys
  • Internal image registry
05Observability
  • Prometheus
  • Thanos
  • Grafana
  • Loki
06Middleware
  • JBoss EAP
  • Apache Tomcat
  • Apache HTTP Server
  • NGINX
  • HAProxy
07Databases
  • PostgreSQL
  • MySQL
  • MongoDB
  • Apache Cassandra
  • IBM Db2
  • Redis
  • Kubernetes operators
08Performance testing
  • Tricentis NeoLoad
  • Offline install from an internal mirror
09Migration tooling
  • rsync
  • Checksum-verified delta sync
  • Storage inventory scripts
  • Connectivity and firewall checks

Constraints we worked within.

One short change window
Pre-sync and checksum-verified delta syncs left only the final changes, the deployment and the checks for the window.
Recovery in minutes, data loss in seconds
Asynchronous replication over a high-bandwidth, low-latency link between sites, and runbooks for data that has not replicated yet.
No direct cluster-to-cluster access
All cross-site traffic through a bastion and HAProxy, key-based access, checksums on every transfer and firewall rules that expire after the move.
Air-gapped
No internet access anywhere: release images, operators, charts and tool images mirrored to an internal registry, and every install run offline.

What changed.

  • Production runs on OpenShift 4, a supported release.
  • A DR site kept in sync by volume mirroring, with switch-over runbooks that have been rehearsed.
  • Load testing with NeoLoad before release, inside the bank’s air gap.
  • One partner accountable for the platform and its toolchain, 24x7 for critical incidents.

What we learned.

  1. 01Measure real data before planning a migration. What was provisioned is not what is stored.
  2. 02Look for volumes shared between services before the move, and split them first.
  3. 03Not all data has to move. For large log stores, a fresh logging stack beat copying the old logs across.
  4. 04Plan firewall approvals between sites as a task with its own timeline, not an appendix.
  5. 05Settings hard-coded per environment become manual steps during a real DR switch. Find them early.

Questions this work answers.

Can OpenShift 3 be upgraded to OpenShift 4 in place?

No. OpenShift 4 is a new platform, so the move is a migration: workloads and persistent volumes are copied to a new cluster. We measure every volume first, pre-sync the data, run checksum-verified delta syncs, and leave only the final sync, deployment and checks for the change window.

How does disaster recovery work on OpenShift 4?

With two clusters, one per site. OpenShift Data Foundation mirrors block volumes to the DR site, Red Hat Advanced Cluster Management places applications and fails them over, and Submariner connects the cluster networks. Runbooks cover switching to DR and back, including data that has not replicated yet.

Can OpenShift and its toolchain run with no internet access?

Yes. Release images, operators, Helm charts and tool images are mirrored into an internal registry, and every install runs from that mirror, including load-testing tools that expect to download at install time.

Start with a conversation.

Tell us what you run and what is getting in the way. You get a reply within 24 hours.

Hours
Mon–Fri, 9:00–17:00 ET
Closed on statutory holidays
Office
110 Place d'Orléans Dr
Ottawa, ON K1C 2L9