Regulators are moving from summary templates to transaction-level data. On a bank’s own Cloudera platform, we proved it can answer two questions most banks cannot: where a filed number came from, and what it looks like cut a new way. Each answer is one SQL query.
Supervisors increasingly want data once, at transaction level, instead of dozens of overlapping summary reports. That moves most of the work from the reporting tool to the data platform: storage, lineage and governance.
Drawn new for this page. No screens or data from the client's systems.
Five source feeds (core banking, payments, treasury, customer and reference data) land in a raw layer that is only ever appended to. Data then moves through a conformed layer with quality flags, a gold layer holding the attribute model, and reporting marts reconciled to gold. The reporting engine sits outside the scope. Each filing is pinned to a snapshot so it can be reproduced. Lineage, masking by role and an audit trail cover every layer.
Confirmed on day one which services, storage, Ranger permissions, Atlas hooks and Kerberos authentication really worked, before building on any assumption.
Five source feeds (core banking, payments, treasury, customer and reference data) landed append-only in Iceberg raw tables through NiFi and Kafka, as batch, change data capture and micro-batch streams, reference data first.
A conformed layer with a quality flag on every row and corrections resolved to their latest version; counterparties kept as SCD Type 2 versions; amounts converted at each reporting date’s exchange rate, stored beside the record; and the attribute model supervisors ask for, with residency, both maturity bases and both immediate and ultimate risk.
Atlas classifies personal data and draws lineage from every filed field back to its source. Ranger masks by tag and role on the same query, filters rows and audits every read.
Reproduced a filed figure from its Iceberg snapshot with time travel, traced a restatement to the records that changed, answered a new question live, and rehearsed the fallback path. Every SQL step was checked offline on DuckDB against expected results first.
A filed figure reproduced from its snapshot, the current figure after a restatement, and the records that explain the difference.
Every change to an Iceberg table makes a snapshot, and the submission register pins the snapshot behind each filing. The same query run with time travel on the filed snapshot reproduces the filed figure exactly; run on the latest snapshot it gives today’s figure after a restatement. The difference traces to two records in the raw layer, the original and its correction, both kept. Exchange rates at the reporting date, counterparties kept as versions and reference codes as of the date keep the old figure from drifting.
Every product and practice in this work, named so an engineer can check it.
Pin every filing to an Apache Iceberg snapshot in a submission register, then query that snapshot with time travel. Exchange rates at the reporting date, counterparties kept as SCD Type 2 versions and reference codes joined as of the date keep the old figure from drifting.
Because every layer above it can then be rebuilt, and a restatement can be explained: the original record and its correction both stay, so the difference between the filed figure and today’s traces to exact rows.
Apache Atlas classifies columns as personal data, and Apache Ranger policies written on that classification mask them by role. An analyst and a supervisor run the same SQL and see different values, every read is audited, and a new table that inherits the tag is protected at once.
Tell us what you run and what is getting in the way. You get a reply within 24 hours.
