Build · Banking data stack · Workshop 1 of 8
The numbers match. The money doesn't.
Harbour Bank · Saturday 17 October, 07:42 · UPI settlement for 16 October is due for sign-off at 09:30.
The overnight check passed, but the settlement GL does not agree with NPCI's net settlement for 16 Oct. I can't sign off a difference I can't explain. I need the cause and the corrections approved before 09:30.
Where do you start?
Pick a line of inquiry. You can follow all three, in any order.
Break register
Correct the books
Data Platform cannot post to the core ledger. Each correction goes to Finance Control for approval, then the correction job posts what was approved. One break is on the clock: under RBI's turnaround-time rules, a beneficiary bank has until T+1, which is today, to credit the customer or return the money. After that it owes the customer ₹100 a day.
The workshop series
Eight workshops on one fictional bank's end-of-day pipeline. Each maps the open-source tool to its counterpart on a Cloudera/Hadoop estate.
| Workshop | Situation | Cloudera/Hadoop counterpart |
|---|---|---|
| 1 · Postgres, DuckDB | UPI settlement does not tie to the ledger | Core banking replica; Impala or Hive over HDFS/Ozone |
| 2 · Kafka | Post every transaction once, through repeats, crashes and month-end volume | Kafka (Cloudera Streams Messaging) |
| 3 · Airflow | Backfill a broken week so each night is published once | Oozie; Airflow in Cloudera Data Engineering |
| 4 · Iceberg In preparation | Restate the March book after a late correction | Iceberg tables in CDP |
| 5 · dbt Core In preparation | One definition of CASA ratio across fourteen reports | Hive and Impala SQL jobs; dbt-hive and dbt-impala adapters |
| 6 · Spark In preparation | The EOD batch slows because one corporate account fills one partition | Spark on CDP, Cloudera Data Engineering |
| 7 · Metabase In preparation | The MD's morning dashboard, built on reconciled numbers only | Cloudera Data Visualization, Hue |
| 8 · Capstone In preparation | Run the whole end-of-day pipeline yourself | Docker on your laptop |