Build · Banking data stack · Workshop 2 of 8
Post every transaction once. Prove it.
Harbour Bank · Monday 19 October · The Kafka posting consumer goes live once Risk signs it off.
Friday's duplicate came from a job that retried. I will sign off the new consumer when it survives what the switch actually sends: repeats, bad messages, crashes and month-end volume. Show me the evidence, not the design.
The test Risk will run
Your consumer runs twice from a clean ledger. Once alone, killed with SIGKILL at 20%, 50% and 80% of the way through and restarted each time. Then as three consumers in one group. After each run, six controls are audited against the ledger, NPCI's file and Kafka itself.
| Control | What it proves |
|---|---|
| Every settled transaction posted exactly once | No event lost, none posted twice |
| Both legs, opposite sides, same amount | Double entry holds for every posting |
| Ledger agrees with NPCI | All 240 accounts and the settlement GL tie to NPCI's file |
| Malformed message parked in the dead-letter topic | Bad data is quarantined, not dropped and not blocking |
| Consumer group caught up, offsets committed | Nothing left unread; a restart picks up cleanly |
| Each account posted in settlement order | Parallelism has not reordered an account's history |
You start from the first draft of the consumer. It fails most of these.
Start
This workshop runs Apache Kafka 4.1 and Postgres 17, so it needs a machine rather than a browser tab. A free GitHub Codespace is the quickest: Kafka and Postgres start on their own. GitHub Free includes about 60 hours a month on the 2-core machine it uses.
Or on a laptop with Docker and Python 3.10 or later:
git clone https://github.com/shashwat-kumar/readsigil-data-stack cd readsigil-data-stack docker compose up -d --wait pip install -e . harbour doctor
Then, in either:
harbour resetandharbour publishput 16 October's events on the topic. Look at them with Kafka's own console consumer.- Run the first draft with
python workshops/02-kafka/consumer.py, thenharbour status. Work out what happened to the ledger and to the consumer group's offsets. - Run
harbour check ws2. Fix one failing control at a time. The guide has a hint for each, folded away until you want it.
On every change, the reference consumer passes all six controls against real Kafka and Postgres, and the first draft plus three broken variants each fail the control that names their bug.On a Cloudera/Hadoop stack
Cloudera's Streams Messaging is Apache Kafka, so consumer groups, offsets, partitions and the dead-letter pattern work exactly as in this workshop. In production you would add Streams Messaging Manager to watch consumer lag, Schema Registry so a malformed message is rejected when it is produced, and Ranger policies on who may write to the topic and who may read the dead-letter topic.
The workshop series
| Workshop | Situation | Cloudera/Hadoop counterpart |
|---|---|---|
| 1 · Postgres, DuckDB | UPI settlement does not tie to the ledger | Core banking replica; Impala or Hive over HDFS/Ozone |
| 2 · Kafka | Post every transaction once, through repeats, crashes and month-end volume | Kafka (Cloudera Streams Messaging) |
| 3 · Airflow | Backfill a broken week so each night is published once | Oozie; Airflow in Cloudera Data Engineering |
| 4 · Iceberg In preparation | Restate the March book after a late correction | Iceberg tables in CDP |
| 5 · dbt Core In preparation | One definition of CASA ratio across fourteen reports | Hive and Impala SQL jobs; dbt-hive and dbt-impala adapters |
| 6 · Spark In preparation | The EOD batch slows because one corporate account fills one partition | Spark on CDP, Cloudera Data Engineering |
| 7 · Metabase In preparation | The MD's morning dashboard, built on reconciled numbers only | Cloudera Data Visualization, Hue |
| 8 · Capstone In preparation | Run the whole end-of-day pipeline yourself | Docker on your laptop |