Build · Banking data stack · Workshop 3 of 8

Backfill the week. Publish each night once.

Harbour Bank · Sunday 18 October, 08:30 · Treasury reads the position table at 09:00.

Treasury to Data Platform08:30

The position table has no figures for 13, 14 or 15 October and three for 16 October, all different. I cannot fund tomorrow's settlement from that. Tell me when each night shows one number I can rely on.

treasury_position, 13 to 16 OctNot reliable

Nights with no position3

Rows for 16 Oct3

Files waiting in landing4

What the week left behind

Tuesday's deploy changed how the pipeline decides which night it is processing. On-call reran it when NPCI's 16 October file landed late, while the file was still being written. NPCI's 14 October file is one record short of its own control totals. You own the DAG.

The test Treasury will rely on

Your DAG runs on real Apache Airflow: a backfill of 13 to 16 October, on-call reruns of two nights, then NPCI's corrected 14 October file. Six controls are then audited against NPCI's files and the ledger.

Control	What it proves
One published position per business date	Treasury sees one figure per night
Published figures agree with NPCI and the ledger	The numbers are right, not just present
A rerun changes nothing	On-call can rerun any night without making it worse
Late file waited for, not read half-written	The pipeline waits for NPCI to finish
Each run processed its own business date	A backfill does what it says
A file that fails its control totals is not published	A wrong file stops the run instead of reaching Treasury

You start from the DAG that has been in production since Tuesday. It fails all six.

Start

This workshop runs Apache Airflow 3.2 and Postgres 17, so it needs a machine rather than a browser tab. In a free GitHub Codespace, Airflow is installed for you. GitHub Free includes about 60 hours a month on the 2-core machine it uses.

Open in GitHub Codespaces Read the workshop guide

Or on a laptop with Docker and Python 3.10 to 3.13:

git clone https://github.com/shashwat-kumar/readsigil-data-stack
cd readsigil-data-stack
docker compose up -d --wait postgres
./scripts/install-airflow.sh

Then, in either:

Start Airflow with airflow standalone and run harbour ws3 reset to put NPCI's four files in the landing folder.
Run one night with airflow dags test harbour_eod 2026-10-13, then harbour ws3 status. Work out which night it actually processed.
Run harbour check ws3. Fix one failing control at a time. The guide has a hint for each, folded away until you want it.

On every change, the reference DAG passes all six controls on real Airflow; the production DAG and four broken variants each fail the control that names their bug; and a real scheduler backfill publishes the same figures.

On a Cloudera/Hadoop stack

Many Cloudera estates schedule end-of-day work with Oozie coordinators, and Cloudera Data Engineering runs Airflow, so this DAG ports across. The rules are the same: take the business date from the run, replace a date's output rather than adding to it (on Hive or Impala, INSERT OVERWRITE of that date's partition), wait for the trigger or control file, and fail closed when control totals disagree.

The workshop series

Workshop	Situation	Cloudera/Hadoop counterpart
1 · Postgres, DuckDB	UPI settlement does not tie to the ledger	Core banking replica; Impala or Hive over HDFS/Ozone
2 · Kafka	Post every transaction once, through repeats, crashes and month-end volume	Kafka (Cloudera Streams Messaging)
3 · Airflow	Backfill a broken week so each night is published once	Oozie; Airflow in Cloudera Data Engineering
4 · Iceberg In preparation	Restate the March book after a late correction	Iceberg tables in CDP
5 · dbt Core In preparation	One definition of CASA ratio across fourteen reports	Hive and Impala SQL jobs; dbt-hive and dbt-impala adapters
6 · Spark In preparation	The EOD batch slows because one corporate account fills one partition	Spark on CDP, Cloudera Data Engineering
7 · Metabase In preparation	The MD's morning dashboard, built on reconciled numbers only	Cloudera Data Visualization, Hue
8 · Capstone In preparation	Run the whole end-of-day pipeline yourself	Docker on your laptop