Case studyIndependent project
From raw taxi files to a managed Spark job
An independent project that moves NYC taxi data through local Spark exploration, cloud storage, and a parameterised Dataproc job.
Explore the decisionsData engineering · Analytics · BI
I build reliable data systems—and explain the decisions behind them.
I'm Aman Gupta. This is where I turn data engineering work into clear system designs, practical guides, and honest lessons.
Selected work
Systems are made of decisions.
Case studies and reference architectures focused on constraints, trade-offs, and repair—not tool lists.
Reference architectureReference design
Designing pipeline health around the consumer
A reference architecture for measuring freshness, completeness, and publication state—not just whether scheduled tasks ran.
Explore the decisionsSelected writing
Technical ideas, made useful.
Notes about building pipelines, defining trustworthy metrics, and operating data products in the real world.
Metadata Over Compute: Visualizing Table History with dlt and Iceberg
A dlt + Iceberg demo showing “metadata over compute”: table history lives in portable metadata files, not in any single catalog or engine.
Ontology engineering: bringing meaning to data
Ontology engineering encodes expert knowledge into machine-readable structures. It is "data modeling for meaning," ensuring all systems see data the same way.
A non-linear path
From civil and nuclear engineering to data.
My engineering path changed disciplines, but not its underlying concern: understand the system, respect its constraints, and make its behaviour legible.
More about me