From raw taxi files to a managed Spark job

This project explores one practical question: how does an analysis that begins in a notebook become a repeatable Spark job that runs against data in cloud storage?

The source is the public NYC taxi dataset. Yellow and green taxi records are ingested, explored with Spark SQL, joined to zone data, stored as Parquet, and then processed through Google Cloud Dataproc.

The problem

Notebook analysis is useful for learning and exploration, but it does not by itself establish a repeatable data path. The project needed to preserve the useful SQL work while separating source ingestion, storage, transformation, and execution.

The resulting flow is deliberately simple:

NYC taxi files
      ↓
Bash ingestion
      ↓
Partitioned Parquet in GCS
      ↓
PySpark transformations on Dataproc
      ↓
Query result written back to GCS

Important decisions

Combine at a shared grain

Yellow and green taxi data have related but distinct schemas. The useful analytical path normalises the fields needed for the question, combines the two datasets, aggregates them at hour and zone grain, and joins the result to taxi-zone reference data.

That ordering matters. Joining raw trip data too early would make the intermediate state larger and the intent harder to inspect.

Use Parquet as the hand-off

Parquet creates a clean boundary between ingestion and analysis. It preserves column types, supports partitioned reads, and avoids repeatedly parsing the source representation.

Parameterise cloud execution

The notebook logic was converted into a Python job that accepts input and output locations as arguments. This makes the Dataproc execution reusable across years and taxi types rather than coupling it to one exploratory run.

parser.add_argument("--input_green", required=True)
parser.add_argument("--input_yellow", required=True)
parser.add_argument("--output", required=True)

What the project demonstrates

• moving from local Spark sessions to managed execution;
• reading partitioned Parquet directly from Google Cloud Storage;
• combining heterogeneous sources with an explicit analytical grain;
• turning notebook logic into a parameterised job;
• writing a durable result back to object storage.

What I would improve next

The next version should add an explicit schema contract, automated data-quality checks, infrastructure as code for the Dataproc resources, and a small orchestration layer that records each published dataset release.

That evolution would turn a working analytical pipeline into an operationally trustworthy one.

View the source project on GitHub