mountaindata.ai

Navit's data stack: three years in production, then its move to dltHub

Looking after Navit's analytics data since 2023. The first stage was building the stack that brought their database, CRM and support desk data into one warehouse. Around three years later came the move to dltHub, which I helped with.

By Aman Gupta3 min read

I have worked with Navit as a Data Engineering Consultant since 2023, and the engagement is ongoing. The work has happened in two stages. The first was building a data stack from separate tools, which ran and served the analytics use cases for around three years. The second was helping move that stack onto dltHub, and maintaining it there since.

Stage one: a stack built from separate tools

Data came from three places: the production Postgres database, HubSpot and Freshdesk. Pipelines I built with dlt loaded all three into BigQuery.

Inside BigQuery the data moved through three layers, built with dbt. Bronze held the data as it arrived. Silver held it cleaned and standardised. Gold held it shaped into facts and dimensions for reporting.

The dlt pipelines and the dbt transformations both ran on Airflow, hosted on Google Cloud Composer. On top of the warehouse I built several BI dashboards in Metabase.

This setup ran for around three years, and the pipelines stayed my responsibility throughout. Loading, transforming and scheduling were three separate pieces, each with its own setup, and keeping them working together was part of the role.

Stage two: the move to dltHub

After around three years the stack moved to dltHub, and I assisted the dltHub team with the move. Having built and maintained the first stack, I knew what each pipeline and table was for, and that knowledge had to carry over with the data.

dltHub now runs all of it. Every pipeline is managed there, from ingesting the raw data to orchestrating the runs, on one platform instead of three separately hosted pieces. The Airflow setup on Cloud Composer was retired.

The move was also the point to redo the data models. The first models were built piece by piece over three years, while the requirements kept changing. Three years on, the business has largely settled on which analyses matter most. With dltHub Transformations, the models were rebuilt as canonical data models around those analyses.

Day-to-day work on the pipelines goes through the dltHub AI harness: a set of skills, rules and tools that a coding agent uses to build, run and fix pipelines.

Tools

Stage one: Postgres, HubSpot, Freshdesk, dlt, BigQuery, dbt, Airflow, Google Cloud Composer, Metabase.

Stage two: dltHub, dltHub Transformations, dltHub AI harness.

Written by Aman

Clearer systems begin with clearer conversations.

If this idea connects to a system you're building, I'd be glad to hear about it.

Start a conversation

Continue reading

Related ideas

Career5 min

Data work at dltHub: documentation, migrations and benchmarks

A Data Engineer consultant at dltHub, contributing to dlt in public since March 2023: 145+ merged pull requests to the dlt repository, mostly documentation, and 32 technical blogs; a community-verified Freshdesk source and maintenance work on a few others; vendor-to-dlt migration guides; performance benchmarks; and articles on data quality and migrations. It started with a dlt workshop in November 2022.

Career3 min

Data work at RentLondonFlat.com: from app schema to dashboards

Almost three years looking after the company's data, from designing the tables its in-house app writes to, through to the dashboards the team reads. Along the way: checks that keep bad records out, a health score for every property, a value score for every customer, and a GPT that turns a plain-English question into a query.