Data work at dltHub: documentation, migrations and benchmarks

My move into data engineering started at a dlt workshop in November 2022, co-hosted by Adrian Brudaru, after eight years in civil engineering. The first contributions to the dlt repository followed in March 2023: setup guides for the Pipedrive, GitHub and Google Sheets sources. I now work at dltHub as a Data Engineer consultant.

Documentation

Most of the early work was setup guides for verified sources: HubSpot, Salesforce, Jira, Notion, Slack, MongoDB, Kinesis, Postgres replication, Zendesk, Asana, Stripe, Shopify, Airtable, Personio, Freshdesk and the sql_database source, among others. On the destination side I updated existing pages for Redshift, Databricks, Synapse, Athena and Iceberg.

I authored or co-authored pages on how dlt can be deployed to Google Cloud Functions, GitHub Actions and similar platforms. Other pages deal with configuration and credentials, incremental loading, schema evolution, data security and encryption, and performance tuning.

In 2023 the work also included end-to-end validation of pipelines across many SaaS sources, including HubSpot, Freshdesk, Pipedrive, Asana, GitHub, Matomo, Notion and Slack, checking data integrity and load behaviour from source to warehouse.

Many of the later changes started as a user's problem: seventeen of the pull requests closed a numbered GitHub issue and two came out of community Slack threads. One issue reported a "File name too long" error on the filesystem destination with deeply nested resources; the fix was a troubleshooting section with the max_identifier_length workaround. Another led to the Databricks docs stating that a staging destination is required rather than optional. Others added Postgres xmin for change data capture, corrected the loader_file_format configuration, fixed the custom config examples to use @configspec, and documented memory tuning through file_max_items and file_max_bytes.

Code

The Freshdesk source was first built for a client, loading into BigQuery, and used in production. It has an API client that handles rate limits and pagination, and loads incrementally on the updated_at field. After review by the dlt team it became a community-verified source, separate from the documentation pull requests above. I later updated the Pipedrive source from the deprecated v1 API to v2.

A HubSpot pipeline was built in the dltHub workbench as part of a migration, described below. Roughly three quarters of it is declarative, using the REST API toolkit. The rest is custom resources for batch associations and fan-out endpoints, which the declarative path cannot express.

Migrations

One project moved a HubSpot-to-MotherDuck pipeline from Fivetran to dlt, with the running Fivetran pipeline kept as the reference. The vendor's output schema was reverse-engineered into a spec covering columns, merge keys, write dispositions and endpoints. The thirteen models in the sales module of Fivetran's dbt_hubspot package were ported into dltHub transformations, keeping the same names and grain in the final tables.

The new mart was then diffed against the Fivetran reference. Row counts did not match, and the reasons turned out to be two connector behaviours: a property the connector had renamed without documenting it, and a routing rule that sent deal stage changes to a separate table instead of the property history. Every mismatch was explained before the migration was called done.

Earlier guides covered moving a MySQL pipeline from Stitch to dlt and another from Fivetran to dlt, both loading into BigQuery. In each case a SQL view joined the vendor's historical data with the new dlt data, so reports kept working across the switch.

Benchmarks

Several of the published articles are benchmarks. Moving 5 million rows of flat and nested JSON from DuckDB to MySQL took 344 seconds with row-based SQLAlchemy loading and 92 seconds with Arrow and ADBC, a 3.7x speedup averaged over five runs. Loading TPC-H data from Postgres to BigQuery with the pyarrow backend held at about 65 GB per hour from 8 GB jobs to 82 GB jobs. Five scenarios on a single 2 vCPU, 4 GB worker showed about a 30x spread in throughput between the fastest and slowest source types.

Earlier comparisons of dlt's SQL backends found ConnectorX the fastest and SQLAlchemy the slowest, with parallelization on ConnectorX cutting one pipeline from 12.58 to 4.65 minutes. A comparison of dlt and Sling on real-world ETL was co-authored with Adrian Brudaru and Shreyas Gowda.

Data quality and governance

The other articles are about keeping data correct and safe. One documents four patterns for protecting PII with dlt and Pydantic, and states what the approach does not cover, such as re-identification across columns. Another collects 11 data quality recipes, each with a runnable marimo notebook. A two-part series on operational health covers auditing data freshness and detecting schema changes from dlt's own load metadata. Others cover the Write-Audit-Publish pattern and SCD2 on nested JSON, where switching to incremental extraction cut query bytes by 25 to 35 percent on updates.

The most recent direction is schema evolution driven by an ontology. A data access policy was written in plain English, and an LLM classified each column as either safe for analytics or rejected before a DuckDB view was built. When a second run added five new columns, they were classified under the same policy with no change to the code: numeric columns passed, PII fragments were rejected.

Tools

dlt, dltHub, Python, SQL, BigQuery, DuckDB, MotherDuck, Postgres, MySQL, dbt, SQLMesh, Pydantic, Arrow and ADBC, ConnectorX, Ibis, marimo, Google Cloud Run.