A Python dlt (dlthub.com) sidecar extracts data from REST APIs, 60+ verified sources, and any SQLAlchemy-supported database into per-profile staging DuckDB files — which the in-process DuckDB hub then ATTACHes read-only. This unlocks a source universe neither the DuckDB ATTACH cluster (Postgres/Mongo/Snowflake/…) nor the SharePoint / Google Sheets snapshot paths can reach.
A long-lived Python sidecar (stdlib http.server — no FastAPI/uvicorn) runs pipelines on demand. Each profile writes to its own staging DuckDB file; the in-process hub ATTACHes it READ_ONLY and auto-discovers collections through every facade (FeatureServer / MVT / OGC). Refresh = re-run → file overwrites in place → next query sees new data.
rest_api_source over any paged JSON endpoint. Pagination: none/offset/cursor/page/header_link. Auth: None/Bearer/ApiKey/Basic.dlt sources pip-installed on demand: Salesforce, HubSpot, Notion, GitHub, Stripe, Airtable, Slack, Google Drive, Zendesk, Shopify, Asana…sql_database over SQLAlchemy: postgresql / mysql / sqlite / mssql / snowflake / oracle / redshift. Especially useful for DBs with no DuckDB scanner (Oracle, SAP HANA, Redshift). Incremental via cursor column.Auto sniffs lat/lon then GeoJSON/geometry; plus LatLon, GeoJson, Wkt, Address (geocoded post-run → GPKG point layer), None.# run a Salesforce pipeline curl -X POST http://localhost:5059/api/dlt/pipelines/sfdc/run → sidecar dlt.pipeline(...).run(source) → writes <dataDir>/dlt/sfdc.duckdb → hub ATTACHes '...sfdc.duckdb' AS "sfdc" (READ_ONLY) → collections discovered → FeatureServer / MVT → geom auto-sniffed (lat/lon → ST_Point)
The Address geometry kind runs a C# post-pipeline geocoder over the staging rows — offline-first (OpenAddresses) with a Nominatim fallback, capped at 50 000 rows per run — and writes a dlt_<id>_geocoded GPKG point layer. Incremental loading persists the cursor value between runs so subsequent runs fetch only new rows.
DltAddressGeocoder post-run step: reads the staging file, geocodes each row, writes a GPKG point layer. Offline-first.
dlt persists the cursor in <dataDir>/dlt/<id>_state/; subsequent runs fetch only new rows.
The host refuses any bind wider than 127.0.0.1 / localhost / ::1; the sidecar additionally validates --host. Never a wider bind.
Profile id == DuckDB schema alias == staging-file basename. Open() only ATTACHes if the file already exists — never spawns Python under the hub lock.
Dlt:PythonPath → bundled tools/python/python.exe → system PATH. Baseline: dlt>=1.4, duckdb>=0.10, pyarrow>=14.
_dlt_* internal tables are filtered out of discovery — operators see only their data collections.
| Group | Endpoint | Notes |
|---|---|---|
| Run | POST /api/dlt/pipelines/{id}/run | Triggers dlt.pipeline(...).run(source) on the sidecar |
| Status | GET /api/dlt/pipelines/{id}/status | Running / completed / failed + lastError |
| Geocode | POST /api/dlt/pipelines/{id}/geocode | Explicit address-geometry post-step → dlt_<id>_geocoded GPKG |
| Sources | GET /api/dlt/sources · /api/dlt/clients | Verified-source catalog + available DB clients |
| Cancel | POST /api/dlt/pipelines/{id}/cancel | Cancel a running pipeline |
Base URL http://localhost:5059; sidecar main port :8012 + health probe :8013. Config "Dlt": { Enabled: true, Host: "127.0.0.1", Port: 8012 }. Documented in DLT_GUIDE.md. Missing Python/dlt/duckdb/pyarrow → every route returns 503 + { error, installHint, available:false }; the sidecar is never auto-started at launch — only on explicit "Run Pipeline."
dlt is ELT, not reverse ETL — no write-back to source (write-back is a separate subsystem; see ServiceNow). Phase-3 roadmap: live progress SSE, scheduled/cron refresh, ADBC integration to surface the staging file to BI tools, and an MCP tool exposing run_pipeline behind the read-only invariant.SaaS apps, REST APIs, and any SQLAlchemy DB — landing as served layers, geometry-aware, loopback-only.