Web Crawler — 3D Map Explorer

How it works

Three crawl modes × four extraction modes

Pick how to find pages and how to pull data out of them. A pure-SQL composer (CrawlerSqlBuilder) assembles the statement; a live preview runs it with LIMIT 10 against an in-memory connection before you commit a full crawl. When geometry is found, the result registers as a project layer — no new layer type.

Pages — a fixed URL list via crawl(['url1','url2']). Simplest; one-shot scrape.
Sitemap — an XML sitemap URL → sitemap('...') joined to crawl_url(loc) via LATERAL, with an optional MaxUrls cap.
Discover — a seed URL + MaxDepth + MaxPages + optional regex UrlPattern → discover(seed, depth, pages) link-following crawl.
Geometry post-step — LatLon→ST_Point, GeoJson→ST_GeomFromGeoJSON, Wkt→ST_GeomFromText, Auto sniffs lat/latitude/y + lon/lng/longitude/x. Reuses the DLT geometry config verbatim.

POST /api/crawler/preview

// preview: sitemap → table extraction
SET crawler_respect_robots = true;
SET crawler_default_delay   = 1000;

SELECT * FROM (
  SELECT loc FROM sitemap('https://site/sitemap.xml')
) s, LATERAL crawl_url(s.loc) c;

→ 10-row preview, no CREATE TABLE
→ commit → crawl_profile table + geom column
→ registered as a project map layer (EPSG:4326)

Extraction modes

Four ways to pull structured data out of a page

Each maps to a DuckDB function. The Table mode is the Google Sheets IMPORTHTML analog; CSS gives you one row per matched element; JS-var reads data baked into a page's script object; Readability runs Mozilla's article extractor for title + body text.

📋

Table

read_html(url, 'table.selector', index) — the IMPORTHTML analog. Pull a named HTML table into rows.

🎯

CSS selector

jq(html.document, 'selector'[, 'attribute']) — one row per matched element, text or attribute.

{ }

JS variable

read_html(url, 'js=varName') — for pages that bake data into a JavaScript object.

📰

Readability

Mozilla Readability article extraction → title + textContent. The content-capture path.

The API surface

Profiles, preview, run, jobs

Group	Endpoint	Notes
Capability	GET /api/crawler/capabilities	Reports whether the `crawler` extension loaded
Profiles	GET / POST /api/crawler/profiles	List / create a crawl profile (source + extraction + geometry + settings)
Preview	POST /api/crawler/preview	10-row preview against a `:memory:` connection — no table created
Run	POST /api/crawler/run/{profileId}	Returns 202; async job → staging DuckDB + registered project layer
Jobs	GET /api/crawler/jobs/{id} · /api/crawler/jobs	Poll a run / list recent runs

Base URL http://localhost:5059. Storage: per-profile <dataDir>/crawler/<profileId>.duckdb + a crawl_<profileId> table; profile config (incl. auth) DPAPI-encrypted in <dataDir>/crawler/profiles.json. Defaults: delay 1000 ms, timeout 30 s, max response 10 MB, respect_robots=true, user-agent 3DMapExplorer-Crawler/1.0, max 200 pages, max depth 3. Disabled by default (Crawler:Enabled=false); missing extension → 503 + INSTALL crawler FROM community; LOAD crawler;.

v1 scope: static HTML only (no JS-rendered pages — a Playwright sidecar would be a separate subsystem), single bearer/cookie auth (no OAuth), no scheduled re-crawl, no CRAWLING MERGE INTO upserts (always re-crawls fresh). Address geometry honestly degrades to an attribute table in v1.

Turn any website into a map layer

Pages, sitemaps, or a discovered link graph — structured + geospatial extraction, previewed live, no Python.

⬇ Download Free Trial Events Tap → DLT Data Loading →

The web is a data source. Crawl it onto the map.