Connectors & CDC

Every source, one typed ingestion layer.

Point SpooqW at your databases, streams and files. Each connector is a typed, validated step — the engine knows the schema before the run starts, and credentials never touch a log line.

$ spooqw run crm_changes --schedule "*/15 * * * *"
kafka crm.events availableNow → drained 18,240 events
parse avro schema registry v14
MERGE INTO customers_silver ON id17,891 upserts · 349 deletes

Databases via JDBC

Postgres first-class, plus any JDBC database — partitioned parallel reads, predicate pushdown and typed column mapping out of the box.

Kafka streaming ingestion

availableNow draining on your cadence: each scheduled run consumes exactly what has arrived, parses JSON or Avro against the registry, and lands it as governed Iceberg tables.

CDC change-log readers

Native change-log ingestion turns inserts, updates and deletes into MERGE operations on the lakehouse — keeping silver tables in sync with the source without full reloads.

Object storage & files

CSV, JSON, Parquet and Avro from S3-compatible object storage, with schema inference validated into typed steps you can trust in production.

Hosted Postgres database

No source yet? On the cloud tiers, provision a managed Postgres in seconds — no VPN, no IP whitelist. You get a connection string and load data with psql, dbt or your own app; it's registered as a pipeline source automatically. Need new credentials? Delete and re-provision — the old database and role are dropped immediately, nothing is silently rotated behind your back.

Encrypted, masked credentials

Connection secrets are encrypted at rest with AES-256-GCM and masked everywhere they could appear — API responses, logs, audit records.

A growing catalog

The connector surface is expanding — native DB2 and more CDC sources are on the roadmap. Missing something you need? Tell us and we'll prioritize it.

Explore the rest of the platform:Pipeline builderAI & AutopilotLakehouseGovernance