Spark pipelines on a real lakehouse.
Built, run and governed.
Design pipelines on a visual canvas — or generate them from a sentence with the AI copilot — run them on a typed Scala engine over Apache Iceberg and Nessie, and let Autopilot self-heal what breaks at 2 AM.
3 pipelines, 2 connections and a managed Spark runner on the Free plan — forever.
Run · zero-copy branch
Catch it before production. Prove it after.
Every signature feature exists to answer one question: can I trust this run with my data?
Predictive dry-run
Sample-execute every step against the live source via Trino before spending a single Spark core. Row estimates, output schema, data preview — per node.
Zero-copy branch runs
Runs write to an isolated Nessie branch. Contracts pass → an atomic, metadata-only merge to main. Contracts fail → the data stays quarantined. No blast radius.
Quality gates, in-engine
Nine rule types — uniqueness, accepted values, regex, freshness and more — enforced inside the Spark run, with a structured per-rule report every run.
Canvas time-machine
Scrub through any past run directly on the canvas: nodes light up in execution order with live record and byte counters, at 1×, 2× or 4× playback.
Run data-diff
What changed in the data between two runs — rows, bytes, duration and per-step deltas — not just what changed in the code.
AI copilot + self-healing Autopilot
Generate pipelines from a sentence, edit the canvas in plain language — then let Autopilot detect incidents, prove the fix on an isolated branch and remember the patch (DPM).
From source to governed table in four steps.
The same loop every run — no notebooks, no hand-rolled orchestration glue.
Connect
Point SpooqW at Postgres, Kafka, JDBC or object storage. Secrets are encrypted at rest; nothing is stored in the clear.
Build
Draw the pipeline on the canvas or write YAML. Predictive dry-run samples each step against the live source before you spend Spark.
Gate
Every run executes on an isolated Nessie branch and must clear its quality rules — uniqueness, freshness, accepted values and more.
Ship
Pass → an atomic, metadata-only merge to main, queryable from Trino instantly. Fail → the data stays quarantined. Zero blast radius.
Bring your data in seconds — no VPN, no whitelisting.
On the cloud tiers, SpooqW provisions a managed Postgres for you the moment you need one. No VPN tunnel, no IP whitelist request, no firewall ticket — just a connection string.
- A managed Postgres, provisioned on demand — spin one up from the Connections page and get a live database in seconds — it's registered automatically as a pipeline source, ready to build on.
- Load data with any client — it's a real Postgres endpoint, not a proprietary API: psql, dbt, your ORM or your own app all just work.
- Credentials you hold, not us — the connection string is shown once at creation and never stored in the clear. Need new ones? Delete and re-provision — the old database and role are dropped instantly.
- Prefer your own network? Connect directly. — on-prem deployments — and cloud projects that already run their own databases — point SpooqW straight at Postgres, Kafka or any JDBC source inside your VPC, with an MCP endpoint so AI agents can query the governed lakehouse too.
A typed Scala engine. An Iceberg lakehouse underneath.
- MERGE INTO upserts and time-travel — merge by key, read any snapshot, branch or tag. ACID by construction, queryable from Trino the moment it lands.
- YAML or visual canvas — same engine — every step is typed and validated, from JDBC and Kafka ingestion to JSON/Avro parsing, SQL variables and UDFs.
- Real metrics, per step — rows, bytes, shuffle, memory peaks and wall-clock per step, pulled from the live Spark API. No estimates.
- CDC-ready ingestion — Kafka availableNow draining and native change-log readers, scheduled on your cadence.
AI builds the pipeline. Autopilot keeps it alive.
Describe what you want and the copilot compiles typed Spark steps. From then on an agent watches every run — and its fixes are proven on isolated branches, never guessed.
- Pipelines from plain language — the AI profiles your source, generates the steps with quality checks included, and edits the canvas on instruction — “add a dedup after orders and write to Iceberg”.
- Self-healing on the night shift — failed runs, schema drift and Spark inefficiencies become deduplicated incidents; transient failures are re-validated on a branch and closed without waking anyone.
- DPM — patch memory that learns — every validated fix is fingerprinted and stored with its success history, so the same incident is resolved faster next time — even on another pipeline.
- Pull requests, not silent changes — the default policy acts on branch + PR; autonomy is a per-project dial from observe-only to auto-merge, and every decision is audited.
Governance your platform team will actually sign off on.
SSO & roles
OIDC with PKCE, JWKS validation and group-to-role mapping. Admin, operator and viewer roles, project-scoped memberships.
Encrypted secrets
Connection credentials encrypted at rest with AES-256-GCM, masked in every API response, log line and audit record.
Audit trail
Every operational and admin action recorded — who ran what, who changed which config, who merged which branch.
Data contracts
Schema-drift radar diffs pipeline expectations against the live source; contract gates block runs before they burn compute.
GitOps promotion
Promote pipelines through Git artifacts with contract gates and rollback metadata. PRs, not dashboards, are the source of truth.
Cloud or on-prem
The same platform runs as managed cloud or fully self-hosted in your VPC — with your own Spark, your own object store.
Start free. Upgrade when you grow.
Every plan includes the visual builder, the Scala engine, quality gates and run observability.
Free
Kick the tires. Very limited, forever free.
- 1 project
- 1 seat
- 2 connectors
- 3 pipelines
Medium
Growing data teams that need governance.
- 10 projects
- 10 seats
- 50 connectors
- 100 pipelines
On-Premise
Your infra, your data. Unlimited, license-gated.
- Unlimited everything
- Air-gapped install
- Dedicated support
Questions teams ask first.
Do I need to run my own Spark?+
No. The cloud tiers include a managed Spark runner. On-prem runs entirely in your VPC against your own Spark, object store and catalog.
What does the Autopilot actually do at night?+
It watches runs, drift and Spark inefficiencies, turns problems into deduplicated incidents, and tests every candidate fix on an isolated branch. Validated fixes arrive as pull requests — or self-heal transient failures — and its patch memory (DPM) means the same incident gets resolved faster next time. It never silently touches main.
Is my data locked in?+
Never. Tables are open Apache Iceberg on your object storage, queryable from Trino, Spark or any Iceberg client — with or without SpooqW.
What happens when a run fails a quality gate?+
Its output is written to an isolated branch and quarantined — main is untouched. You see the exact failing rule and can fix and re-run.
Can I start without a credit card?+
Yes. The Free plan gives you 3 pipelines, 2 connections and a managed runner forever. Upgrade only when you outgrow it.
How do pipelines get promoted to production?+
Through GitOps: SpooqW emits Git artifacts with contract gates and rollback metadata, so changes ship as reviewable pull requests.
Which sources are supported?+
Postgres, Kafka, JDBC databases, object storage and more, plus CDC-style change-log ingestion — all through typed, validated steps.
How do I get my data into the cloud version?+
Provision a managed Postgres from the Connections page — no VPN or IP whitelist needed — and load it with any client: psql, dbt, your ORM or your own app. It's registered as a pipeline source automatically. Running on-prem or already have a database? Connect it directly, or query the lakehouse over MCP from an AI agent.
Is the hosted database mine?+
Yes. It's a plain, open Postgres database — no proprietary format, no export tax. Dump it with pg_dump, point another tool at it, or leave anytime; nothing about it locks you into SpooqW.
From zero to a governed lakehouse pipeline in one afternoon.
Connect a source, draw the pipeline, watch the dry-run, ship behind a quality gate. The Free plan is enough to prove it.