Pipelines that build with AI and heal on their own.
Describe the pipeline you want and the copilot generates the typed steps. Once it runs, Autopilot takes the night shift: it detects incidents, proves the fix on an isolated branch, and only then proposes it — as a pull request, never a silent change.
Generate pipelines from a sentence
Point the builder at a source, describe the goal — the AI profiles your tables and compiles typed Spark steps, quality checks included. Edit the canvas in plain language afterwards: “add a dedup after orders and write to Iceberg”.
Describe a dashboard, get the pipeline
Analytics Studio turns one sentence into the feeding pipeline and a live, shareable dashboard. The pipeline it generates is a normal SpooqW pipeline — governed, gated and observable like everything else.
An agent on the night shift
Autopilot runs an Observe → Orient → Decide → Act loop with three sensors: failed runs (root-cause classified), schema drift, and Spark optimization gaps. Every problem becomes a deduplicated incident with severity and a full lifecycle — nothing gets re-reported every five minutes.
Fixes are proven, not guessed
The agent never trusts an AI suggestion blindly. Every remediation is executed on an isolated Nessie branch, contract-gated, without touching main or the saved pipeline. If the re-run passes, the incident self-heals; if not, it escalates to a human after a bounded number of attempts.
DPM — patch memory that learns
Data Pipeline Management remembers every fix: incidents are fingerprinted into signatures, and validated patches are stored with their success history. The next time the same failure appears — even on another pipeline — the known patch is replayed and re-validated first. Your operations get faster with every incident.
Autonomy is a policy, not a default
Per project you decide how far the agent goes: observe only, suggest, act on branch + pull request (the default), or auto-merge on dev. PRs — not silent changes — are how fixes reach production, and every decision lands in the audit trail and the live timeline.
A planner that learns your workloads
The learning execution planner regresses runtime against input size from your real run history, recommends shuffle partitions and executor memory, and predicts runtime and cost with a confidence score. Explainable per choice — not a black box.