Ingestion & workflows
Write data pulls as workflows and land the results as immutable snapshots. Every refresh leaves a version behind.
A sentence you have heard
Can we rerun last month's report on the data as it was then?
Why it happens
Data moves by manual export and spreadsheet, in files named “final” and “final 2”. When the data changes nobody can reproduce what it looked like, and when an upstream column is added, a downstream row quietly goes missing.
How Shuzi does it
However messy the source, it lands the same way.
Databases, APIs and uploaded files are pulled, transformed and merged by workflows, then landed as immutable, versioned snapshots. Every refresh is a new version, and a bad one can be rolled back.
Sources
- PostgreSQL
- SQL Server
- REST API
- CSV · XLSX
Workflow
- 01Extract
- 02Transform
- 03Merge by key
Snapshots
- v13parquet immutable
- v12parquet
- v11parquet
snapshot v13 · parquet · immutable
- 01
Workflow-based pulls
Orchestrate pulls and transforms visually. Data from different sources is first synced into managed snapshots, then analyzed together.
- 02
Immutable versions
A released snapshot is never edited. It can be traced and rolled back, and a referenced version is never deleted by mistake.
- 03
Separated reads and writes
Only the scheduler writes; queries read a read-only replica. The API never changes data behind your back.
In detail
Many sources
Databases, file upload and API pulls, with SSH jump hosts for production databases.
Primary-key rules
Primary-key columns cannot be null, so a batch with mismatched column names cannot silently merge into one row.
Incremental merge
Merge incrementally by primary key, schedule refreshes, and see progress and failure reasons.
Version references
References from models, queries and exports are recorded, so cleanup never deletes a version in use.
Take your datafrom tangled to clear.
Try it on sample data first, then bring your own.