Skip to content
DataSquares
Start free

DataSquares Pipelines

Move and shape data between your systems

Build and schedule data pipelines on a visual canvas. Extract, transform and load across your systems, with a clear timeline for every run.

Start freeBook a demo
Read the Pipelines documentation
  • Extract, transform, route and load steps on a visual canvas
  • Schedules, plus file-arrival, upstream-pipeline and webhook triggers
  • Assert steps that fail a run or quarantine the rows that break a rule

Pipelines is an add-on on every plan, in three sizes. A company adds it on the Billing page in the app. See plans

Every run, explained

Each run keeps a per-step timeline with rows, bytes, attempts and logs, with credentials scrubbed. A failed run names the step, the cause in plain words, and where to make the fix.

  • Retry, cancel, and backfill a date window as one run per date
  • Choose whether an overlapping run is skipped, queued or takes over
  • Failure events to Slack, Microsoft Teams or your own endpoint

What you can do with Pipelines

  • Many ways in

    Tables and SQL from databases, files from cloud storage, paginated REST APIs, change data capture from Postgres, and curated connectors for Salesforce, HubSpot, Stripe, Google Analytics 4, Shopify and Jira.

  • Transforms you can read

    Filter, join, aggregate, pivot, window and more as declarative operations, or raw SQL when you need it. A preview shows the compiled SQL and sample rows.

  • Quality gates in the flow

    An assert step checks rules and unique keys. Failing rows either fail the step before anything is published, or flow to a quarantine port for inspection.

  • Loads that respect your tables

    Replace, append or merge by key into managed datasets or your own database or warehouse, or write Parquet, CSV and JSON files. Replace keeps your table’s indexes, constraints and grants.

  • Schedules and triggers

    Hourly, daily, weekly, cron with a timezone, or micro-batch. Triggers fire when files land in a folder, when an upstream pipeline finishes, or when an external system calls the pipeline’s webhook URL.

  • Pipelines as code

    Export a pipeline as a deterministic file with no ids, data or credentials in it, review the change as a diff, and promote it between environments.
Visual
Extract, transform, route and load on one canvas
Per step
Timelines, logs and diagnosis on every run
Governed
Activations keep row-level security and masking

Inside Pipelines

Change data capture

Capture inserts, updates and deletes from PostgreSQL-family databases and merge them into your destination on a schedule you set. It runs as frequent micro-batches, not a continuous stream.

  • Pair a CDC stream with a merge load to keep an exact replica
  • Avoid re-extracting whole tables on every run
  • Streams are managed under Pipelines, CDC streams

Governed data, sent back out

Activations send governed metrics and managed datasets to other systems on a schedule, as signed, batched JSON. Row-level security applies, and masked columns are refused.

  • Metrics evaluated under a pinned run-as identity
  • Managed datasets read through tenant-scoped drivers
  • Configured under Pipelines, Activations

How it works

  1. Step 1: Start a pipeline

    Name it, or begin from a template

  2. Step 2: Add steps

    Extract, transform, route and load

  3. Step 3: Schedule or trigger it

    Cron, micro-batch, file arrival or a webhook

  4. Step 4: Watch the runs

    Per-step timelines, logs and diagnosis

Questions, answered

Something else? Ask our team or search the documentation.

What happens when a run fails?
The run names the step that failed, the cause in plain words and the fix. Retry re-runs the pipeline from the start, as there is no resume from a failed step, so use an incremental extract for expensive loads.
Can a pipeline write back into our own systems?
Yes. Load into your own Postgres-compatible, MySQL-compatible or SQL Server databases, or write Parquet, CSV or JSON files to your own storage.
Can the AI build a pipeline?
Yes, with your approval. Describe the pipeline in AI Chat and it shows one approval card with every step, each destination and the schedule in plain words before anything is created.

Works well with Pipelines

All products
IntegrateDataSquares ConnectConnect the systems you already runStoreDataSquares WarehouseA dedicated home for analytics dataGovernDataSquares Data QualityCheck your data before you publish

See Pipelines on your own data.

Start a free trial in minutes, or book a walkthrough with our team.

Start freeBook a demo