Why pure SQL suits a small team

Most data pipeline tooling was designed by and for organisations with a data platform team. That is not a criticism of the tools — they solve real problems at the scale they were built for. It is a statement about fit. A company of forty people with one engineer who also looks after data is not a small version of a company with a platform team; it is a different situation, and the thing that sinks it is rarely the hard part of the problem.

Data Conductor's position is that for that company, the pipeline should be SQL and nothing else. Here is the practical case, and the honest limit.

The argument is about what you don't take on

SQL is not better than Python at transforming data. The argument is not about expressiveness. It is about everything that arrives alongside a general-purpose language when you only wanted to move some rows.

A language your team already has. Your analysts write SQL. Your backend engineers write SQL. Nobody needs convincing, trained, or hired. When the pipeline is SQL, the set of people who can read it at 3am is everyone who already understands the data, rather than the subset who also know the framework.

No runtime of your own to keep alive. The queries run on the warehouse or database you already pay for. There is no worker pool to size, no dependency set to keep current, no environment that works on one machine and not another. The compute is the one you were already operating.

Fewer places for a change to hide. When a step is one query, reading it tells you what it does. There is no base class, no shared helper three directories away, no decorator changing the behaviour of the function underneath it. That matters most at the moment you care least about elegance — something is wrong in production and you did not write it.

A hire you can actually make. "Someone who writes good SQL" is a person you can find. A data platform engineer is scarcer, more expensive, and — if they are the only one — a single point of failure wearing a salary.

What you still get, which is the real question

Dropping the framework is only sensible if you do not also drop the things the framework was giving you. The ones that matter to a small team:

  • Versions. Every step and every pipeline is versioned, and a deployment pins the version it runs. Editing a deployed pipeline creates a new version rather than rewriting the running one. See Deployments.
  • Environments. The same pipeline runs against QA or PROD values without being rebuilt or copied, because variables are per-environment.
  • Dependencies. Steps chain, with conditions on the edges when a step should only run sometimes.
  • Checks. Assertions gate a run and can warn, pause or stop it.
  • A record. Run history, per-step timings, error text, and an audit trail of what the agent did on your behalf.

That list is the actual content of "a data engineering practice". None of it requires a new language; all of it requires somebody to have built it once.

What this is not

It is not an argument against dbt. dbt is also SQL-first, it is good at what it does, and if you are already running it happily this page is not trying to move you. The honest comparison is not about the language, because both answers are SQL. It is about what you stand up around it: dbt gives you a modelling framework, lineage and tests, and you supply the place it runs, the schedule, the monitoring and the project conventions. Data Conductor supplies those and gives you less modelling structure. Which trade is right depends on whether you have someone whose job is to own the first set.

It is not "SQL can do everything". It cannot, and pretending otherwise would waste your time. See the limits below.

It is not a rejection of code. Pipelines here are deployable by API, so a system you have written can trigger one and read the result. The boundary is that the pipeline's own work is SQL, not that code is unwelcome at the edges.

Where SQL is the wrong tool

Two of these are not arguments about our product. They are arguments about SQL, and they would still be true of any SQL-first tool.

  • Machine learning. Training a model, managing a model artefact, evaluating one. This is a different discipline with its own tooling, and a pipeline tool that claimed it would be wasting your time.
  • Genuinely procedural work. Loops carrying state, recursive traversal of something your warehouse does not already understand as a tree, parsing an awkward text format row by row. Warehouses do have procedural extensions; if you find yourself reaching for them for the whole job, the job is not a SQL pipeline.

If most of your work looks like that, be sceptical of this page.

Where the product stops today

These two are different. They are not statements about SQL — they are the current edges of what we have built, and we treat them as edges to be narrowed.

  • Calling an API. Fetching from a third-party service, paginating, handling its rate limit. A pipeline step cannot do this today, so ingestion from services you do not own happens outside Data Conductor and lands in a database you do own.
  • Very large reshaping between systems. Moving data across systems is supported — see Moving data between sources — but buffered transfers are capped per run, and past some size you want your warehouse's own bulk loading rather than any intermediary.

The distinction matters because of a failure mode we would rather avoid: a team that is well served by SQL for ninety per cent of its work, blocked by one small thing that exists elsewhere, and told to adopt a second platform for the remainder. That is a bad answer to a common situation. The better one is usually a modest version of the adjacent capability — enough to unblock the ordinary case, not a competitor to the specialist tool.

So the right way to read this section is as a description of today, not a position. What it will not do is describe work in the present tense or promise dates: the two limits above are real right now, and this page gets shorter by being edited when that changes.

So what is actually left

The useful question is not whether any of the four ever appear. It is whether they are most of your work. For the teams this is built for they usually are not: the bulk is joining things that are already in a database and putting the result somewhere a person can see it, on a schedule, reliably. That part is SQL, and it is also the part that breaks.

A reasonable way to decide

Three questions, in order:

  1. Is your data already in databases you control? If most of it arrives by API from services you do not own, your first problem is ingestion, not transformation.
  2. Does anyone own your data platform as their actual job? If yes, a framework will repay the operating cost, because somebody is there to pay it. If no, that cost comes out of the time of whoever is nearest.
  3. Could a new joiner read one of your pipelines and say what it does? If the honest answer is no, that is the cost you are already paying, and it compounds.

If the answers are yes, no, and not-really, pure SQL is likely the right shape — and that combination describes most companies between ten and three hundred people.

Where to start