The Pipeline That Worked Until It Didn't
You have built a nightly ETL job that pulls daily sales files from a partner's SFTP server, normalizes the columns, and loads the result into a warehouse. For months the pipeline ran without a hiccup, but this week the partner changed the CSV delimiter from a comma to a pipe and added a new optional column for promotional codes. Within minutes the job failed, the alert flooded your Slack channel, and the downstream reporting team was left with incomplete dashboards. The same pattern repeats when you try to ingest PDF invoices from a new vendor, or when a regulator asks for a manual review of every high-value transaction before it is posted. Your deterministic ETL code is solid, but the world around it is no longer stable.
What Traditional ETL Does Well
Traditional ETL pipelines excel when the data source, schema, and transformation logic are well defined and change rarely. A deterministic series of extract-transform-load steps can be versioned, unit-tested, and reproduced across environments. Because each step is explicit, a data engineer can trace a row from source to destination, audit the logic, and guarantee that the same input always yields the same output. Tools such as Airflow or Prefect provide reliable scheduling, retry policies, and clear dependency graphs, making it easy to orchestrate hundreds of jobs without surprising side effects. When you need to move large volumes of structured data from a stable ERP system into a data lake, a classic ETL pipeline delivers the predictability that business stakeholders demand.
When schemas are stable, volumes are high, and the transformation logic fits neatly into SQL or Python, there is no reason to introduce additional complexity. The deterministic path has decades of tooling, documentation, and engineering culture behind it.
Where ETL Breaks Down
The deterministic nature of ETL becomes a liability when the source data is ambiguous. Consider a stream of semi-structured logs where the same event type appears under different field names, or a batch of PDF invoices where the line-item table is rendered with varying column orders. A rule-based parser must be constantly updated to handle each new variation, and every change introduces a regression risk. The maintenance burden grows as edge cases multiply.
Complex validation logic also strains traditional pipelines. In a financial institution, a tax calculation must satisfy both internal policy and external regulatory rules. A deterministic script can encode the known formulas, but when a new tax jurisdiction is added, the script must be rewritten and retested. If the validation fails, the pipeline either drops the record silently or raises an exception that halts the entire batch, both of which are undesirable in a high-throughput environment.
Human-in-the-loop gates are another blind spot. Regulations may require a compliance officer to approve any transaction over a certain threshold. Classic ETL frameworks have no native concept of a pause for manual review. Engineers must build custom pause-and-resume mechanisms using ad-hoc database flags or external ticketing systems. These workarounds are fragile, hard to monitor, and add latency that is difficult to quantify. The cumulative cost shows up as increased manual intervention, higher on-call fatigue, and a growing body of special-case code that no one feels comfortable touching.
Where Agentic Approaches Add Value
Agentic workflows address ambiguity by design. Large language models excel at interpreting unstructured or semi-structured inputs, extracting fields from PDFs, normalizing free-text categories, and generating schema suggestions on the fly. When an LLM is embedded in a pipeline node, it acts as a flexible parser that adapts to new document layouts without a code change. The model's probabilistic output is not a weakness here -- it becomes a source of confidence scores that downstream nodes can use to decide whether to accept, reject, or flag a record.
Self-validation layers, often called the maker-checker pattern, turn the deterministic-agentic split into a safety net. In a maker-checker node, the traditional code (maker) produces a result -- say, a calculated tax amount -- while an LLM (checker) independently recomputes the same value based on the raw inputs and the relevant policy text. If the two results diverge beyond a tolerance threshold, the record is routed to a human reviewer. In Labyrinth's own 19-node financial pipeline that aggregates data from seven sources, this pattern caught a miscalculation that had escaped unit tests for months. The deterministic code had applied an outdated tax rate; the LLM, working from current policy text, flagged the discrepancy; and the human reviewer confirmed the code was wrong.
Adaptive routing reduces the impact of schema drift. Using LangGraph as the orchestration framework, each node can emit a state object that includes a schema-version field. Conditional edges in the graph evaluate this field and route the payload to a transformation node that knows how to handle that version. When a new column appears, the pipeline automatically follows the edge to a schema-evolution node that updates the internal mapping and then rejoins the main flow. Human-in-the-loop checkpoints become first-class citizens in this model -- LangGraph supports explicit pause nodes that expose the current state to a review interface, where a compliance officer can approve or reject a transaction. The graph resumes execution once the decision is recorded.
When to Use Each Approach
Choosing between pure ETL, a hybrid augmentation, or a full agentic pipeline starts with an honest assessment of data stability. If your source schemas have changed less than once a quarter over the past year and the transformations are simple aggregations, the deterministic path remains the most cost-effective. When the data source is a mixture of structured files and unstructured documents, and the schema evolves with each new vendor, the risk of constant re-coding pushes the balance toward adding agentic nodes.
Validation complexity is the second axis. Straightforward type checks and key constraints are well handled by ETL. When business rules involve natural-language policy documents, multi-jurisdictional tax tables, or fuzzy matching of customer names, an LLM-based checker can reduce the engineering effort substantially. However, each LLM call adds latency and compute cost, so the organization must be comfortable with that trade-off before committing to the approach.
Human review requirements often act as the deciding factor. If regulations mandate manual sign-off for a subset of records, embedding a maker-checker node with a LangGraph pause checkpoint is far cheaper than building a custom Airflow sensor that polls a ticketing API. The presence of such gates typically justifies a hybrid design: keep the bulk of the pipeline deterministic, but insert agentic validation and review stages where the policy demands flexibility. Cost tolerance and team expertise matter here too -- deploying LangGraph and maintaining LLM prompts requires a different skill set than writing SQL scripts. Starting with a single agentic validation node as a proof of concept, measuring the ROI, and deciding whether to expand is a lower-risk path than committing to a full rewrite.
When Agentic Is Overkill
Agentic workflows are not a silver bullet. Running an LLM for every row of a ten-million-record import can increase processing time by an order of magnitude and raise cloud spend significantly. For a simple order import that receives a CSV with a stable schema from a trusted vendor, the deterministic ETL path remains the most efficient. Over-engineering an agentic layer in that scenario adds latency, introduces a new failure mode if the model service is unavailable, and creates operational overhead for prompt versioning. The key is matching the tool to the problem, not assuming that every pipeline should be AI-enhanced.
The teams that get the most value from agentic consulting tend to have at least one pipeline where the existing approach requires more than 20% manual intervention, or where schema drift has caused at least one significant incident in the past six months. If neither of those conditions applies, the current setup is probably working well enough.
Adding Agentic Capabilities to Existing Pipelines
Most organizations do not rip out their existing Airflow DAGs. Instead, they layer agentic capabilities on top of the proven foundation. A common pattern is to insert a maker-checker validation stage after a deterministic transformation node. In a recent Labyrinth engagement, we wrapped a legacy revenue-recognition script inside a LangGraph node that first executed the original code and then called an LLM to verify the recognized revenue against contract language. Disagreements were routed to a review channel where a finance analyst could approve or correct the entry. The rest of the DAG continued unchanged, preserving the existing scheduling, alerting, and logging infrastructure.
Another approach is to replace a single brittle parser with an LLM node while keeping the surrounding orchestration intact. For a pipeline that ingested PDF invoices, we replaced a custom OCR script with a LangGraph node that called a vision-capable LLM to extract line items. The node emitted a structured JSON payload that downstream Airflow tasks consumed without modification. This incremental upgrade eliminated a source of nightly failures without requiring a rewrite of the broader system.
These augmentation strategies let teams modernize their data pipelines at a measured pace. The pilot-node approach -- start with one agentic stage, measure success, and expand incrementally -- has consistently produced better outcomes than wholesale rewrites in the engagements we have seen.
What Comes Next
If you are evaluating whether your data pipeline is a good candidate for agentic augmentation, our team can help you map this decision framework to your specific workloads, design a hybrid architecture, and implement the LangGraph components that fit your regulatory and operational requirements. See our data pipeline design and agentic AI workflow services at /services.