A coding assistant is relatively easy to trust.
Ask it to explain a SQL query. Generate a test. Suggest a transformation. If the answer looks wrong, ignore it and move on.
The architecture becomes much more interesting when the assistant can do something.
Suppose it can inspect pipeline logs, search a repository, trace dependencies, modify code, run tests, rerun a failed task, or propose a production remediation.
At that point, the model is no longer the architecture.
It is one component inside a much larger engineering system.
And I think this is where many discussions about AI-Augmented Data Engineering become too simple.
The production question is not:
Which model should we use?
It is:
What has to exist around the model before we allow it to participate safely in an engineering workflow?
That is the architecture I want to explore here.
AI-Augmented Data Engineering changes when the assistant gets tools
In the first article in this series, I looked at AI-Augmented Data Engineering across the modern data life-cycle and described a progression that becomes increasingly important as AI takes on more responsibility:
Assist → Reason → Recommend → Act → Verify
The early stages are relatively forgiving.
If an assistant explains a failed query incorrectly, an engineer can reject the explanation.
If it generates poor SQL, code review should catch it.
But once the assistant can execute a tool, the consequences change.
A recommendation can now become a code modification.
A diagnosis can become a pipeline rerun.
A schema observation can become a production change.
That means we need to separate two questions that often get mixed together:
Can the assistant work out what should happen?
and
Is the assistant allowed to make it happen?
Those are not the same architectural decision.
A capable model does not automatically deserve production authority.
Start with a real engineering scenario
Imagine our fictional retailer, NovaRetail.
Its order platform sends change events into a data platform using Change Data Capture.
A normal flow might look roughly like this:
Order Application
→ CDC
→ Raw Order Events
→ Transformation
→ Curated Orders
→ Inventory Planning / Reporting / Customer Analytics
Now imagine that an overnight order-processing job fails.
An engineer asks the AI assistant:
“Why did the order pipeline fail, and what should we do?”
A basic assistant might see the error message and suggest some code.
A production-grade engineering assistant needs a much wider view.
It may need the pipeline definition, repository, schema history, recent commits, execution logs, dependency information, test results, deployment history, engineering standards, and information about what it is allowed to inspect or execute.
Even then, I would not want the path to be:
Prompt → AI → Production
A more realistic architecture is:
Engineer → AI Assistant → Engineering Context → Permitted Tools → Controlled Execution → Validation → Approval → Action → Verification → Audit
The distinction matters.
Every layer exists because something can go wrong.
The engineer should remain the source of intent
The workflow starts with a person or another authorized system expressing an intent.
That sounds obvious, but intent should not be reduced to a free-text prompt.
“Fix the pipeline” is not enough.
The assistant needs to understand the objective and the boundaries around it.
Are we investigating only?
Can it generate a candidate change?
Can it run a test?
Can it open a pull request?
Can it rerun a development pipeline?
Can it touch production?
A useful architecture carries those boundaries into the workflow rather than hoping the model infers them.
I would treat the initial request as a combination of:
intent + scope + environment + permitted outcome
For example:
Investigate the failed NovaRetail order pipeline.
You may inspect logs, lineage, schema history and repository code.
You may propose a fix and run tests in the development environment.
Do not modify production.
That is much more useful than a cleverer prompt.
It tells the surrounding system what authority this particular interaction has.
The AI assistant is an orchestrator, not the whole system
The assistant sits at the centre of the interaction, but I would resist designing the platform as though the model itself owns every responsibility.
The model is good at interpreting language, connecting evidence, proposing explanations and choosing among permitted next steps.
It should not have to become the system of record for permissions, lineage, execution history, approvals or audit evidence.
Those responsibilities already belong elsewhere.
In practice, the assistant may need to coordinate several capabilities:
repository search, metadata lookup, pipeline history, log retrieval, test execution, dependency analysis and perhaps controlled code or orchestration tools.
Its role is to reason across those capabilities.
That is very different from giving a model broad credentials and asking it to “operate the platform.”
Context is where generic assistance becomes engineering assistance
This has been one of the strongest themes for me throughout this series.
An assistant seeing only the prompt can help write code.
An assistant seeing the schema can write more relevant code.
An assistant seeing the repository can understand how this particular platform is built.
Add execution history and it can compare good and bad runs.
Add dependencies and it can reason about downstream impact.
Add engineering standards and it can propose something that fits the platform rather than merely something that works.
This is why the context layer deserves to be treated as a first-class architectural component.
For NovaRetail, that layer might expose:
the current pipeline definition, schema history, recent code changes, runtime evidence, upstream and downstream dependencies, data-quality checks, relevant documentation and environment-specific standards.
But there is a subtle issue here.
More context is not automatically better context.
A runtime metric saying a job took 47 minutes is measured evidence.
A ticket saying “the source team probably changed the schema” is an assumption.
A passing test tells us a defined condition passed.
A contract tells us what a producer and consumer expected.
If all of these arrive as undifferentiated text, the assistant can quietly treat them as equally authoritative.
So I would want context to carry more than content.
It should carry enough information to answer questions such as:
Where did this statement come from?
Was it observed, inferred or asserted?
Has somebody validated it?
Who owns the underlying definition?
Has its basis changed?
This does not require inventing an entirely new metadata universe.
Many data platforms already generate much of this information.
The architectural opportunity is to make it available to the reasoning process without flattening everything into the same kind of text.
Tool access should be explicit and narrow
Once the assistant understands the problem, it may need tools.
This is where I would become very deliberate.
There is a big difference between:
reading pipeline logs,
running a read-only metadata query,
modifying a branch,
opening a pull request,
rerunning a development pipeline,
and altering a production schema.
A practical architecture should expose tools with clearly bounded capabilities rather than handing the assistant a generic administrative identity.
I would much rather provide several narrow tools:
get_pipeline_logs
get_schema_history
run_test_suite
create_candidate_change
open_pull_request
rerun_nonprod_task
than one tool effectively called:
do_whatever_you_need
The first design makes authority visible.
The second makes the model responsible for discovering its own boundaries.
That is not a control model I would be comfortable with.
The execution environment matters as much as the tool
A permitted action should also be tied to an environment.
An assistant being allowed to run code does not mean it should be allowed to run it everywhere.
Development, test, staging and production should remain meaningfully different.
For many organizations, a sensible path will be asymmetric.
The assistant may be allowed to act quite freely in a disposable or development environment.
In production, the same action may require explicit approval—or may not be exposed at all.
This gives us a useful pattern:
freedom to experiment where failure is cheap; stronger controls where failure is expensive.
That sounds less exciting than “autonomous Data Engineering.”
It is also much closer to how I would actually introduce this capability into a platform.
Validation should happen before approval, not only after execution
Suppose the NovaRetail assistant identifies a likely cause.
An upstream order field changed from an integer representation to a string representation, and one transformation still expects the old type.
The assistant proposes a change.
Before anyone approves that change for production, the platform can already do useful work.
Run unit tests.
Execute the affected transformation against representative data.
Compare row counts.
Check expected schema.
Evaluate known quality rules.
Identify affected downstream assets.
Show the diff.
Estimate whether the remediation touches anything outside the agreed scope.
The important idea is that approval should be informed by evidence.
An engineer should not be approving a paragraph generated by a model.
They should be approving a proposed action accompanied by enough evidence to understand why it was recommended and what happened when it was tested.
This changes the human role from rubber-stamping AI output to reviewing an engineering decision.
Verified context does not automatically authorize an action
There is another distinction I think becomes important once we move towards production.
Suppose the assistant has strong evidence.
The schema definitely changed.
The transformation definitely failed afterwards.
The proposed fix definitely passes the development tests.
That still does not mean the assistant is authorized to deploy it.
Evidence supports the recommendation. It does not grant authority.
The action itself needs an admission decision.
Depending on the organization and the consequence, that might be an engineer approving a pull request, an operational owner authorizing a rerun, or a policy allowing a low-risk action automatically.
I would record that decision separately:
What action was proposed?
What evidence supported it?
Who or what admitted the action?
Under which policy?
What exactly was executed?
That creates a clean boundary between reasoning and authority.
It is also much easier to audit than trying to reconstruct afterwards why an AI system changed something.
Approval should depend on consequence, not novelty
Not every AI-generated action needs a human sitting in front of a button.
That would simply turn automation into bureaucracy.
The control should reflect the consequence of being wrong.
An assistant regenerating documentation does not need the same approval path as one dropping a table.
A non-production pipeline retry does not have the same blast radius as a production schema migration.
A pull request is different from a direct deployment.
This is where policies become useful.
Low-risk, reversible actions with strong validation may eventually be admitted automatically.
Higher-risk actions may require one or more human approvals.
Some actions may remain prohibited regardless of model capability.
The architecture therefore needs something between “the AI wants to do this” and “execute it.”
Call it a policy layer, approval service, action gateway or control plane.
The name matters less than the responsibility.
It decides whether a proposed action is admissible.
Production execution should be boring
If an action is approved, I would want execution to look as ordinary as possible.
Use the same deployment pipeline.
The same orchestration system.
The same service identities.
The same change controls.
The same logging.
The same rollback mechanisms.
AI should not create a parallel production pathway simply because the change originated from an assistant.
If anything, AI-generated changes deserve more traceability, not less.
One of the architectural mistakes I would avoid is giving the model a privileged side door around the engineering platform.
The assistant should participate through controlled interfaces that the organization already knows how to operate.
That makes the AI component easier to replace and keeps operational responsibility in familiar systems.
Verification closes the loop
Execution success is not enough.
This is especially important in Data Engineering because a technically successful pipeline can still produce the wrong data.
After the NovaRetail change is deployed and the job reruns, verification should ask a different set of questions.
Did the pipeline complete?
Did the expected records return?
Did row counts behave as expected?
Did the previously failing transformation recover?
Did downstream datasets remain consistent?
Did business-level checks pass?
Did anything unexpected change?
This is the Verify stage from the Assist → Reason → Recommend → Act → Verify progression.
Without it, the architecture stops too early.
The assistant has performed an action.
The engineering system has not yet demonstrated that the action achieved the intended outcome.
Those are different things.
Auditability is not an afterthought
By this point, the workflow has produced a valuable chain of evidence.
We know what the engineer asked.
Which context was retrieved.
Which evidence influenced the diagnosis.
What the assistant recommended.
Which tools it called.
What validation ran.
Who approved the action.
What was executed.
And what happened afterwards.
I would preserve that chain.
Not because every organization needs a heavyweight governance programme around every AI interaction.
But because once AI participates in operational decisions, traceability becomes useful for ordinary engineering reasons.
Something will eventually go wrong.
When it does, “the agent decided to do it” is not a useful incident record.
We need to be able to reconstruct the path from intent to effect.
What happens when the assistant is wrong?
Reference architectures often look clean because every arrow represents a successful step.
Production systems are interesting because arrows fail.
The assistant can retrieve stale context.
A lineage graph can be incomplete.
A test can pass while the business meaning is wrong.
A recommendation can be based on a mistaken assumption.
A tool call can time out.
An approval can be granted for the wrong reason.
A deployment can succeed and create a downstream regression.
Verification itself can be incomplete.
The architecture therefore needs failure paths.
If evidence is weak, the assistant should be able to say so.
If two sources disagree, that disagreement should be visible.
If validation fails, the action should stop.
If approval is missing, execution should not occur.
If execution partially succeeds, the platform needs a known recovery path.
If verification fails after deployment, rollback or escalation should be possible.
This is one reason I am cautious about using autonomy as the main measure of maturity.
A system that knows when to stop can be more valuable than one that can do more things.
The practical architecture
If I reduce the design to its essential flow, it looks like this:
Engineer / authorized request
↓
AI assistant
↓
Engineering context and evidence
↓
Permitted tools
↓
Controlled execution environment
↓
Pre-action validation
↓
Policy / approval boundary
↓
Execution through normal platform controls
↓
Post-action verification
↓
Audit trail and feedback
The model is important.
But notice how much of the architecture is not the model.
That is the part I think will determine whether AI-Augmented Data Engineering becomes genuinely useful in production.
I would not build all of this on day one
There is also a practical trade-off.
At smaller scale, building a sophisticated context platform, fine-grained tool gateway, approval engine and immutable audit system may be far more infrastructure than the problem deserves.
Start with the level of capability you actually need.
A team using AI only for explanation and code generation may need little beyond existing review practices.
A team using AI for incident investigation needs reliable read-only access to operational evidence.
A team allowing automated non-production actions needs permissions, isolation and validation.
A team permitting production actions needs substantially stronger controls.
Architecture should grow with delegated authority.
Not with enthusiasm.
What I would build first
If I were introducing this into a real data platform today, I would begin with investigation rather than production autonomy.
Give the assistant read-only access to high-value engineering context.
Let it correlate logs, schemas, repository changes and dependencies.
Let it explain what it believes happened.
Make assumptions visible.
Then allow it to propose candidate changes.
Next, give it a controlled development environment in which it can test those proposals.
Only after the team understands the quality and failure modes of those workflows would I consider increasingly consequential actions.
That progression gives the organization something more useful than a demo.
It gives it evidence about where AI is actually reliable.
From copilot to production is mostly an architecture problem
The first generation of AI engineering tools made it easy to focus on the model.
That made sense when most interactions ended with generated text or code.
The moment the assistant gains tools, the surrounding architecture becomes more important.
Context determines what it understands.
Provenance helps us judge the evidence.
Permissions determine what it can touch.
Validation tells us whether a proposal survives contact with reality.
Approval determines whether the proposed action is admitted.
Execution controls the blast radius.
Verification tells us whether the outcome is actually correct.
Auditability lets us reconstruct the path afterwards.
The result is not an autonomous engineer hiding inside the platform.
It is an engineering system in which AI can take on more responsibility without removing the controls that make production engineering trustworthy.
That, for me, is the important shift.
The goal is not to ask:
How much can the AI do?
It is to ask:
What would have to be true before we safely let it do the next thing?