Making a film requires more than shooting a scene.
A complete production needs a script, cameras, lighting, sound, sets, actors, editing, visual effects, and distribution. Each discipline brings its own tools, language, and expertise. A filmmaker working alone creates within the boundaries of the jobs they personally know how to perform.
For most of my career, data engineering worked the same way.
I could shoot certain scenes. I could pull data from a source, write a Python script, build a pipeline, create a data model, or publish a report. Those were the parts of the production I understood.
Turning that work into a complete application required a much larger crew.
The natural ending was usually a pipeline, a model, or a report, because carrying the work further required backend development, interface design, testing, security, deployment, and maintenance.
Agentic coding is changing that boundary.
What looked like vibe coding a few months ago has become a serious production workflow. My role is starting to resemble a producer.
I decide what we are making and why it matters. I bring together the data and business context, establish the constraints, review the work, test the result, and decide what is ready to move forward.
The agent becomes the crew.
The individual crafts still matter. One person can now coordinate far more of them—and produce an experience that carries the data closer to a decision or action.
For the first time, a data engineer can reasonably produce the whole film.
That freedom creates a new challenge.
The rough cut can come together remarkably quickly. Releasing it inside an enterprise still requires security, governance, reliability, support, and ownership.
That is the transition I have been working through: how to preserve the speed and creative freedom of agentic development while producing systems an enterprise can understand, support, and trust.
The workflow I am moving toward separates production from distribution: build and prove the application quickly in a local studio, then move it into an enterprise platform that can govern, support, and distribute it.
AI has given data engineers a crew.
Now we have to learn how to run the production.
From Shooting Scenes to Producing the Film
The change is easy to underestimate.
I now write none of the code behind the applications I build. I am also opening fewer of the tools involved in building them.
That has become so routine that I sometimes miss how strange it is. From outside the set, AI coding can still look like autocomplete: someone begins writing a function, the model completes it, and the developer carries on as before.
My workflow looks more like a producer working from the production office.
I spend most of my time inside an agent harness such as Codex or Claude Code. From there, the agent moves between the different departments of the production. It writes the Python, SQL, and TypeScript. It calls the services, modifies the configuration, runs the tests, inspects the data, and revises the interface.
The same thing is happening with the control rooms behind the set. I can work with Power BI without opening Power BI to build the model. I can work with AWS without moving through a collection of AWS consoles. The agent operates against the files, APIs, repositories, and command-line tools underneath those interfaces.
The cameras still exist. The editing equipment still exists. Someone still has to light the set and assemble the footage.
I have stopped operating every piece of equipment myself.
That changes the scale of what I can produce.
The natural output of my old workflow was usually one part of the production. A Python script collected the data. A model organized it. A dashboard presented it. Each artifact completed its assigned scene, and then the user carried the story forward through an email, spreadsheet, meeting, or another system.
The project often ended at that handoff because continuing required a larger crew: backend development, frontend development, automated testing, infrastructure, deployment, and design.
The agent harness brings those capabilities onto the same set.
Now the script can lead into the model, the model can lead into the interface, and the interface can lead into an actual workflow. The result can collect the data, validate it, explain it, let someone interact with it, and help them take the next action.
The script is a scene. The model is a scene. The dashboard is a scene. The application is the film.
My role increasingly resembles the producer responsible for bringing that film together. I decide what we are making, why it matters, and which constraints shape the production. I provide the business context, evaluate the data, test the assumptions, review the experience, and decide what belongs in the final cut.
The agent supplies more of the crew. That gives me more time to focus on whether the data supports the story and whether the finished experience helps someone understand the problem, make a decision, and act.
The starting question changes with it.
I used to ask what I could realistically build with Python, Power BI, and the time available. Now I can begin with the outcome: What should the complete experience do?
My coding ability has stopped defining which scenes we can shoot. The interfaces of individual tools have stopped defining the shape of the production.
I am writing no code. I am operating less equipment. I am producing complete applications.
The opportunity is not simply to produce more software. It is to tell a more complete story with the data—and carry that story closer to a decision.
The Value Is in the Story
A dashboard is a camera pointed at a question.
Set it up correctly and it can capture the same scene every day. How many support tickets came in? Is the number rising or falling? Which team has the largest backlog?
That fixed shot is useful. A recurring business question benefits from a consistent angle. Everyone sees the same frame, the same measures, and the same comparison each time they return.
The limitation appears when the camera position begins defining the boundary of our curiosity.
The business usually wants to know what is happening outside the frame. What kind of tickets are increasing? Which customers are affected? Did the change begin after a product release? Are several teams describing the same underlying problem in different ways? Which issue deserves attention first?
That is where the story begins.
Raw data gives us footage.
It may contain thousands of tickets, customer records, release dates, comments, ownership changes, and resolution histories. Every record captures something that happened, but the events arrive without a plot. Their meaning lives in the relationships between them: what came before, what changed, who was affected, and how one event connects to another.
The job of analysis is to find that structure.
A useful story might begin with a rise in ticket volume, follow that signal into a particular category, connect the category to a recent release, and reveal that a small group of high-value customers accounts for most of the impact. The count provides the opening scene. The context reveals why the scene matters.
Data gives us the footage. Insight gives it a story.
For years, changing that story required a new production.
Someone had to locate the additional footage, bring it onto the set, organize it, check its continuity, edit it into the existing model, create a new view, and publish another version of the report. Each follow-up question introduced another round of work.
The cost of moving the camera encouraged us to keep asking the questions already in frame.
That shaped more than our reports. It shaped which business problems felt worth investigating. Stable, recurring questions earned dashboards. Messier questions—with several possible explanations, unfamiliar sources, or changing requirements—often remained in the backlog because the production cost exceeded the expected value.
Agentic development makes that exploration more flexible.
The same application can begin with the ticket count, follow the pattern into the underlying records, connect those records to customers and product releases, and help someone decide what to investigate next. As the question changes, the agent can bring in another source, test a different explanation, and revise the experience around what we are learning.
That flexibility still depends on trustworthy data. If a customer appears under two identifiers, a timestamp shifts time zones, or a join quietly duplicates a record, the story can fall apart. The agent can build and rerun checks as the work evolves, helping preserve the relationships the analysis depends on.
A report gives the story a consistent frame. An application can let the audience explore what happens next.
Applications also support more kinds of work.
Some business problems call for an investigation. Others need a planning exercise, a guided workflow, a simulation, or an operational control panel. A support application might help find the cause of an emerging issue. A planning application might let leaders change assumptions and compare possible futures. An operational application might surface an exception, explain its context, and assign the next action.
The technology available to us often narrowed those experiences into the format we knew how to produce. A chart and a filter became the answer because that was the experience the BI platform made practical.
Access to a larger crew lets the story choose the format.
The business value is not simply faster execution. Speed gives us more time to ask better questions, compare explanations, and carry the insight closer to a decision.
An application can also make the audience part of what happens next. Someone can explore a record, adjust an assumption, correct a classification, add context, assign an owner, or initiate the next step. That closes the distance between understanding what happened and deciding what to do about it.
The value compounds when the experience becomes repeatable. An analysis that once required eight hours of manual work can become available whenever the decision needs to be made. The business gains more than saved time; it gains a greater capacity to understand itself.
The data engineer consequently spends less time transporting data between systems and more time shaping questions, examining evidence, and improving the decision the application was meant to support.
A trustworthy data story follows the evidence, preserves uncertainty, and recognizes when several explanations remain possible. The agent can surface patterns and produce alternatives. Human judgment still determines which story the data actually supports.
The value is in the story we can finally tell with the data.
Once that story is worth telling, the next question is how to produce it without paying for distribution before it has proven its value.
Your Laptop Is the Studio
Most data applications begin with a sentence that sounds something like this:
“It would be so cool if we could…”
If we could connect the support tickets to recent product releases. If we could let someone explore different forecast assumptions. If we could bring several operational systems together and show what needs attention right now.
Those ideas used to carry an immediate cost. Exploring one meant finding the data, learning the source, writing the integration, choosing where everything would live, building an interface, and creating enough infrastructure to let someone see it. The idea often needed a budget and an architecture before anyone knew whether it was useful.
That kept many interesting questions hypothetical.
The surprising reality is that many data engineers already own enough equipment to develop the application.
It is sitting on our desks.
A modern laptop can store and query substantial datasets, run a database, host a backend, render a complete web application, and execute an automated test suite. For many early data applications, the available compute is already adequate. The limiting factor has often been the time and expertise required to assemble the software and operate the entire studio.
The equipment was there. The crew was missing.
That helps explain why development moved into managed tools and cloud platforms. The data went into a warehouse. The interface went into Power BI. The application went into a hosting service. Each platform supplied another piece of equipment we lacked the time to operate ourselves.
The cloud also reliably solved a problem a laptop could not solve for an enterprise on its own: distribution. It gave other people a way to find the work, authenticate, and depend on it from somewhere beyond the computer where it was created.
Over time, production and distribution became bundled together in our minds. We began renting the soundstage before we had filmed the first scene.
Agentic development changes that sequence.
Your laptop is already a film studio. AI finally gives you the crew to use it.
My local studio uses a small, consistent setup. DuckDB holds the data. FastAPI provides the backend. React provides the interface. Git preserves each take, and automated tests maintain continuity as the production changes.
Local does not mean ungoverned. The data still has to comply with organizational policy, and sensitive production data may need to be masked, sampled, or replaced with representative data.
The complete experience can take shape there.
The agent can inspect the footage, revise the logic, change the interface, run the tests, and review the evolving experience. A problem in the data can lead to a change in the model, which can lead to a different interaction, which can reveal a better way to tell the story.
The application running on my laptop can look and behave like the eventual product. I can use representative data—or approved real data—test the complete workflow, and put it through private screenings with the people helping shape it. The local version can continue changing until the experience does what we need it to do.
That turns “Wouldn’t it be cool if…” into something we can actually see and challenge.
The direct infrastructure economics are radically different from beginning in the cloud. The computer is already paid for, and the database, backend, interface, and tests can run without metered infrastructure. Developer time still matters, but one of the few new direct tooling costs is the agent helping operate the production.
A weak idea can disappear without leaving behind a database, storage account, app service, networking configuration, and a collection of recurring charges. A promising idea can earn another scene, another source, and another screening.
Local development lets the story prove its value before the organization pays to distribute it.
This is where the film analogy becomes especially useful. In software, we call the next step “going to production.” In this workflow, distribution is a useful parallel, although an enterprise release also requires hardening the application for operation.
The local version can already prove the story and the complete experience through private screenings. Reaching a wider audience requires packaging it, delivering it through an approved channel, controlling who can access it, and ensuring that every screening works.
Your laptop can produce the film. The cloud gives it an audience.
Consumer vibe-coding platforms often bundle production and distribution into one experience. They can quietly assemble a database, storage, authentication, compute, and hosting behind the scenes. The creator sees one application while the platform coordinates a collection of services underneath it.
That abstraction is powerful for a personal application. The film can reach an audience before the creator understands every piece of the distribution network.
An enterprise release brings a different standard.
The organization needs to understand the distribution agreement. Where is the film stored? Which outside services receive it? Who is allowed to watch it? How is sensitive material protected? What happens when a screening fails? Who keeps it available after the original producer moves on?
In software, those questions become data location, service dependencies, identity, permissions, security, reliability, monitoring, and ownership.
I ran directly into this while deploying an application to Azure Container Apps.
The container built. The application started. The private screening worked.
I had successfully packaged the film.
Then the distribution questions arrived.
The application still needed a durable place for its data, storage for files, authentication, secrets, network controls, monitoring, search, data-loss protections, and someone responsible for supporting it. Each answer introduced another service, another configuration, another approval, and another recurring cost.
A packaged film still needs a distributor. A packaged application still needs an environment the enterprise trusts.
The database decision captured the problem. PostgreSQL looked like the conventional choice for the application. Inside the enterprise, that choice also carried security, configuration, maintenance, and support requirements. The architecture I was evaluating for this particular application was approaching a four-figure monthly commitment before the application had established enough value to justify the release.
PostgreSQL remained a sound technical choice. The distribution budget had arrived before the audience had established the value of the film.
Agentic development can accelerate that mistake. An agent can recommend a database, scheduler, search service, ingestion process, API, and hosting environment in minutes. Each recommendation may be technically sound. Together, they can create an expensive distribution network around a film that is still finding its audience.
The cloud remains essential when the application needs reach, shared identity, governance, reliability, or scale. Those capabilities become worth paying for when the application has demonstrated value, needs a broader audience, and the organization needs to depend on it.
Until then, the local studio gives us a much cheaper place to produce, test, revise, and learn.
Produce locally. Distribute through the cloud.
The next step is choosing a distributor the organization already knows and trusts.
Use the Distributor the Company Already Trusts
A film distributor does more than upload a file.
It knows how to reach the audience, which theaters can show the film, who owns the rights, which formats are accepted, and what has to happen when a screening fails. The producer delivers the finished work into a system that already knows how to distribute it.
That is the advantage of moving a local application into a managed enterprise platform.
The alternative is building a private distribution network for every film. A database comes from one service. Storage comes from another. Authentication, hosting, networking, monitoring, and search each arrive through another product. The application eventually reaches its audience, but the machinery surrounding it becomes larger than the thing being distributed.
For me, Microsoft Fabric has become a practical distributor because it is already where the audience and the data live.
My organization already uses Microsoft identity. The data has a natural home in OneLake and lakehouses. Python logic can run in notebooks and User Data Functions. Semantic models and Power BI remain available when a report is the right way to show the story. The platform provides a governed boundary around storage, compute, permissions, and delivery.
I am not choosing Fabric because every component wins a feature-by-feature comparison. I am choosing it because the organization already knows how to secure it, pay for it, and support it.
A technically perfect component can still be the wrong choice if it adds another vendor, credential, approval, bill, and owner.
This is not really an argument for Fabric. It is an argument for using the platform your company already knows how to trust. In my world, that is Fabric. In another organization, it might be Databricks or something else entirely.
Produce in the studio you own. Distribute through the platform the business already trusts.
The local and enterprise versions do not have to use identical equipment.
DuckDB may hold the data while the application is taking shape. In Fabric, the durable data may move into a lakehouse or another governed data item. Local Python logic may become a notebook or a reusable User Data Function. A React prototype may reveal that the final experience belongs in Power BI, a container application, or an emerging Fabric application surface.
The important pieces survive the move: the data we now understand, the business definitions we tested, the quality rules we established, the interactions users found valuable, and the evidence that the application deserves an audience.
The distributor receives a production with a clear story instead of an idea looking for an architecture.
Git is what makes that handoff workable for agentic development.
Fabric workspaces can synchronize supported item definitions with GitHub or Azure DevOps. The data remains in the platform while the definitions of the production become visible as files in a repository. The agent can inspect those files, understand how the pieces relate, propose changes, and preserve the history of each revision.
That means I can keep working from the production office.
I do not have to open every notebook, model, or report and manually operate each interface. The agent can work against the underlying code and definitions, while Git provides a reviewable boundary between a proposed change and the version distributed to the audience.
The platform supplies governance and reach. The repository keeps the production legible to both the agent and the people responsible for it.
Fabric is beginning to give the audience more than a screen.
Power BI remains a reliable place to present an analytical story. Fabric User Data Functions can now place authenticated Python logic behind an action or write-back workflow. Fabric Apps goes further, bringing hosting, data, APIs, authentication, and local development into the same release path.
Fabric Apps is still in preview. Availability is limited, administrators have to enable it, and preview services are not a settled production answer. But the direction matters.
The enterprise data platform is evolving from a place where people view the finished scene into a place where they can interact with the film.
That creates a more natural path for the applications data engineers can now produce. The film can be developed locally, proven through real use, and then distributed inside the identity, data, and governance environment the organization already understands.
The best distributor is often the one the audience already knows how to access.
This approach also preserves choice.
Some productions will end as Power BI reports. Some will become interactive applications. Some will expose reusable functions to other systems. A smaller number will establish enough value and scale to justify dedicated cloud architecture of their own.
The story earns the distribution model.
Managed platforms reduce the need to assemble a custom network before that story has proven itself. They give the enterprise a known place to secure the data, control the audience, observe the screenings, and decide who supports the release.
They also make the final responsibility clearer.
The agent can help produce every scene and prepare every format. The platform can distribute the result. The producer still has to decide whether the film is accurate, safe, supportable, and ready for the audience.
The agent can produce every take. The producer still decides what ships.
The agentic software production process: build locally, prove value with users, and move only validated applications into the enterprise.
Final Cut: Produce the Decision
For years, data engineering was measured by the scenes we delivered.
The pipeline ran. The model refreshed. The dashboard loaded. Each artifact proved that our part of the production was complete.
Those scenes still matter. The opportunity now is to carry them further.
Agentic development gives one person access to more of the crew. I can move across data, logic, testing, interfaces, and infrastructure while writing none of the code and operating fewer of the tools directly. The boundaries of my own technical execution no longer determine where the production ends.
That changes the deliverable.
The script can become part of an application. The dashboard can become part of a workflow. The insight can lead directly into the decision it was meant to support.
It also changes where the work begins.
An idea can start in the studio already sitting on my desk. I can test the story, screen it with someone, and revise the experience without renting an entire cloud production around it.
Once the application earns a wider audience, the enterprise platform provides the release path. Identity, governance, reliability, and support become part of the distribution agreement.
The producer remains accountable. AI can supply more takes, more formats, and more possible stories. Human judgment determines which evidence is trustworthy, which architecture fits, which costs are justified, and whether the final experience helps the audience make a better decision.
That is where the role of the data engineer moves up the production.
We can spend less time operating every camera and more time deciding where to point it. Less time moving footage and more time understanding what it shows. Less time delivering isolated scenes and more time shaping the complete experience around a business decision.
The future of data engineering belongs to people who can turn data into a story, a story into an application, and an application into a decision.
That leaves us with a more important question: Which story is worth telling—and what should the audience be able to do after seeing it?
The studio is already on your desk. The crew is ready.
Produce something worth acting on.