How many red 2×4 bricks do we have?
In a warehouse full of sealed Lego kits, that question takes days. Every kit has to come off the shelf. Every box gets opened, every bag inside gets torn open and sorted through, every red 2×4 gets counted, and everything goes back in the box. Then the next kit, and the next, across thousands of sets.
In a warehouse where the pieces are sorted into bins by type, it takes about a minute. Walk to the bin of red 2×4s and read the count on the label.
Same bricks, different layout. That difference is the core of how data is stored for applications versus how it's stored for analysis, and it decides what happens when you point an AI agent at your data.
Kits are built for shipping orders
A warehouse of sealed kits is arranged for one job: shipping orders. A customer orders the fire station, and someone walks to the shelf labeled with that set number, picks up the box and ships it. Everything that order needs is already packed together. It takes seconds, and nobody opens anything.
An application's database works the same way. Each customer's record is stored together, like a sealed kit, and an index works like the shelf label. When someone opens a customer page, the database goes straight to that one customer, joins a few related tables for their orders and settings, and returns the page in milliseconds. These databases are usually relational and store each record as a row. For an application handling thousands of small requests a minute, each about a single customer, that's exactly the right layout.
Inventory means opening every kit
Counting red bricks is the opposite kind of question. It isn't about one kit. It's about every kit in the building, and in a kit warehouse the only way to answer it is to open them all.
Analysis asks a database the same kind of question. "How has every customer used this feature over the past year?" touches every customer, every event they've generated and every table that connects them, with large joins across all of it. The database can answer, but it was laid out to fetch one customer quickly, so it has to open nearly every kit to do it.
And in the kit warehouse, the people opening boxes are the same crew who are supposed to be packing today's orders. While they're counting, shipments stall. A database works the same way: the analysis uses the same processors and memory the application needs to serve its users. I've watched queries like that eat the CPU an application depended on, slowing every page for every user and occasionally taking the app down. Monthly reporting is the usual culprit: on the first business day of the month, someone aggregates the entire previous month in one query, against the same database customers are using at that moment.
Bins are built for counting
A warehouse of bins is arranged for the inventory question. Every red 2×4 is in one bin, every window in another, every wheel in a third. Counting means reading one bin and ignoring the rest.
That's column-oriented storage. Instead of keeping each record together, it keeps each field together: all the dates in one place, all the amounts in another, all the feature names in a third. A question about how much was sold reads the amounts and leaves everything else on the shelf. Parquet is a file format built this way, and one of the most widely used formats for analytical data.
The bins have a few more advantages. Their labels stick, so each column keeps one data type and a stray word can't end up in a column of numbers. A bin of a million identical bricks can be recorded once instead of a million times, so the files compress well. Each bin is labeled with the range of values inside it, so a query looking for last month can skip bins that only hold older data. And several workers can count different bins at the same time. A question that would mean opening every kit comes back in seconds.
Keep a second warehouse for counting
Customers still need kits, and the application still needs its database laid out for one order at a time. The fix is a second warehouse. Every night, someone takes a copy of what came in and went out that day, sorts the pieces into bins and stocks the counting warehouse. The shipping floor never stops to take inventory, because inventory happens somewhere else.
That's a big part of why data engineering exists. Pipelines copy data out of the application's database on a schedule, reshape it for analysis and store it in a lake or warehouse, often as Parquet. The application keeps serving customers, reports run faster, and the two jobs stop competing for the same crew. The trade-off is freshness: the bins are only as current as the last restock, so the schedule has to match how current the answers need to be.
Once the counting warehouse grows, it needs its own inventory system: which bins make up which table, which ones were replaced, what a reader should see while a restock is in progress. That's what table formats like Delta Lake and Iceberg add on top of Parquet files. I covered how lakes, warehouses and lakehouses fit together in The Data Kitchen.
Before you point an agent at your data
Now hand the inventory clipboard to a new employee who never gets tired. Send them to the kit warehouse, and they'll start opening boxes, because that's where the bricks are. Every time you ask a follow-up question, they'll open them all again.
That's an AI agent with access to the production database. It connects to whatever you give it, and it iterates: ten versions of a query in a few minutes, each one as heavy as the monthly report. It doesn't know it's pulling the crew off shipping, and it won't notice the application slowing down. The monthly-reporting problem stops being a once-a-month event and becomes something that can happen any afternoon.
So before you point an agent at your data, know which warehouse you're sending it into. Give it the bins: the warehouse, the lake, or a copy of the data on your own machine, where heavy exploration costs next to nothing, which I covered in Give the Agent a Working Environment. The layout that keeps the application safe is also the one that makes the agent fast, because it can read the one bin it needs.
Sometimes the question really does live in the kit warehouse: an order that shipped an hour ago, a customer record that changed this morning. Then set rules at the door. Give the agent a visitor's badge, so it can look at the kits but can't move anything: read-only access. Send it to a copy of the warehouse that shipping never touches: a read replica. Put a time limit on every trip, so a runaway question ends on its own: a query timeout. And cap how many boxes a single trip can open: a limit on how much data one query can scan, where the database supports one. The agent still counts slowly in there, because the kits haven't changed. The rules keep shipping moving while it does.
Kits are for shipping. Count in the bins.