When LLMs meet enterprise data

A 5-part field report from K2view – written by developers, for developers.

Introduction

A quick note on the style before we start. We’ve written these blogs the way we’d talk to colleagues – honestly, with the parts that went wrong left in. To highlight the K2view approach to each topic, I added "How we did it" sections. Even without these sections, the engineering lessons still hold true.

This isn't a thought experiment. We’re running AI agents in customer production environments today. The use cases are purposefully narrow and operational – characterzied by 3 requirements.

1. They answer questions from an enterprise database about a single customer, and they answer fast. 

2. We’re not talking about long-running, multi-step autonomous agents that go off and think for 5 minutes. A user asks something about their account, and the agent answers at conversational speed. 

3. Every interaction must demonstrate cost efficiency at scale. 

The 5 parts in this series are about how these requirements – single customer, conversational AI speed, and cost efficiency – result from the data architecture.

Agentic text-to-SQL against a real database without losing sleep

Your basic LLM text-to-SQL AI agent might be one prompt in, one SQL string out. You run it and pray.

That's not what we do. Our agent behaves like an analyst. It looks at what tables exist in the data product’s schema, decides which parts of the data are relevant, and writes a query. If that query has errors, it reads the errors, makes the changes, and tries again. It’s a loop with tools, not a translation function.

Why is that? Because an AI agent poking around a live database is scary. The reason it doesn't scare K2view customers comes down to one decision: What it's allowed to run against.

Letting an LLM write SQL against a customer databases is reckless. Imagine an agent:

• Writing a SELECT * across a 400-million-row table and taking down production. 
• Wandering into a table it shouldn't have access to, and pulling another customer's data by accident.
• Creating a schema so enormous the model can't even reason about it. 

Such issues don’t exist when the agent is pointed at a single entity's Micro-Database. A Micro-Database is a “data lake of one” that unifies and stores all the data for one business entity – like a specific customer, order, or product – from all your source systems. 

And, take the failure everyone fears most: A bad query. You can't guarantee a language model never writes one, so don't try. Make it not matter instead. 

The answer is a data product approach, discussed in more detail below. When each customer’s data lives in its own isolated Micro-Database, an inefficient query runs against only one entity's data (e.g., one customer's worth of rows). So there are no shared tables to lock, and no path for a generated query to reach into another customer's data.

Simply put, an LLM agent with a free hand on the schema is dangerous. But, an LLM agent with a free hand on a single entity's tiny isolated database is not.

Additionally, a Micro-Database holds one entity's schema (and not the whole enterprise's hundreds of cryptically-named tables), so the agent isn't buried in context it can't use, and you're not paying to send 400 table definitions for every interaction. 

For fast, single-customer agents that have to stay cost-efficient, that smaller context makes all the difference.

How we do it

At K2view, a data product is the model for a business entity (e.g., customer, invoice, loan, or order). The data product pulls all the data about that entity from every source system into one schema, so the agent sees only a miniscule slice of your enterprise data.

The Micro-Database is the isolation layer. Each entity instance physically lives in its own embedded SQLite database – with one file per customer, not one shared table for all of them. Reads are microsecond-fast and a runaway query can't escape it. 

K2view was built for performance and compliance long before LLMs showed up. That it's also the perfect sandbox for an agent writing its own SQL is a wonderful coincidence.

The lesson

The lesson I'd impart to anyone using agents on real data: Don't spend all your time trying to make the agent behave. Spend it on isolating what it can and can’t touch. A sandboxed agent with a free hand to explore beats a tightly-instructed one let loose in the chicken shed every time.

Coming soon: "It worked in the demo" is not a test, and other things I learned shipping an AI agent