Part 4: Using LLMs with company data

Series: What Every CEO Needs to Know About Generative AI

In November 2022, OpenAI released the first version of ChatGPT, the first Large Language Model (LLM) that anyone could use. Since then, generative AI has spread quickly across both personal and business use. For most organizations, LLMs have so far been used mainly to draft emails, write marketing text, or power simple chatbots, mostly for customer service.

But this is now changing. Companies are no longer asking, “Can we use this technology?” Instead, they’re asking a far more important question: “How do we make this thing smart about our company and our processes?”

Almost all valuable use cases require a model that understands a company’s own data. And that’s where the limitations of general-purpose LLMs become visible.

In this article, I’ll walk you through the different ways you can use your company’s data with large language models, along with the challenges and risks that come with it. The article is a bit longer than usual, but my goal is to explain the key issues in a way that remains both accessible and technically accurate.

Why “Pretrained” Limits What LLMs Know

The “P” in ChatGPT stands for Pretrained, meaning the model has already been trained on a huge dataset and arrives ready to use “out of the box.” That sounds convenient, but it also means you’re working with a generic, one-size-fits-all model that cannot adapt itself to your company’s specific knowledge. Whatever the model learned during training is what it knows. And nothing more.

LLM Knowledge Is Always Outdated

Training an LLM requires massive datasets and extensive processing. Cleaning, preparing, and training on that data takes months. By the time a model is released, large parts of its knowledge are already outdated. Every LLM has a knowledge cut‑off date. This is the point after which it knows nothing about the world.

Take ChatGPT 5.1 as an example. It’s a fairly recent model, but its knowledge cut-off date is September 30, 2024. When the model was released in November 2025, its information was already more than a year old. So if you ask it about anything that happened after that date, it simply doesn’t know.

LLMs Don’t Know Your Business

And then there’s the even bigger issue: Your internal company data was never part of the model’s training set. Not because it wasn’t important, but because it’s private and protected. This is exactly the kind of information you wouldn't want on the open internet: business processes, customer data, internal documents, strategy papers, product specs, and so on.

Yet this internal knowledge is precisely what you need if you want an LLM to support your operations, act as an internal assistant, or help automate processes. Without access to this information, the model remains extremely smart, but completely unfamiliar with your actual business.

How Do We Get an LLM to Understand and Work With Our Company Data?

You might initially think: “Let’s just retrain the LLM with all our internal data.” In practice, that’s simply not an option for most companies.

First, the best-performing models today are proprietary and cannot be retrained by external organizations. And even if you choose an open‑source model, retraining it is extremely challenging. Full retraining requires enormous computing power, often costing several million dollars, plus access to highly specialized and expensive AI hardware.

But the difficulty isn’t just financial or technical. Your data would also need extensive, careful preprocessing to ensure it fits the model’s training format. And even if you managed all that, there is a serious technical risk known as catastrophic forgetting: when you retrain a model with new data, it may accidentally overwrite or “forget” large parts of what it previously learned.

This is one of the main reasons why foundational models are treated as essentially untouchable once they’ve been trained.

There Are Basically Only Two Ways to Bring Your Own Data Into an LLM

LLMs are not something you can easily reprogram from the inside. They work like sealed boxes, and we have only a couple of ways to teach them anything new. In practice, there are really just two options if you want the model to understand things about your company:

• You can fine‑tune the model

• Give the model the information each time you ask a question as Context

Most companies actually end up using a mix of both.

Fine‑tuning is the first option. Think of it as giving the model a small “training session” so it gets used to your language. It’s helpful when you want the model to understand your industry terms, your special wording, or certain concepts your business uses all the time.

But fine‑tuning comes with some downsides: it’s slow, it’s not cheap, and you definitely won’t be doing it every day. Because of that, it’s not a good solution for anything that changes quickly, like a customer’s order history or today’s production numbers. Those things would be outdated again by the time the training is finished.

So fine‑tuning is great for helping the model “sound like your company,” but it’s not how you give it up‑to‑date facts. For anything current or fast‑moving, you’ll still need to give the model that information when you ask your question, through its context window.

Providing Information to the LLM as Context (The Context Window)

The most practical way to make an LLM work with your data is actually quite simple: you just give the model the information it needs every time you talk to it.

This might sound almost too easy, but it’s exactly how nearly all real‑world applications work today. And there’s a good reason for that: LLMs don’t remember anything. They process your request, generate an answer, and the moment they’re done, everything you just told them is gone. No short‑term memory at all.

Even ChatGPT works this way. When you type a message, the model doesn’t magically “remember” the conversation. Instead, the ChatGPT app quietly packs your entire chat history plus some hidden system instructions into the input it sends to the model again. The model only knows what’s in that package, nothing more.

This package of information is what we call the context window. It’s basically the model’s temporary view into the world. Whatever sits inside that window is what the LLM can use to think and respond.

LLM's Context Window

So if you want the model to use your internal documents, your processes, your product specs, it all has to be placed into that context window at the moment you ask your question. Everything the model knows for that one answer comes from whatever you put in there.

Context Window Limitations / The RAG Approach

If we want an LLM-based application to work with our internal company data, we need to put the relevant information into the model’s context window.

But here’s the catch: context windows are limited. They have a fixed size. For example, GPT‑4.0 has a context window of about 32,000 tokens, which translates to roughly 50 pages of text.

Fifty pages might sound like a lot, but it’s nothing compared to the terabytes of documents, emails, reports, and databases a company has. There’s no way to cram all of that into the window. So in practice, we need to select only the pieces of information that are truly relevant to the user’s question.

This is exactly where Retrieval-Augmented Generation, or RAG, comes in.

RAG isn’t a specific product, it’s an architecture pattern. Its job is to look at the user’s prompt, search through a large pool of company data, and pull out only the small set of documents or facts the model needs. These selected pieces are then fed into the LLM.

If you look at the diagram below, you’ll see the process is actually quite straightforward. One thing worth highlighting: the company data that RAG retrieves is passed into the LLM behind the scenes. As a user, you don’t see this happening, but it’s at the core of how the system works

Data Quality as the Key to Good LLM Results

Picking out the most relevant, accurate information from all your company’s data sounds straightforward. In reality, it’s anything but. To match a user’s question with the right documents, the system needs to understand the meaning behind the words. This requires semantic search, which is usually powered by vector databases.

The technology behind this is complex, but definitely manageable. What’s not so easy is the part that everything depends on: your data must be clean and consistent. If the data quality is poor, even the most advanced AI system will produce poor results.

This may sound obvious, but in most companies, data quality is far from ideal. It needs significant improvement before an AI system can use it reliably. This is why, before you even think about launching an AI project, you need to fix your data first. And that’s often a massive undertaking.

Manual cleanup is almost impossible at scale. There are tools that can help identify bad or incomplete data, but even with these tools, a lot of human work is still required. Many companies end up hiring temporary staff or contractors just to go through the data preparation phase. In fact, for most organizations, data preparation is more expensive than building the AI solution itself.

Here’s a simple example that illustrates the problem: in many companies, the same document exists in multiple places. Some are duplicates, some are outdated versions, some have slight formatting differences. If you don’t consolidate these, the system may feed several versions of the same document into the LLM. That creates two issues:

1. The model might pick an outdated version.

2. Seeing the same text multiple times makes the model think it’s more important than other information, which biases the output.

That’s why data cleanup and consolidation matter so much. And yes, it costs money. And there’s no shortcut. If you try to save on data preparation, the system simply won’t perform well. You’ll get unreliable answers. In the best case, that hurts your credibility. In the worst case, it leads to incorrect decisions, financial losses, or even safety issues.

It’s also important to remember that an LLM is never 100% reliable, even with perfect data. These models occasionally produce incorrect or imaginary information. That’s what we call hallucinations. This occurs because LLMs are non‑deterministic, due to their probabilistic nature and the intentionally introduced randomness in the generation process.

If you combine that inherent uncertainty with bad data, the whole system becomes unusable.

Data Privacy, Data and IP Protection

The RAG setup you see in Figure 2: Simple RAG Architecture has some very serious implications for data privacy. In most cases, the LLM itself runs in the cloud and is operated by a third‑party provider. And even though your users won’t see it happening, the moment they ask a question, the system quietly sends the relevant pieces of your internal company data to that external LLM as plain text.

If you’re not careful, that means your valuable, sensitive information, including your intellectual property could end up on servers located in different countries, under different laws and jurisdictions.

Cloud providers do have privacy policies and contractual safeguards, and on paper they usually look solid. But at the end of the day, you’re still placing a great deal of trust in an outside organization. And the risks aren’t limited to commercial leaks or accidental mishandling. In some jurisdictions, government agencies can obtain access to cloud‑stored data through legal processes or other channels.

For certain industries, this simply isn’t acceptable. Defense, aerospace, finance, life sciences, and public‑sector organizations all have legitimate reasons to be extremely cautious. These concerns aren’t hypothetical: There are well-documented cases where government intelligence agencies have provided critical information from competitors to domestic industries. 

The bottom line is: if you plan to use LLMs with your company’s data, you need to think seriously about where that data goes, who might have access to it, and what protections are in place.

On‑Premises LLMs to Safeguard Your Data

When an application needs to work with highly sensitive information, including personal data, trade secrets, research results, regulated documents, there’s really only one way to guarantee maximum protection: you run the entire LLM stack on your own premises.

That means everything lives inside your secure environment: the model itself, the RAG system, all supporting services, and all data sources. Nothing leaves your infrastructure.

Of course, you’re trading convenience for control. Cloud‑based models are easier to use and always up to date, while on‑premises systems require more effort to deploy and maintain. Another downside is that you won’t have access to the very latest, cutting‑edge model releases.

In practice, though, this is usually not a problem. Since you’re providing the model with your own data and controlling the entire environment, you don’t need the absolute newest LLM to get excellent results. There is a wide range of high‑quality models such as Llama 3, Mistral, Qwen, DeepSeek and others that provide more than enough capabilities for enterprise‑grade applications.

Four Key Takeaways for Using LLMs With Your Company Data

Bringing your own data into LLMs can unlock enormous value, but only if you approach it thoughtfully. Good planning, careful preparation, and strict attention to security make the difference between a transformative solution and an expensive disappointment.

Here are the four most important lessons:

1. LLMs don’t know your business.Their knowledge is always outdated, and they have zero visibility into your internal data. If you want useful, accurate answers, you must give the model the missing information.

   
   

2. There are only two ways to provide that information: a) fine‑tuning, and b) supplying the necessary context with each prompt. Fine‑tuning helps with terminology, tone, and high‑level concepts, but not with specific facts. In almost all real implementations, you’ll rely on the context window, which means you must extract the most relevant pieces from terabytes of internal data. That’s exactly what RAG systems are designed to do

   
   

3. Data quality is everything. You need to budget properly for cleaning, consolidating, and preparing your data. If you skip this part or try to cut corners, the AI system simply won’t work. The outputs will be unreliable, and the project will fail.

   
   

4. Protect your sensitive information.Whenever your data is sent to a cloud‑hosted LLM, you need to understand the privacy implications. For highly confidential or regulated information, the only secure option is to run the entire stack on‑premises.