Part 3: The Overpromise Problem in LLMs

▶ What Every CEO Needs to Know About Generative AI

Key Insights from Part 2: How LLMs Operate

In Part 2: How LLMs Operate, I outlined the core features of Large Language Models (LLMs) that define their behavior:

• Language-focused: LLMs predict the next word in text by analyzing patterns learned from vast training data.

• Non-deterministic: The same prompt can yield different outputs, making LLMs less reliable than traditional software.

• Hallucinations: Incorrect outputs (hallucinations) are not rare exceptions but rather an inherent characteristic of LLM architecture.

• Outdated Knowledge: Since LLMs rely on pretraining data that may be months or years old, they require up-to-date information to be provided in prompts or context.

These limitations greatly affect the usability and accuracy of LLMs, making them unsuitable or prone to failure for certain tasks. However, these shortcomings are rarely acknowledged by industry representatives, who often promote an idealized image of LLM capabilities.

Science or Sales? The Overpromise Problem in LLMs

Interviews with leaders from AI tech companies often highlight an unrealistic vision of LLMs. Leaders from companies developing or selling LLM-based services frequently promote these technologies as universal tools, or even as miracle solutions, to boost efficiency and reduce manpower requirements.

While many of these leaders present themselves as scientists, their role often resembles that of salespeople. The claims they make frequently exaggerate the power of their products, driven by the dual goals of increasing sales and inflating the valuation of their companies.

But such overhyped claims can have real-world consequences. When company leaders fail to distinguish between fiction and reality regarding LLMs, the result can be costly and embarrassing mistakes.

Case Study: Eric Schmidt on AI and Programmers

• Claim: In April 2025, former Google CEO Eric Schmidt stated, “We believe as an industry that in the next one year the vast majority of programmers will be replaced by AI programmers.”

  
  

• Reality: A study published in July 2025, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” by METR (Model Evaluation & Threat Research), found the opposite. The study concluded that developers using AI tools took 19% longer to complete tasks than those working without AI assistance.

From my own experience using LLMs almost daily for software engineering tasks, the output they generate for real-world coding challenges in the context of specific projects is rarely usable without significant reworking. However, LLMs do provide value for specific use cases, such as creating initial frameworks for development projects or building quick prototypes for demonstration purposes.

Case Study: OpenAI’s Claim About Solving 10 unsolved Math Problems

• Claim: About a month ago, OpenAI VP Kevin Well falsely claimed in a now-deleted tweet that GPT-5 had solved 10 previously unsolved Erdös math problems and made significant progress on 11 others.

• Reality: What GPT-5 actually did, was locate published solutions to problems marked as “unsolved” on the Erdös website. By searching the web, it retrieved relevant papers containing solutions or significant progress. While impressive, this result hardly constitutes a breakthrough, but it reflects the LLM’s strength in text analysis and information retrieval, not in solving unsolved math problems.

Exaggerated Claims Lead to Unrealistic Expectations and Failures

The above examples, where claims and reality diverge significantly, are far from isolated incidents. Many similar cases highlight the risks of overpromising.

These conflicting narratives reveal a clear divide: marketing-driven optimism from AI industry representatives versus the sober realism of AI experts warning against hype-induced misconceptions. Claims designed to grab headlines often overshadow verified outcomes, leaving decision-makers with unrealistic expectations. This can result in the misuse of LLMs for unsuitable tasks or the uncritical reliance on their outputs, leading to inaccurate results and costly failures.

Real-World Failures Stemming from Overreliance on LLMs

When the unique behaviors and limitations of LLMs are not properly accounted for, the consequences can be significant, spanning financial, legal, reputational, and operational failures. Here are some notable examples:

• Air Canada’s Chatbot Misstep: A passenger was misinformed by the airline’s chatbot regarding bereavement fares, contradicting the company’s written policy. This led to a tribunal ruling holding Air Canada liable for the chatbot’s misleading output.

  
  

• US Legal Sanctions: Lawyers were fined after submitting briefs containing nonexistent legal citations generated by AI tools. These incidents underline the critical need to verify AI outputs, especially in regulated industries.

  
  

• Deloitte Australia’s Report Errors: A government report created with AI assistance included fabricated references and misattributed quotes. This caused public scrutiny, damaged Deloitte’s credibility, and forced the company to refund the contract.

  
  

• IBM Watson for Oncology’s Inaccurate Recommendations: Marketed as a revolutionary tool for cancer treatment, IBM Watson for Oncology failed to deliver safe and reliable recommendations. Internal reports revealed that Watson’s treatment suggestions were often “unsafe and incorrect,” leading to canceled pilots and tarnished credibility.

Choose the right use cases

The architecture and nature of LLMs make them highly effective for certain tasks while limiting their suitability for others. LLMs excel in areas requiring human-like skills but also inherit some of the same challenges humans face, such as errors, biases, and unpredictability. Below are some criteria to help assess whether a problem is suitable for LLM-based solutions:

✅ Language-intensive Tasks: LLMs are highly effective for tasks involving natural language understanding and generation. Examples include summarization, translation, creative writing, code generation, search, and answering questions.

✅ Unstructured Data: LLMs excel when working with raw, unstructured data like text, conversations, or logs.

✅ Open-ended Problems: Task involves creative exploration, ideation, or producing multiple potential outcomes, LLMs can add value.

✅ Complex Reasoning Requirements: For tasks involving inference, pattern recognition, or multi-step reasoning, LLMs can provide valuable insights

✅ Natural Language, personalized user interaction: LLMs are well-suited for human-like communication tasks, such as powering chatbots, virtual assistants, and conversational interfaces that require personalized responses.

Below some criteria where application of LLM is not the first choice or at least required thorough review of the resulting output:

❌ Deterministic and Highly Precise Tasks: Tasks needing 100% reliability and precision, such as basic arithmetic, balancing financial accounts, or mission-critical computations. Example: Mathematical validation, accounting software.

❌ Rule-based Processes: Deterministic tasks that can be easily addressed with rule-based or programmatic systems without the need for probabilistic reasoning. Examples: Parsing a well-defined input-output format, calculating tax brackets.

❌ Black-box Limitations: Tasks where decision-making must be explainable, transparent, and auditable. LLMs can provide plausible-sounding output without clear reasoning, which undermines trust. Example: AI recommending treatments in healthcare without a clear explanation of the decision path.

❌ High Stake and mission critical Applications: Tasks where incorrect outputs or poor decisions could cause significant harm, such as physical danger, financial loss, or reputational damage. Example: Autonomous vehicle control systems, where even a minor error could endanger lives.

❌ Real-time and Low-latency Applications: Time-critical applications requiring low latency or real-time responses are unsuitable due to the computational demands of LLMs. Example: High-frequency trading systems or real-time traffic control.

Set Realistic Expectations, Define Quality Standard and Success Criteria

When developing applications based on large language models (LLMs), it’s vital to maintain realistic expectations about their capabilities and limitations. LLMs exhibit unique behaviors that must be carefully considered during design and implementation.

Communicate openly with stakeholders about what LLMs can and cannot do. While they can handle a wide range of tasks effectively, their outputs often require validation, additional context, or post-processing to meet quality standards.

Since LLM-generated results are not always fully accurate, it’s crucial to define the desired level of precision for your application. Establish measurable performance metrics to determine when the application is valuable and acceptable. The tolerance for inaccuracies will vary depending on the criticality of the use case—minor errors might be acceptable in some applications, but others demand a much higher level of precision.

Additionally, create clear guidelines for reviewing and validating outputs based on these metrics. Implement robust quality assurance processes and testing strategies to ensure that the application performs reliably and meets expectations over time.

Key Steps for Planning LLM-Based Applications

To ensure the success of LLM-based applications, thoughtful planning is essential. Follow these key steps to address both their potential and limitations:

▶ Manage Expectations: Clearly communicate to all stakeholders that while LLMs are powerful tools, their results often require review and refinement to ensure accuracy and trustworthiness.

▶ Define Accuracy Needs and Success Metrics: Assess how precise the application needs to be in order to deliver value. Set measurable success criteria tied to the use case, allowing some flexibility for less critical tasks but requiring stricter standards for high-stakes applications.

▶ Plan Oversight and Quality Assurance: Establish processes for reviewing, validating, and monitoring the system’s outputs. Integrate thorough testing to maintain consistent performance and ensure the application meets defined success metrics.

By following these steps, organizations can leverage the strengths of LLMs while minimizing their weaknesses, ensuring that applications deliver real value without compromising reliability or trust.