AI Agent Cost Optimization: Cut Your LLM Bills Without Sacrificing Quality

TL;DR

• Big OpenAI bill? These tactics can cut costs substantially while maintaining quality.
• Model tiering (cheapest tactic, highest impact): Use a cheap model for simple tasks and a frontier model for complex ones.
• Prompt compression: Remove unnecessary context to cut the token count of every query.
• Intelligent caching: Cache common queries so repeated calls cost nothing.
• Batch processing: Queue non-urgent requests and use OpenAI's Batch API at a 50% discount.
• Output limiting: Set max_tokens appropriately, don't pay for tokens you don't need.

# AI Agent Cost Optimisation: Cut Your LLM Bills

Your OpenAI bill last month: £12,000.

This month you'll process 50% more queries. At current spend, that's £18K. Your CFO is asking questions.

Here's how to cut costs without your agent getting dumber. Practical tactics, no "just use a cheaper model and hope for the best."

The Cost Problem

Example AI agent cost breakdown (illustrative figures, 10K queries/month):

Key insight: For most agents, input tokens (context plus query) are the largest share of cost. Most optimisation should focus here.

Tactic 1: Model Tiering

Don't use GPT-4 for everything. Most queries don't need GPT-4's reasoning power.

Strategy: Route queries by complexity.

Implementation

def select_model(query, complexity_score):
    """
    Tier 1 (Simple): Classification, lookup, FAQ → GPT-3.5 Turbo (£0.001/1K)
    Tier 2 (Moderate): Analysis, summarization → Claude Sonnet (£0.003/1K)
    Tier 3 (Complex): Deep reasoning, code gen → GPT-4 Turbo (£0.01/1K)
    """
    if complexity_score < 0.3:
        return "gpt-3.5-turbo"  # 70% of queries
    elif complexity_score < 0.7:
        return "claude-3-5-sonnet"  # 25% of queries
    else:
        return "gpt-4-turbo"  # 5% of queries

def estimate_complexity(query):
    """Simple heuristic or use cheap classifier"""
    # Method 1: Rule-based
    if any(word in query.lower() for word in ["explain", "analyze", "compare"]):
        return 0.8
    elif any(word in query.lower() for word in ["summarize", "list", "find"]):
        return 0.5
    else:
        return 0.2

    # Method 2: Use GPT-3.5 as classifier (£0.001 vs £0.01)
    classifier_prompt = f"Rate query complexity 0-1: {query}"
    # ... call GPT-3.5, parse score

What to expect: If most of your queries are simple, routing them to a cheap model can cut the bill dramatically, with only a small drop in accuracy. Measure quality before and after on a sample of real queries to confirm the trade-off is worth it.

Tactic 2: Prompt Compression

Most prompts have bloat. Every unnecessary word costs money.

Before Optimization

prompt = f"""
You are a helpful customer support assistant for our company.
Our company sells software products to businesses. We have a knowledge
base of support documentation that you should reference when answering
questions. Please provide accurate, helpful responses based on the
context provided below.

Context from knowledge base:
{retrieved_docs}  # 5 docs × 800 tokens = 4,000 tokens

User question: {user_question}  # 50 tokens

Please answer the question thoughtfully and comprehensively, making sure
to reference specific sections from the context where relevant.
"""
# Total: ~4,200 tokens

Cost: 4,200 tokens × £0.01/1K = £0.042 per query

After Optimization

prompt = f"""
Answer using context below. Cite sources.

Context:
{compressed_docs}  # Top 3 docs × 400 tokens = 1,200 tokens

Q: {user_question}  # 50 tokens
"""
# Total: ~1,300 tokens

Cost: 1,300 tokens × £0.01/1K = £0.013 per query

Savings: 69% reduction in input tokens = £0.029 saved per query

Compression Techniques

1. Remove fluff

• ❌ "You are a helpful customer support assistant for our company"
• ✅ "Answer using context below"

2. Limit retrieved context

• Before: Top 5 docs (4,000 tokens)
• After: Top 3 docs (1,200 tokens)
• Test on your own data: the top 3 documents often contain the answer nearly as often as the top 5

3. Compress retrieved docs

def compress_document(doc, max_tokens=400):
    """Extract only relevant sentences"""
    sentences = doc.split('. ')
    # Use cheap model to score relevance
    scored = [(sent, relevance_score(sent, query)) for sent in sentences]
    top_sentences = sorted(scored, key=lambda x: x[1], reverse=True)[:5]
    return '. '.join([s[0] for s in top_sentences])

4. Use chain-of-thought only when needed

Don't add "Let's think step by step" to every prompt. Reserve for complex reasoning tasks.

Tactic 3: Intelligent Caching

Many queries repeat. "What's your return policy?" asked 50 times/day = 50× the same LLM call.

Implementation

import hashlib
from functools import lru_cache

# In-memory cache (simple)
response_cache = {}

def get_cached_response(query, ttl=3600):
    cache_key = hashlib.md5(query.lower().encode()).hexdigest()

    if cache_key in response_cache:
        cached = response_cache[cache_key]
        if time.time() - cached['timestamp'] < ttl:
            return cached['response']  # Cache hit - £0 cost

    # Cache miss - call LLM
    response = call_llm(query)  # £0.025 cost
    response_cache[cache_key] = {
        'response': response,
        'timestamp': time.time()
    }
    return response

# Redis cache (production)
import redis
r = redis.Redis()

def get_cached_response_redis(query, ttl=3600):
    cache_key = f"llm:{hashlib.md5(query.lower().encode()).hexdigest()}"
    cached = r.get(cache_key)

    if cached:
        return json.loads(cached)

    response = call_llm(query)
    r.setex(cache_key, ttl, json.dumps(response))
    return response

How the hit rate behaves: On day one the cache is empty and every query goes to the LLM. As it warms up, the hit rate climbs and then levels off. For an FAQ-style agent where a small set of questions dominates, a large share of calls can end up served from cache.

Semantic Caching (Advanced)

Exact match caching misses variations:

• "What's your return policy?"
• "How do I return an item?"
• "Tell me about returns"

Solution: Semantic similarity caching

from sentence_transformers import SentenceTransformer
import faiss

embedder = SentenceTransformer('all-MiniLM-L6-v2')
cache_index = faiss.IndexFlatL2(384)  # 384 = embedding dim

def semantic_cache_lookup(query, threshold=0.85):
    query_embedding = embedder.encode([query])[0]

    # Search for similar queries
    distances, indices = cache_index.search(query_embedding.reshape(1, -1), k=1)

    if distances[0][0] < (1 - threshold):  # Cosine similarity > 0.85
        return cached_responses[indices[0][0]]

    return None  # Cache miss

Result: Semantic caching catches paraphrased questions that exact matching misses, raising the hit rate further. Tune the threshold carefully so different questions don't get the same answer.

Tactic 4: Batch Processing (50% Savings for Async)

OpenAI Batch API: 50% discount for 24-hour turnaround.

When to use: Non-urgent tasks (reports, analysis, bulk processing)

from openai import OpenAI
client = OpenAI()

# Create batch file
requests = [
    {"custom_id": f"request-{i}",
     "method": "POST",
     "url": "/v1/chat/completions",
     "body": {
         "model": "gpt-4-turbo",
         "messages": [{"role": "user", "content": queries[i]}]
     }}
    for i in range(1000)
]

# Submit batch
batch = client.batches.create(
    input_file_id=upload_file(requests),
    endpoint="/v1/chat/completions",
    completion_window="24h"
)

# Retrieve results 24h later (50% cheaper)

Use cases:

• ✅ Daily report generation
• ✅ Bulk data enrichment
• ✅ Historical analysis
• ❌ Customer-facing real-time queries

Tactic 5: Output Token Limiting

Stop paying for tokens you don't use.

Before

response = client.chat.completions.create(
    model="gpt-4-turbo",
    messages=[...],
    # No max_tokens set - model decides
)
# Model returns 800-token response when 200 would suffice

Cost: 800 tokens × £0.03/1K = £0.024

After

response = client.chat.completions.create(
    model="gpt-4-turbo",
    messages=[...],
    max_tokens=250  # Enforce limit
)

Cost: 250 tokens × £0.03/1K = £0.0075

Savings: 69% on output tokens = 22% overall savings

Set appropriate limits by use case:

Tactic 6: Streaming with Early Termination

For interactive use, stream responses and let users stop early if satisfied.

def stream_with_early_stop(query, max_tokens=500):
    stream = client.chat.completions.create(
        model="gpt-4-turbo",
        messages=[{"role": "user", "content": query}],
        stream=True,
        max_tokens=max_tokens
    )

    tokens_used = 0
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            print(delta, end='')
            tokens_used += len(delta.split())

            # User can press 'q' to stop early
            if user_satisfied():
                break

    # Only pay for tokens generated before stop
    return tokens_used

Savings: If users stop at 40% of response on average → 60% output token savings

Tactic 7: Smart Context Window Management

Don't stuff context with irrelevant history.

Conversation Memory (Bad)

# Keep entire conversation history
conversation_history = []  # Grows unbounded

conversation_history.append({"role": "user", "content": user_msg})
conversation_history.append({"role": "assistant", "content": ai_response})

# After 10 turns: 10K tokens of context (£0.10 per query!)

Sliding Window (Better)

MAX_HISTORY = 3  # Last 3 turns only

def get_context(conversation_history):
    return conversation_history[-MAX_HISTORY:]  # 1.5K tokens (£0.015)

Savings: 85% reduction in context tokens

Summarization (Best for long conversations)

def manage_context(conversation_history):
    if len(conversation_history) > 10:
        # Summarize old context with cheap model
        old_context = conversation_history[:-3]
        summary = summarize_with_gpt35(old_context)  # £0.005

        return [
            {"role": "system", "content": f"Conversation summary: {summary}"},
            *conversation_history[-3:]  # Recent context
        ]
    return conversation_history

Putting It Together: A Support Agent Example

Here is how these tactics might stack up for a B2B SaaS customer support agent (knowledge base Q&A, ticket creation):

Caching also makes cached responses faster, which users notice. Applied together, these changes are usually a small engineering project rather than a rebuild.

Cost Optimization Decision Tree

Start
  ↓
Are >50% queries simple? → YES → Implement model tiering
  ↓ NO
  ↓
Do queries repeat? → YES → Add caching
  ↓ NO
  ↓
Are prompts >2K tokens? → YES → Compress prompts
  ↓ NO
  ↓
Responses >500 tokens? → YES → Set max_tokens limits
  ↓ NO
  ↓
Any async workloads? → YES → Use batch API (save 50% on batched)
  ↓ NO
  ↓
Long conversations? → YES → Implement sliding window or summarization
  ↓
Monitor and iterate

Monitoring Cost Metrics

Track these dashboards:

# Per-query cost tracking
def track_query_cost(query, model, input_tokens, output_tokens):
    cost = calculate_cost(model, input_tokens, output_tokens)

    metrics.log({
        'timestamp': datetime.now(),
        'model': model,
        'input_tokens': input_tokens,
        'output_tokens': output_tokens,
        'cost': cost,
        'query_type': classify_query(query)
    })

# Daily cost rollup
SELECT
    DATE(timestamp) as date,
    SUM(cost) as daily_cost,
    AVG(input_tokens) as avg_input,
    AVG(output_tokens) as avg_output,
    model
FROM query_costs
GROUP BY date, model
ORDER BY date DESC

Set alerts:

• Daily cost > £500
• Avg tokens per query > 3,000
• Cache hit rate < 40%

Frequently Asked Questions

Will cheaper models hurt quality?

For many tasks, not noticeably. On routine customer support queries, a cheaper model is often only slightly less accurate than a frontier model that costs many times more. For tier-1 support, that premium is rarely worth it. Test on a sample of your own queries to confirm.

Use GPT-4 where it matters: Complex reasoning, code generation, high-stakes decisions.

How aggressive should prompt compression be?

Test incrementally. Start by removing obvious fluff ("You are a helpful assistant..."). Then reduce retrieved docs (5 → 3). Monitor quality. If accuracy drops >5%, you've compressed too much.

Golden rule: Compress until quality drops 3-5%, then back off one step.

Is caching safe for dynamic data?

Set appropriate TTL (time-to-live):

• Static FAQs: 7 days
• Product info: 24 hours
• Live data (stock prices): 5 minutes or no caching

Always include timestamp in cache key for time-sensitive queries.

What's the fastest win?

Model tiering. A basic version takes a few hours to implement and pays off immediately. Start there.

---

Bottom line: Cutting your bill substantially is realistic with these tactics. Most teams over-optimise for quality and under-optimise for cost. A small, measured quality drop for a large cost saving is almost always the right trade-off.

Next: Read our Complete Guide to RAG to optimize retrieval costs specifically.