Context Compression: Reduce Tokens Without Losing Quality

Before compression (2,500 tokens, $0.00625 at GPT-4o rates):
"You may find it helpful to know that the project under discussion
pertains to a web application framework that was originally developed
by a team of engineers at..."

After compression (600 tokens, $0.0015):
"Project: web framework by Meta, React-based, 2013, 200K+ GitHub stars"

from llmlingua import PromptCompressor

compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-bert-base-multilingual",
    use_llmlingua2=True
)

long_prompt = """
The Model Context Protocol is an open standard developed by Anthropic
that enables AI applications to connect with various tools and data
sources in a standardized way. This protocol is designed to provide
a common interface for AI systems to interact with external resources...
"""

compressed = compressor.compress_prompt(
    long_prompt,
    rate=0.5,           # Compress to 50% of original tokens
    force_tokens=["!", "?", "."]  # Preserve these
)
print(f"Original: {len(long_prompt.split())} words")
print(f"Compressed: {len(compressed.split())} words")

def select_relevant_chunks(query, chunks, max_tokens=2000):
    """Select only chunks likely to help answer the query."""
    token_count = 0
    selected = []

    # Sort by relevance score (from retrieval)
    for chunk in sorted(chunks, key=lambda c: c.score, reverse=True):
        chunk_tokens = count_tokens(chunk.text)
        if token_count + chunk_tokens > max_tokens:
            break
        selected.append(chunk)
        token_count += chunk_tokens

    return selected

# Step 1: Summarize
"Summarize the following in 150 words, capturing only the
key facts relevant to understanding the situation:"

# Step 2: Answer from summary
"Based on this summary: [compressed context], answer: [question]"

Remove if:
- Stop words that don't change meaning ("the", "a", "an")
- Redundant phrases ("in order to" → "to")
- Filler text ("you may find it helpful to know that" → "")
- Repeated information across RAG chunks
- Whitespace and formatting tokens