Prompt Caching: When It Saves Money and When It Does Not

Two common cache layers

Provider prompt caching reduces the cost of repeatedly processing a stable prompt prefix while the provider still generates a fresh response. Application or gateway response caching stores a prior result and returns it without making the same provider call again.

The economic and safety profiles differ. Prompt-prefix reuse can be suitable for stable policies or documents even when the final user request changes. Full-response reuse is strongest when the complete request is identical or when a carefully defined semantic match is acceptable.

The cache key is a product decision

A reliable cache key normally includes the model, provider, endpoint, prompt or messages, tool definitions, response format, temperature and sampling controls, user or tenant scope, and any external-data version that affects the answer. Ignoring one of these fields can turn a cache hit into an incorrect response.

Helicone documents request-body and header-based cache identity, while Portkey documents simple and semantic caching through gateway configuration. Those implementations demonstrate why caching belongs in the request-control layer rather than being treated as a generic database shortcut.

Good caching candidates

• Repeated development and test requests where deterministic reuse is acceptable.
• Stable classification, extraction, or transformation jobs with identical inputs.
• Large reusable prompt prefixes such as policies, schemas, and reference documents.
• Public or non-sensitive answers with explicit freshness windows.
• High-volume workloads where the avoided provider cost materially exceeds cache operating cost.

Poor or high-risk candidates

• Requests that depend on live account balances, inventory, legal status, or other rapidly changing data.
• Cross-tenant workloads without strict identity and authorization boundaries.
• Highly creative responses where variation is part of the product value.
• Agent actions whose environment or available tools may have changed.
• Sensitive prompts or outputs that cannot be stored under the required privacy policy.

Measure the full economics

A cache program should report eligible requests, hit rate, tokens avoided, latency change, storage and lookup cost, stale-response incidents, and the effect on quality. It should also distinguish provider prompt-cache discounts from full provider calls avoided by response caching.

Token Pilot's optimization model is intended to evaluate caching as one controlled option among several, not automatically treat every repeated request as safe to reuse.