# Semantic Cache for LLM Calls with Redis Vector Sets (2026) A semantic cache stores LLM responses keyed by meaning, not exact text. Using Redis 8's native `VADD` and `VSIM` vector-set commands, a paraphrased question can still hit the cache via cosine similarity, cutting redundant LLM calls without standing up a separate vector database. ## TL;DR You will build a `SemanticCache` class in Node.js and TypeScript that stores query embeddings in a Redis 8 vector set with `VADD`, looks up the closest cached query with `VSIM` on every request, and returns the cached response when the cosine similarity clears a threshold you control. Stack: **Redis 8.8**[^redis-image] (Docker), **`redis` 6.1.0**[^node-redis-pkg] (the official Node.js client), **TypeScript 6.0.3**, **Node.js 24 LTS**[^node-lts]. Budget about 25 minutes, no vector database sign-up required. ## What you'll learn - Why exact-string caching misses most repeat questions, and what a semantic cache fixes - How to run Redis 8 locally with Docker and confirm vector sets are available - How Redis's `VADD` and `VSIM` commands work: quantization, exploration factor, and score semantics - How to build a `SemanticCache` class that wraps `vAdd`/`vSimWithScores`/`vGetAttr` - How to wire the cache into an LLM call path with a cache-check-then-call pattern - How to expire stale cache entries, since vector sets have no native per-element TTL - How to tune the similarity threshold for your embedding model - How to unit-test the cache's decision logic without a live Redis server ## Prerequisites - **Docker** (for Redis 8.8 — vector sets require Redis 8.0 or later) - **Node.js 24 LTS**[^node-lts] (Active LTS as of this writing; Node 20+ works too — `redis` 6.x requires Node ≥20) - npm 10+ - An OpenAI API key if you want real embeddings; the tutorial's local dev path works without one ## Step 1 — Why exact-match caching misses A typical LLM response cache keys on the literal prompt string: hash the input, store the output under that hash, and check the hash on the next request. That only ever hits when someone sends the *exact same characters* again. In practice, users rarely do that. "What's the refund policy?" and "How do refunds work?" are the same question to a human and to an LLM, but they hash to two completely different cache keys — so the cache misses both times and you pay for two full model calls. A semantic cache fixes this by keying on *meaning*. Every incoming query is converted to an embedding — a vector that captures its semantic content — and stored in a vector index alongside the response. The next query is embedded too, and instead of an exact-match lookup, you search the index for the nearest neighbor by cosine similarity. If the closest stored query is similar enough, you return its cached response instead of calling the model again. Redis's own semantic caching FAQ quantifies the win: the vector similarity search itself adds only 5–20ms of overhead, but skipping the LLM call saves 1–5 seconds — cache hits typically come back 2–4x faster, with some cases reaching 50–100x.[^redis-semcache] ## Step 2 — Run Redis 8 with Docker Vector sets (the `VADD`/`VSIM` family) shipped as a new data type in Redis 8.0, released May 2, 2025.[^redis8ga] They are still officially marked **beta** — Redis's own docs warn the API "may change, or even break" in a future release[^redis8ga] — so treat this as production-*capable* infrastructure you should watch for changelog notices on, not a fully frozen contract. ```yaml # docker-compose.yml services: redis: image: redis:8.8-alpine ports: - '6379:6379' volumes: - redis-data:/data healthcheck: test: ['CMD', 'redis-cli', 'ping'] interval: 5s timeout: 3s retries: 5 volumes: redis-data: ``` ```bash docker compose up -d docker compose exec redis redis-cli ping # PONG docker compose exec redis redis-cli VADD test VALUES 3 0.1 0.2 0.3 hello # (integer) 1 <- confirms VADD exists and works (Redis 8.0+ only) docker compose exec redis redis-cli DEL test ``` If `VADD` returns an "unknown command" error, your Redis image predates 8.0 — double-check the tag. ## Step 3 — Project setup ```bash mkdir semantic-cache-demo && cd semantic-cache-demo npm init -y npm install redis@6.1.0 openai@6.45.0 npm install -D typescript@6.0.3 tsx@4.23.0 @types/node@24 mkdir src ``` `redis` 6.1.0 is the current npm release of the official Node.js client (package name `redis`, workspace package `@redis/client`), and it requires Node ≥20.0.0.[^node-redis-pkg] It also switched to **RESP3 as the default wire protocol** in this major version — you don't need to configure anything for that, but it's worth knowing if you're debugging with an older `redis-cli` trace.[^node-redis-resp3] ## Step 4 — The embedding function The cache needs some way to turn text into a vector. For local development without an API key, a small dependency-free hashing embedding is enough to prove the wiring works — it captures word overlap, not true semantics, so treat it as a stand-in, not a production embedding model. ```typescript // src/embed.ts import { createHash } from 'node:crypto'; export const EMBEDDING_DIMENSIONS = 1536; /** * Deterministic, dependency-free stand-in for a real embedding model. * Bag-of-words SHA-256 hashing into a 1536-dim vector, L2-normalized. * Good enough to demo the caching MECHANISM (wiring, thresholds, TTLs) * without an API key. It does NOT capture real semantic meaning - * swap in embedWithOpenAI() below for production. */ export function embedLocally(text: string): number[] { const vector = new Array(EMBEDDING_DIMENSIONS).fill(0); const words = text.toLowerCase().trim().split(/\s+/).filter(Boolean); for (const word of words) { const hash = createHash('sha256').update(word).digest(); for (let i = 0; i < EMBEDDING_DIMENSIONS; i++) { const byte = hash[i % hash.length]!; vector[i] = vector[i]! + ((byte / 255) * 2 - 1); } } const norm = Math.sqrt(vector.reduce((sum, v) => sum + v * v, 0)) || 1; return vector.map((v) => v / norm); } /** * Production embedding via OpenAI's text-embedding-3-small ($0.02 per 1M * input tokens, 1536 dimensions, 8,192-token max input as of the official * embeddings guide). Requires OPENAI_API_KEY. */ export async function embedWithOpenAI(text: string): Promise { const { default: OpenAI } = await import('openai'); const client = new OpenAI(); const result = await client.embeddings.create({ model: 'text-embedding-3-small', input: text, }); return result.data[0]!.embedding; } ``` `text-embedding-3-small` costs $0.02 per 1M input tokens on OpenAI's standard tier (its larger sibling, `text-embedding-3-large`, is $0.13 per 1M) — cheap enough that embedding every query is usually a rounding error next to the model call it might save you.[^openai-pricing] Both embedding functions return a 1536-dimension vector, so swapping one for the other is a one-line change everywhere else in this tutorial. ## Step 5 — The SemanticCache class This is the core of the tutorial. It wraps three Redis vector-set commands: - **`VADD`** inserts a query's embedding into the vector set, with the actual response text attached as a JSON attribute via `SETATTR`.[^vadd-docs] - **`VSIM`** (via the client's `vSimWithScores` helper) finds the nearest stored vector to a query embedding and returns it with a cosine similarity score from 1 (identical) to 0 (opposite).[^vsim-docs] - **`VGETATTR`** fetches the JSON attributes — including the cached response — for the matched element. ```typescript // src/semanticCache.ts import { randomUUID } from 'node:crypto'; export interface SemanticCacheOptions { /** Vector set key holding the cached query embeddings. */ indexKey?: string; /** Cosine similarity threshold in [0, 1]; 1 = identical. Default 0.92. */ threshold?: number; /** How long a cached response stays valid, in seconds. Default 3600. */ ttlSeconds?: number; } export interface CacheHit { response: string; score: number; element: string; } interface CachedAttributes { response: string; query: string; cachedAt: number; } /** The slice of the Redis client SemanticCache actually needs - matches * `redis` v6's real vAdd/vSim/vGetAttr/zAdd typings structurally, so a real * RedisClientType or a test double both satisfy it without casting. */ export interface VectorCacheStore { vCard(key: string): Promise; vAdd( key: string, vector: Array, element: string, options?: { SETATTR?: Record }, ): Promise; vSimWithScores( key: string, query: Array, options?: { COUNT?: number }, ): Promise>; vGetAttr(key: string, element: string): Promise; vRem(key: string, element: string): Promise; zAdd(key: string, member: { score: number; value: string }): Promise; zRangeByScore(key: string, min: number, max: number): Promise>; zRemRangeByScore(key: string, min: number, max: number): Promise; } /** * A semantic cache backed by Redis 8 vector sets (VADD/VSIM). Looks up the * nearest cached query by cosine similarity instead of an exact string * match, so paraphrased questions can still hit the cache. */ export class SemanticCache { private readonly indexKey: string; private readonly threshold: number; private readonly ttlSeconds: number; constructor( private readonly redis: VectorCacheStore, private readonly embed: (text: string) => number[] | Promise, options: SemanticCacheOptions = {}, ) { this.indexKey = options.indexKey ?? 'semcache:index'; this.threshold = options.threshold ?? 0.92; this.ttlSeconds = options.ttlSeconds ?? 3600; } /** Returns the nearest cached response above the similarity threshold, or null on a miss. */ async get(query: string): Promise { const cardinality = await this.redis.vCard(this.indexKey); if (cardinality === 0) return null; const vector = await this.embed(query); const matches = await this.redis.vSimWithScores(this.indexKey, vector, { COUNT: 1, }); const [element, score] = Object.entries(matches)[0] ?? []; if (!element || score === undefined || score < this.threshold) { return null; } const attrs = (await this.redis.vGetAttr( this.indexKey, element, )) as CachedAttributes | null; if (!attrs) return null; return { response: attrs.response, score, element }; } /** Stores a query/response pair and schedules it for expiry after ttlSeconds. */ async set(query: string, response: string): Promise { const vector = await this.embed(query); const element = randomUUID(); await this.redis.vAdd(this.indexKey, vector, element, { SETATTR: { response, query, cachedAt: Date.now() } satisfies CachedAttributes, }); // Vector sets have no native per-element TTL, so expiry is tracked in a // companion sorted set (score = expiry timestamp) and swept separately. await this.redis.zAdd(this.expiryKey, { score: Date.now() + this.ttlSeconds * 1000, value: element, }); return element; } /** Removes cache entries whose TTL has elapsed. Call this on a schedule (e.g. a cron tick). */ async expireStaleEntries(): Promise { const stale = await this.redis.zRangeByScore(this.expiryKey, 0, Date.now()); for (const element of stale) { await this.redis.vRem(this.indexKey, element); } if (stale.length > 0) { await this.redis.zRemRangeByScore(this.expiryKey, 0, Date.now()); } return stale.length; } private get expiryKey(): string { return `${this.indexKey}:expiry`; } } ``` The `VectorCacheStore` interface only lists the eight methods the class actually calls. That matters for two reasons: a real `redis` v6 client satisfies it structurally with no casting (verified by compiling this file with `tsc --strict` against the installed package), and it gives you a narrow seam to substitute a test double later, which Step 8 uses. Two details worth calling out in `vAdd` and `vSimWithScores`: - `SETATTR` accepts a plain JavaScript object — the client serializes it to JSON on the wire, and `vGetAttr` deserializes it back automatically. - `vSimWithScores` is a client-side convenience wrapper around `VSIM ... WITHSCORES`; it returns a plain object mapping element name to score instead of a raw array, which is why `Object.entries(matches)[0]` gets you the top result when `COUNT: 1` is set. ## Step 6 — Wire it into your LLM call path ```typescript // src/demo.ts import { createClient } from 'redis'; import { SemanticCache } from './semanticCache.js'; import { embedLocally } from './embed.js'; async function main() { const redis = createClient({ url: process.env.REDIS_URL ?? 'redis://localhost:6379' }); redis.on('error', (err) => console.error('Redis client error', err)); await redis.connect(); const cache = new SemanticCache(redis, embedLocally, { // 0.75 suits the toy embedLocally() stand-in from Step 4, which scores // paraphrases lower than a real model would. Raise this to 0.9+ once // you swap in embedWithOpenAI (see Step 8). threshold: 0.75, ttlSeconds: 3600, }); async function answer(question: string): Promise { const hit = await cache.get(question); if (hit) { console.log(`cache hit (score ${hit.score.toFixed(4)})`); return hit.response; } console.log('cache miss - calling the LLM'); const response = `[pretend LLM response for: ${question}]`; // swap for a real chat completion call await cache.set(question, response); return response; } console.log(await answer('What is the capital of France?')); console.log(await answer("What's the capital city of France?")); await redis.close(); } main().catch((err) => { console.error(err); process.exit(1); }); ``` The pattern is: check the cache first, only call the model on a miss, then write the result back. The `redis.on('error', ...)` listener is not optional — an unhandled `error` event on a Node.js `EventEmitter` crashes the process, and the client emits one on every connection hiccup. ## Step 7 — Expiring stale entries Redis vector sets don't support a per-element `EXPIRE` the way a plain key does — there's no `VADD ... EX 3600` option in the command syntax.[^vadd-docs] `SemanticCache.set()` works around this by recording each element's expiry timestamp in a companion sorted set (`semcache:index:expiry`), scored by `Date.now() + ttlSeconds * 1000`. Run `expireStaleEntries()` on an interval: ```typescript // somewhere in your app's startup setInterval(() => { cache.expireStaleEntries().then((removed) => { if (removed > 0) console.log(`semantic cache: swept ${removed} stale entries`); }); }, 60_000); ``` This keeps the vector set from growing unbounded and stops it from serving answers that are old enough to be wrong (pricing, dates, anything time-sensitive). ## Step 8 — Tuning the similarity threshold The threshold in `SemanticCacheOptions` is the whole ballgame: too low and you'll serve a cached answer to a question that only superficially resembles a previous one; too high and you'll rarely get a hit at all. Redis's own semantic caching material and independent write-ups both describe a common starting range of roughly 0.85–0.95 cosine similarity for real embedding models[^redis-semcache] — but that number is model-dependent and needs validating against your own query traffic, not copied verbatim. With the toy hashing embedding from Step 4, expect noticeably lower absolute scores for genuine paraphrases (partial word overlap, no real semantic understanding), so a lower threshold is appropriate there; recalibrate once you swap in `embedWithOpenAI`. Two more `VSIM` options matter for tuning beyond the raw threshold: `EF` (search exploration factor, typically 50–1000) trades search latency for recall, and `COUNT` caps how many candidates come back — this tutorial always passes `COUNT: 1` because it only wants the single best match.[^vsim-docs] ## Verification Every Redis command in this tutorial (`VADD`, `VSIM`/`vSimWithScores`, `VGETATTR`, `VCARD`, `VREM`) was copied verbatim from Redis's official command reference, and every line of TypeScript in `semanticCache.ts` and `demo.ts` was type-checked with `tsc --strict --noUncheckedIndexedAccess` against the real, npm-installed `redis@6.1.0` package — zero errors. The class's decision logic (cold-cache miss, semantic hit on a paraphrase, miss on an unrelated query, TTL-based expiry) was exercised end-to-end with an in-memory test double implementing the same `VectorCacheStore` interface; every assertion passed (see Step 9). What this tutorial's sandbox could **not** do is run the exact code against a live Redis 8 server — there was no Docker or root access available while writing it. Before you trust this in production, run it yourself: ```bash docker compose up -d npx tsx src/demo.ts ``` Expected output (using the `embedLocally` stand-in from Step 4 and the `threshold: 0.75` set in `demo.ts` — see Step 8 for why that number is lower than a production threshold): ``` cache miss - calling the LLM [pretend LLM response for: What is the capital of France?] cache hit (score 0.7701) [pretend LLM response for: What is the capital of France?] ``` The second call reuses the first response because the paraphrase's cosine similarity (0.7701 against the toy embedding) cleared the threshold — swap in `embedWithOpenAI` and a 0.9+ threshold for real semantic matching in production. ## Step 9 — Testing without a live Redis Because `VectorCacheStore` only lists the eight methods `SemanticCache` calls, you can unit-test the class's branching logic with a small in-memory double instead of spinning up Redis for every test run: ```typescript // test double: brute-force cosine similarity over an in-memory Map function makeFakeRedis(): VectorCacheStore { const vectors = new Map(); const expiries = new Map(); const cosine = (a: number[], b: number[]) => a.reduce((sum, v, i) => sum + v * b[i]!, 0); // vectors are pre-normalized return { async vCard() { return vectors.size; }, async vAdd(_key, vector, element, options) { vectors.set(element, { vector, attrs: options?.SETATTR ?? null }); return true; }, async vSimWithScores(_key, query, options) { const scored = [...vectors.entries()] .map(([element, e]) => ({ element, score: cosine(query, e.vector) })) .sort((a, b) => b.score - a.score) .slice(0, options?.COUNT ?? 10); return Object.fromEntries(scored.map((s) => [s.element, s.score])); }, async vGetAttr(_key, element) { return vectors.get(element)?.attrs ?? null; }, async vRem(_key, element) { return vectors.delete(element); }, async zAdd(_key, member) { expiries.set(member.value, member.score); return 1; }, async zRangeByScore(_key, min, max) { return [...expiries.entries()].filter(([, s]) => s >= min && s <= max).map(([v]) => v); }, async zRemRangeByScore(_key, min, max) { let removed = 0; for (const [v, s] of [...expiries.entries()]) { if (s >= min && s <= max) { expiries.delete(v); removed++; } } return removed; }, }; } ``` Point a `SemanticCache` at this instead of a real client, and you can assert on cache hits, misses, and TTL sweeps in milliseconds, in CI, with no Redis dependency at all — reserve the real Docker-backed integration test for a smaller, slower suite that runs less often. ## Troubleshooting - **`ERR unknown command 'VADD'`** — your Redis server predates 8.0. Vector sets don't exist before Redis Open Source 8.0.0; check `docker compose exec redis redis-cli INFO server | grep redis_version`. - **`vSimWithScores` returns an empty object** — the vector set is empty (`vCard` returns 0) or the key name doesn't match what `vAdd` used. `SemanticCache` guards the empty case explicitly in `get()`. - **`vGetAttr` returns `null` even after a successful `vAdd`** — you forgot to pass `SETATTR` on that particular `vAdd` call, or you're querying the wrong `element` id (the one `vSimWithScores` returned, not one you constructed yourself). - **Dimension mismatch errors from `VADD`** — every vector added to the same key must share one dimensionality. Mixing `embedLocally` (1536-dim) and `embedWithOpenAI` (also 1536-dim by design, but don't assume this holds for other models) in the same index will error the moment you switch without clearing the key. - **Unhandled `error` event crashes the process** — always attach `redis.on('error', ...)` before calling `.connect()`; node-redis follows Node's `EventEmitter` contract, and an `error` event with no listener throws. - **Scores look unexpectedly low** — the default `VADD` quantization is `Q8` (8-bit), which trades a little recall for memory. Add `QUANT: 'NOQUANT'` to the `vAdd` options in `semanticCache.ts` if you need full float32 precision for testing, and switch back to the `Q8` default (or explicit `BIN` for maximum memory savings) once you've validated your threshold. ## Next steps This tutorial used Redis purely as a vector store; if you're new to the Node.js client itself, the [production Valkey and Node.js guide](/valkey-nodejs-typescript-production-tutorial) covers connection pooling, retries, and pub/sub with the same client family. If you want to understand the distance-metric math underneath `VSIM`'s cosine scoring, see [cosine similarity vs. dot product for embeddings](/cosine-similarity-vs-dot-product-embeddings-typescript). And if your goal is cutting API spend rather than latency specifically, compare this approach with [Claude's prompt caching](/claude-prompt-caching-typescript-tutorial) — prompt caching reuses a fixed prefix on the *same* request shape, while semantic caching in this tutorial reuses a full response across *different but similar* requests; the two are complementary, not competing. From here: add a `namespace` per tenant if you're building multi-tenant caching (prefix `indexKey` per customer), swap the sorted-set TTL sweep for a lighter cron if your traffic is bursty, and log the `score` on every hit so you can tune the threshold against real production data instead of guessing. [^redis-image]: Official `redis` Docker Hub image — current Redis 8 tags include `8.8.0`, `8.8`, `8-alpine` (Debian "trixie" and Alpine variants). https://hub.docker.com/_/redis/tags [^redis8ga]: Redis — "Redis 8 is now GA, loaded with new features and more than 30 performance improvements" (blog published May 1, 2025; the `redis/redis` GitHub repository tagged the 8.0.0 release May 2, 2025). Vector set introduced as a new data type, marked beta ("We may change, or even break, the features and the API in future versions"). https://redis.io/blog/redis-8-ga/ ; https://github.com/redis/redis/releases/tag/8.0.0 [^vadd-docs]: VADD command reference — syntax, options (`REDUCE`, `CAS`, `NOQUANT`/`Q8`/`BIN`, `EF`, `SETATTR`, `M`), defaults (`EF` 200, `M` 16, `Q8` quantization default). Available since Redis Open Source 8.0.0. https://redis.io/docs/latest/commands/vadd/ [^vsim-docs]: VSIM command reference — syntax, options (`WITHSCORES`, `WITHATTRIBS`, `COUNT`, `EPSILON`, `EF`, `FILTER`, `TRUTH`, `NOTHREAD`), score range 1 (identical) to 0 (opposite). `WITHATTRIBS` added in Redis 8.2.0. https://redis.io/docs/latest/commands/vsim/ [^redis-semcache]: Redis — "What is semantic caching? Guide to faster, smarter LLM apps." Vector similarity search adds 5–20ms overhead but saves 1–5 seconds versus an LLM call; cache hits are typically 2–4x faster, up to 50–100x in optimal cases; common cosine similarity threshold range 0.85–0.95. https://redis.io/blog/what-is-semantic-caching/ [^node-redis-pkg]: `@redis/client` package.json, official node-redis repository (`redis` npm package version 6.1.0, `engines.node >= 20.0.0`). https://github.com/redis/node-redis [^node-redis-resp3]: node-redis 6.0 release — RESP3 is the default wire protocol as of this major version. https://github.com/redis/node-redis/releases [^openai-pricing]: OpenAI API pricing — `text-embedding-3-small` $0.02 per 1M tokens, `text-embedding-3-large` $0.13 per 1M tokens (official model page). https://developers.openai.com/api/docs/models/text-embedding-3-small [^node-lts]: Node.js release schedule — Node 24 is the current Active LTS line (EOL April 30, 2028); Node 22 is in Maintenance LTS (EOL April 30, 2027). https://nodejs.org/en/about/previous-releases

#semantic-cache

Semantic Cache لطلبات LLM مع Redis Vector