Design dynamic few-shot example retrieval from a vector database

WHAT THIS TESTS: Whether you can architect a production retrieval pipeline that balances relevance, latency, and token economy. Interviewers want to see you understand that few-shot examples are not static assets but dynamic context windows that compete for the same token budget as the user query and model response. They are probing for experience with embedding alignment, approximate search, and prompt assembly at scale.

A GOOD ANSWER COVERS: First, embedding strategy: use a shared bi-encoder or sentence transformer to embed both the user query and the example inputs into the same dense vector space so cosine similarity is meaningful. Second, indexing: store examples in an approximate nearest neighbor index like FAISS, HNSW, or a managed vector database with metadata filters for domain, task, or user tier to narrow the search space before vector comparison. Third, retrieval logic: fetch a larger candidate set than needed, then apply a secondary reranking step for diversity or task-specific relevance to avoid redundant examples. Fourth, prompt assembly: format retrieved input-output pairs with clear delimiters, reserve a token budget for examples using the tokenizer, and truncate or drop the lowest-similarity example if the prompt exceeds the limit. Fifth, latency management: run embedding and retrieval asynchronously or cache frequent query clusters to keep inference time predictable.

COMMON WRONG ANSWERS: Proposing exact k-nearest neighbor search on millions of vectors, which is O of N and too slow for online serving. Using different embedding models for examples and queries without projection or alignment. Treating retrieval and LLM calls as synchronous blocking steps without latency budgeting. Ignoring token limits by stuffing retrieved examples without counting tokens, causing prompt truncation or API errors. Selecting examples based only on embedding similarity without considering diversity, which can yield redundant demonstrations.

LIKELY FOLLOW-UPS: How would you handle a cold-start scenario with no similar historical examples? How do you update the index when new examples arrive without full reindexing? What would you do if the retrieved examples conflict with each other? How would you measure whether dynamic selection actually improves accuracy over static examples?

ONE CONCRETE EXAMPLE: In a customer support system, you store thousands of resolved ticket pairs as embeddings using all-MiniLM-L6-v2. When a user asks why their bill is higher, the system retrieves the top three most similar past billing questions from the vector store, reranks to ensure one covers refunds and one covers proration, then injects them into a FewShotPromptTemplate with XML tags. This improved response relevance by 25 percent and cut token usage by 30 percent compared to a static three-example prompt.

Source: app.readytensor.ai