Does the search find the right material?
Task: given the correct document IDs and a list of search results, compute the share that was found and the position of the first hit. Decide what to do with duplicates and a question with no correct answer.
Tests: understanding the metrics, not just the formula.
recall@k says what share of the needed documents is in the first k results. Reciprocal rank for one question is the inverse of the first hit's position; MRR is the average of such scores over many questions.
Contract: retrievalMetrics(relevant, ranked, k) returns {recallAtK, reciprocalRankAtK}. Duplicates are removed before taking the top k; with no relevant documents recallAtK is null, and an invalid k throws RangeError.
Follow-up: how would you evaluate correct refusals separately?
You will practise
- Compute recall@k
- Compute reciprocal rank
- Handle duplicates and the absence of relevant documents
Examples
- retrievalMetrics(['A','B'], ['X','A','A','B'], 3)→ { recallAtK: 1, reciprocalRankAtK: 0.5 }
- retrievalMetrics([], ['X'], 1)→ { recallAtK: null, reciprocalRankAtK: 0 }
The 1 tests
Tests ported from the source material's reference.test.ts (node:test). In the browser they run with the same set of assertions.
eval: dedupe, recall, MRR and a question with no evidence
assert.deepEqual(retrievalMetrics(['A', 'B'], ['X', 'A', 'A', 'B'], 3), { recallAtK: 1, reciprocalRankAtK: 0.5 });
assert.deepEqual(retrievalMetrics(['A', 'B'], ['X', 'A'], 2), { recallAtK: 0.5, reciprocalRankAtK: 0.5 });
assert.deepEqual(retrievalMetrics([], ['X'], 1), { recallAtK: null, reciprocalRankAtK: 0 });
assert.throws(() => retrievalMetrics(['A'], [], 0), RangeError);Related questions
Source: AI & Agents — practice tasks — An exercise written for the study programme, not a confirmed interview task.