Two offline benchmarks reproduce InnerMatch's live demos with a simulated user asking 500 questions in a row, every retrieval and vote scored against a known ground truth. The question in both: does adaptive learning measurably beat a static nearest-neighbor baseline — and does the answer depend on the embedding space?
August 13, 2026 · Behrang Mehrparvar
InnerMatch's pitch is simple to state and easy to overclaim: it starts from a principled baseline and gets better as people interact with it. Demos are a good way to feel that, but a demo is one browser session — it doesn't tell you whether the improvement holds up over hundreds of queries, or whether it depends on the kind of embedding space it's operating in. So we built two offline benchmarks that reproduce our live demos exactly, replace one browser session with a simulated user asking 500 questions in a row, and score every retrieval automatically against a known ground truth.
Both benchmarks follow the same protocol. A simulated user asks a random query built from the corpus itself (so ground truth is known by construction), each retrieval method returns its ranked top results, and — for the methods that learn — the top result gets a simulated vote: +1 if relevant, −1 if not, exactly matching the vote our live demos send. Everything runs with a fixed random seed (42) for reproducibility, and every adaptive method accumulates its learned weights across the entire 500-query session, the same way a real browser session persists until the page resets.
Five standard information-retrieval metrics are computed per query and macro-averaged: Precision@K and Recall@K (share of the top-K results that are relevant, and share of all relevant items the top-K captured), HitRate@K (did at least one relevant item appear at all), NDCG@K (rewards relevant items ranked higher, not just present), and MRR (how close to #1 the first relevant result landed, across the full ranking, not just the top-K).
One thing we were careful about: both benchmarks substitute a local embedding for the paid API (Voyage AI) our live demos use in production. That substitution is disclosed everywhere in the underlying methodology, and it's the reason the numbers below should be read as evidence about the learning mechanism, not as a claim about production-embedding accuracy.
Our four-panel MCP (Model Context Protocol) tool-matching demo lets a user describe an everyday task in plain language — planning a trip, tracking a workout, managing a playlist — and see four retrieval methods each surface their top-5 matching tools from a shared corpus. Two panels work in a keyword (TF-IDF) space, two work in a semantic (dense embedding) space; within each pair, one is a static baseline and one is InnerMatch, adapting to votes. This benchmark reproduces that comparison offline.
121 real MCP tools across 12 "Daily Life" categories — Calendar & Scheduling, Tasks & To-dos, Notes & Writing, Email & Communication, Weather & News, Shopping & Finance, Music & Entertainment, Travel & Maps, Health & Fitness, Social Media, Food & Recipes, and Home & Smart Devices — each with a name, category, and one-sentence description (Google Calendar, Todoist, Spotify, Google Maps, Apple Health, and 116 others, hand-curated, not synthetically padded). A query is built by sampling 2–4 tokens from a randomly chosen tool's own name and description (e.g. "playlists discover music" from Spotify's description); the sampled tool's category is the ground truth, so a retrieved item counts as relevant if its category matches — this benchmark tests category-level retrieval, not exact-tool recovery.
Because the demo's keyword and semantic panels are genuinely different technologies, the benchmark needed two genuinely different local spaces: real TF-IDF for the keyword panels, and truncated SVD / LSA (50 components, 62.2% explained variance, fit on the same TF-IDF matrix) standing in for the dense embedding panels — a real co-occurrence-based semantic space, not a relabeled copy of the TF-IDF vectors.
500 simulated queries, top-5 shown, seed 42:
| Method | Space | Learns? | P@5 | R@5 | NDCG@5 | MRR |
|---|---|---|---|---|---|---|
Keyword baseline (tfidf) | TF-IDF | No | 0.4856 | 0.2406 | 0.5761 | 0.9675 |
Keyword + InnerMatch (tfidf_im) | TF-IDF | Yes | 0.5516 | 0.2734 | 0.6056 | 0.8886 |
Semantic baseline (euclidean) | LSA | No | 0.5916 | 0.2928 | 0.6595 | 0.9464 |
Semantic, untrained (innermatch_cold) | LSA | No | 0.6288 | 0.4597 | 0.7048 | 0.9500 |
Semantic + InnerMatch (innermatch) | LSA | Yes | 0.6168 | 0.4572 | 0.6784 | 0.8850 |
Full run also included cold and kernel-scored variants of each trained method, used as sanity checks and stress tests — see the kernel section below.
The semantic space beats the keyword space on every metric, learning aside. Even euclidean, the untrained semantic baseline, beats tfidf_im, the trained keyword method, on every metric except MRR. A query like "playlists discover music" shares real vocabulary with Spotify's own description, but a query sampled from a different tool in the same category may share almost no literal wording — TF-IDF misses that connection where LSA's latent structure can still catch it.
The two spaces respond to online learning in opposite directions. In the keyword space, learning helps clearly: tfidf_im climbs from 0.5200 to 0.5832 Precision@5, first half of the session to second half — a real, accumulating gain from feedback alone, with the vector space itself untouched. In the semantic space, learning doesn't help and mildly regresses: innermatch sits below its own untrained counterpart innermatch_cold on every single metric, and its within-session curve is flat rather than climbing.
A plausible mechanism: InnerMatch's feedback rule adapts one weight per vector dimension. TF-IDF dimensions are sparse and individually interpretable — "this term matters, that one doesn't" is exactly the kind of thing a per-dimension weight can learn from a single vote. LSA/SVD dimensions are dense latent factors, each an entangled mixture of many original terms with no individual meaning — there's much less clean, learnable structure for the same update rule to exploit, and each feedback step risks perturbing a dimension that was doing useful work for unrelated reasons. Online learning adds real value on top of a weaker starting point, and adds slight noise on top of an already-strong one.
Our second demo is a two-panel retrieval-augmented-generation setup: a user types a physics question, and two methods each surface their top-2 supporting facts. One panel (Euclidean) is a static nearest-neighbor baseline that never changes; the other (InnerMatch) starts identical but adapts to votes across the conversation.
25 physics topics × 10 facts each = 250 facts (Rotational Dynamics, Thermodynamics, Electromagnetism, and 22 others), each topic carrying its own keyword list. A query joins 2–4 of a randomly chosen topic's keywords into a short string (e.g. "torque angular centripetal"); the sampled topic is the ground truth. Embeddings here are TF-IDF, substituting for the same production Voyage AI embedding the live demo uses, ported line-for-line from our own tokenizer and IDF formula.
Both InnerMatch variants rerank a top-50 candidate pool pre-filtered by Euclidean distance — the same payload-limiting step production uses. innermatch_cold is InnerMatch with its weight matrix left at its untouched starting point; it isolates how much of any InnerMatch result is just "reranking a pre-filtered pool with normalized vectors" versus the learning itself. innermatch is the same setup with weights evolving via feedback across all 500 queries.
500 simulated queries, top-2 shown, one continuous session, seed 42:
| Method | Learns? | P@2 | R@2 | HitRate@2 | NDCG@2 | MRR |
|---|---|---|---|---|---|---|
Static baseline (euclidean) | No | 0.6950 | 0.1390 | 0.8480 | 0.7034 | 0.8225 |
Rerank only, untrained (innermatch_cold) | No | 0.7098 | 0.2610 | 0.8614 | 0.7181 | 0.8313 |
InnerMatch, trained (innermatch) | Yes | 0.8133 | 0.3274 | 0.9016 | 0.8213 | 0.8829 |
Every metric orders the same way: euclidean < innermatch_cold < innermatch, and two separate effects stack on top of each other. Reranking a pre-filtered pool with normalized vectors — zero learning involved — already beats raw full-corpus Euclidean distance on every metric, most sharply on Recall (0.139 → 0.261). Online learning then adds a comparably sized further jump on top: Recall climbs to 0.327, NDCG to 0.821, Precision to 0.813.
The learning curve isolates that second effect directly. Only innermatch climbs within the session — Precision@2 goes from 0.759 in the first 250 queries to 0.868 in the last 250, a real gain purely from accumulated feedback with the starting weights fixed at the same point as innermatch_cold. Both static methods stay flat within query-sampling noise, exactly what's expected of methods that structurally cannot learn. This within-session climb is the actual value proposition the live demo is built to demonstrate — and here, unlike the MCP benchmark's semantic panel, learning helps decisively.
Both benchmarks also ran a second scoring mode built for dense, signed features in [-1, 1]: a continuous proximity kernel, rather than a plain dot product. We ran it specifically to see how it behaves outside the conditions it was designed for — and it failed, in two different, fully diagnosed ways.
+1, not 0 — so on vectors that are 97%+ zeros, the kernel comes back fully dense regardless of actual overlap, injecting a near-uniform, term-irrelevant bias into the weight matrix on every vote. In the physics benchmark this collapsed the trained variant to 0.367 Precision@2, below the static baseline, with a learning curve that declines the longer the session runs.Neither failure is a fluke of one run — both were verified numerically on independent random vectors built the same way as each corpus. We're including this section because a benchmark that only reports the numbers that flatter the product isn't a benchmark; the kernel scorer is a real, useful tool for the dense-signed-feature domains it was designed for, and these results are exactly the boundary condition that tells us where not to reach for it.
Taken together, these two benchmarks say something more specific than "InnerMatch works." They say: online learning is not a universal multiplier — it adds clear, measurable value on top of sparse, per-dimension-interpretable feature spaces (TF-IDF keyword matching, TF-IDF-based RAG retrieval), and its benefit is muted or even slightly negative on top of dense latent spaces where individual dimensions carry no standalone meaning. That's a real property of the update mechanism's fit to different feature geometries, worth knowing before choosing a feature representation for a new InnerMatch integration — not a caveat we'd rather leave out.
It's also a demonstration of something we care about independent of any one product number: reproducibility. Both benchmarks parse their corpora directly from the same source our live demos use, run with a fixed seed, and disclose every substitution (embedding model, candidate-pool size, scoring mode) rather than tuning the setup to a favorable outcome. If a number here looks worse than you'd expect, that's deliberate — it's supposed to be the same benchmark whether the result flatters us or not.