Skip to content

Show cross-repository MCP example and measured internal results #253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
m1rl0k merged 1 commit into from
Oct 19, 2026
Merged

Show cross-repository MCP example and measured internal results #253

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions e2e/demo.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ test('home page explains model-agnostic intelligence and paid Starter pricing',
await page.goto('/');
await expect(page.locator('h1')).toContainText('Codebase intelligence');
await expect(page.locator('.hero-subtitle')).toContainText('agents, IDEs, and any model');
await expect(page.locator('.code-window')).toContainText('Who consumes CheckoutCreated?');
await expect(page.locator('.code-window')).toContainText('Illustrative source-backed context');
await page.getByText('Read the method and trade-offs', { exact: true }).click();
await expect(page.locator('.evaluation-method')).toContainText(
'not a general coding-success rate'
);
await expect(page.locator('.evaluation-method')).toContainText('183 to 216 seconds');
await expect(
page.getByRole('heading', { name: 'Organization Knowledge', exact: true })
).toBeVisible();
Expand Down
32 changes: 31 additions & 1 deletion src/app.scss
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1101,6 +1101,36 @@ body::before {
font-weight: 500;
}

.evaluation-results {
display: grid;
grid-template-columns: repeat(3, minmax(0, 1fr));
gap: 24px;
border-block: 1px solid var(--border);
padding: 32px 0;
}

.evaluation-results strong {
display: block;
color: var(--accent);
font-size: clamp(24px, 3vw, 40px);
font-weight: 500;
}

.evaluation-results span, .evaluation-method {
color: var(--text-secondary);
}

.evaluation-method {
margin-top: 24px;
max-width: 850px;
font-size: 14px;
}

.evaluation-method summary {
cursor: pointer;
min-height: 44px;
}

.open-source-foundations {
max-width: 1280px;
margin: 0 auto 40px;
Expand Down Expand Up @@ -1143,7 +1173,7 @@ a:focus-visible, button:focus-visible {
}

@media (max-width: 768px) {
#capabilities .features-grid, .open-source-links {
#capabilities .features-grid, .open-source-links, .evaluation-results {
grid-template-columns: 1fr;
}
}
Expand Down
53 changes: 45 additions & 8 deletions src/routes/+page.svelte
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,15 +300,22 @@
<span class="dot red"></span>
<span class="dot yellow"></span>
<span class="dot green"></span>
<span class="code-title">mcp_tools.py</span>
<span class="code-title">MCP · cross_repo_search</span>
</div>
<pre class="code-content"><span class="c-comment"># Find authentication flow</span>
result = <span class="c-fn">repo_search</span>(
query=<span class="c-str">"user authentication"</span>,
limit=<span class="c-num">5</span>,
compact=<span class="c-key">True</span>
)
<span class="c-comment"># Grounded source evidence</span></pre>
<pre class="code-content"><span class="c-comment">// Your agent asks across the business</span
>
<span class="c-fn">cross_repo_search</span>({'{'}
<span class="c-key">"query"</span>: <span class="c-str">"Who consumes CheckoutCreated?"</span>,
<span class="c-key">"target_repos"</span>: [<span class="c-str">"storefront"</span>, <span
class="c-str">"billing"</span
>, <span class="c-str">"shared"</span>],
<span class="c-key">"trace_boundary"</span>: <span class="c-key">true</span>
{'}'})

<span class="c-comment">// Illustrative source-backed context</span>
<span class="c-str">shared/events.ts</span> <span class="c-comment">→ event contract</span>
<span class="c-str">storefront/checkout.ts</span> <span class="c-comment">→ publisher</span>
<span class="c-str">billing/invoices.ts</span> <span class="c-comment">→ consumer</span></pre>
</div>

<div class="hero-visual-features">
Expand All @@ -335,6 +342,36 @@ result = <span class="c-fn">repo_search</span>(
</section>

<!-- Graph Section -->
<section class="features-section" aria-labelledby="evaluation-title">
<p class="open-source-label">
Measured with real agent tasks · internal evaluation · October 5, 2026
</p>
<h2 class="section-title" id="evaluation-title">
More relevant context. <span class="gradient-text">Less searching around.</span>
</h2>
<div class="evaluation-results">
<div><strong>+12 points</strong><span>file-and-symbol recall score</span></div>
<div><strong>14% fewer</strong><span>tokens per task, on average</span></div>
<div><strong>14% fewer</strong><span>tool calls per task, on average</span></div>
</div>
<details class="evaluation-method">
<summary>Read the method and trade-offs</summary>
<p>
Historical internal development study: Claude Sonnet with and without Context Engine, 40 tasks
across two internal repositories, pooled over two paired cohorts (80 comparisons). The recall
score is the average of correct-file recall and correct-symbol recall, not a general
coding-success rate. It rose from 0.522 to 0.643; the paired increase had a 95% bootstrap
interval of +0.038 to +0.204.
</p>
<p>
Mean reported tokens fell from 1,014,587 to 876,388 and tool calls from 25.512 to 21.837. The
trade-off: mean task time increased from 183 to 216 seconds. Cost savings were inconclusive.
Figures are recovered from archived evaluator output, not a fresh run of today’s deployment or
an independent benchmark. Results depend on workload, model and index freshness.
</p>
</details>
</section>

<section class="graph-section">
<h2 class="section-title">How It <span class="gradient-text">Works</span></h2>
<p class="section-subtitle">
Expand Down
Loading