Skip to content

Observer prompts miss the prompt cache because the first line is the current minute - #103

Merged
Seopril merged 2 commits into
elpapi42:masterfrom
rnavarro:pr/observer-timestamp-cache-prefix
Nov 2, 2026
Merged

Seopril merged 2 commits into
elpapi42:masterfrom
rnavarro:pr/observer-timestamp-cache-prefix

Conversation

@rnavarro

Copy link
Copy Markdown
Contributor

I was checking prompt cache hit rates across my pi sessions and found the memory workers well behind the main agent loop: about 72% of input tokens read from cache for worker calls, against 95% for the main loop on the same model. Observer requests were the worst. A typical observer prompt of tens of thousands of tokens read only the shared system prompt from cache, even when its reflections and observations were the same as the previous run.

Diffing consecutive observer requests, the first byte that differed was nearly always the time on the first line of the user message, for example 19:04 vs 19:06. The observer user message in src/agents/observer/agent.ts opens with Current local time: ${now}, and nowTimestamp() has minute resolution, so the prefix changes on almost every run and the reflections and observations after it are sent as uncached input.

This PR moves the Current local time: line below the reflections and observations, just ahead of the instruction paragraph, so those blocks stay in a stable prefix between runs. The instruction text says "fall back to the current local time above", which is still accurate, so no prompt wording changes.

A test in tests/observer.test.ts checks that the user message starts with CURRENT REFLECTIONS: and that the time line sits after the prior observations and before NEW CONVERSATION CHUNK:. It fails on the current layout. I also ran it in a live session and confirmed the observer request now starts with CURRENT REFLECTIONS:.

The dist/index.ts diff is larger than this change: CI on master has failed its git diff --exit-code -- dist step since #93, and a fresh npm run build also picks up the src/ changes the committed bundle was missing.

#96 also edits this user message and keeps the timestamp first. If #96 lands first, I can rebase this onto it.

@Seopril

Seopril commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator

Thanks for taking the time to open this PR. I reviewed it and it looks good to me. Would you mind rebuilding the dist/index.ts? I'll get it merged once that's done. I'll also be reviewing our pipeline to avoid having to do this for commits/pipelines that aren't being directly published to npm, so apologies for that.

The observer user message opened with the minute-resolution local time, so its prefix changed on every run and the reflections and observations after it missed the provider prompt cache. Move the time line below them, ahead of the instructions, so the instruction's 'current local time above' reference still holds.
@rnavarro
rnavarro force-pushed the pr/observer-timestamp-cache-prefix branch from 52b007e to 998fc63 Compare November 1, 2026 23:55
@rnavarro

rnavarro commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the feedback, updated!

@Seopril
Seopril merged commit a94a98d into elpapi42:master Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants