PRECOG: Optimizing Edge Language Models with O(1) Persistent State Retrieval

Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State

Those states are injected at query time, reducing the stated prefill cost to O(1). The proposal targets a specific systems bottleneck: moving corpus processing out of the interactive inference path rather than expanding the context window.

The result is an architectural claim, not yet a benchmark result. The available evidence identifies the mechanism but provides no reported accuracy, latency, memory, hardware, corpus, or baseline comparison.

The proposed execution model

PRECOG separates corpus preparation from query execution. Documents are processed offline and represented as fixed-size recurrent hidden states. At query time, the model receives those precomputed states instead of recomputing the full corpus representation.

The important term is “fixed-size.” The paper’s stated mechanism does not describe a retrieval index with an expanding sequence of document tokens. It describes a recurrent state representation that can be injected into an SSM. This is a different trade-off from conventional context-based retrieval, where the cost of handling retrieved material is commonly tied to the amount of text passed into the model.

The O(1) claim applies to prefill costs in the description supplied by arXiv. It should not be read as an assertion that total inference cost, memory traffic, state construction, or query latency is constant under all workloads. Offline encoding still exists as a separate phase. The available material does not specify how often states must be regenerated, how many states can be injected, or how retrieval quality changes as the corpus grows.

For edge deployment, this distinction is operationally relevant. A system can move computation away from the user-facing path, but it must then manage persistent representations and their validity. No evidence is provided on whether PRECOG supports incremental corpus updates, source attribution, state eviction, or conflicting documents.

What the evidence supports—and what it does not

The supported claim is narrow: PRECOG introduces a retrieval mechanism for SSMs based on offline corpus encoding and query-time state injection. The mechanism is presented as reducing prefill costs to O(1).

The evidence does not establish that PRECOG improves answer quality. There is no confirmed ablation study comparing recurrent hidden-state injection with token-based retrieval, standard SSM baselines, or other memory architectures. There is also no reported parameter-efficiency analysis. Without these measurements, it is not possible to determine whether the fixed-size state preserves the distinctions required for retrieval, or whether computational overhead has simply shifted from query time to preprocessing and storage.

The absence of benchmark details is material. A useful evaluation would need at least a corpus scale, query distribution, model configuration, hardware setup, end-to-end latency, memory consumption, and retrieval or generation quality. It would also need to separate the cost of state construction from the cost of serving queries. An O(1) prefill statement is insufficient to estimate the complete deployment profile.

The implementation audience should apply the same accounting to other constrained runtimes, including projects that move substantial work out of an interactive path, such as running retro Flash games with a WebAssembly emulator. The analogy is limited: PRECOG is a model-memory architecture, not an emulation technique. The shared engineering question is where computation is performed and which state must persist between requests.

Practical checks before adoption

The first check is reproducibility. An implementation should verify that the claimed constant prefill behavior holds when corpus size, number of retrieved states, and query length vary independently. Measuring only a fixed corpus would not validate the scaling claim.

The second check is representational fidelity. Fixed-size recurrent states impose a compression boundary. Evaluation should test exact retrieval, multi-document queries, stale documents, and near-duplicate content. These tests are necessary to determine whether the latent representation retains the information needed by downstream generation.

The third check is lifecycle cost. Offline encoding, state storage, updates, and invalidation belong in the system budget. The supplied evidence does not indicate how PRECOG handles any of them. Until those measurements are available, the paper is best treated as an architectural proposal with a precise optimization target, not as evidence of a generally superior retrieval system.

For developers working on edge language models, the immediate action is therefore not to assume a speedup. It is to reproduce the prefill measurement, define the accounting boundary, and compare quality and total computational overhead against a token-based baseline.