Merlin: Deterministic Byte-Exact Deduplication for Lossless Context Optimization in Large Language Model Inference

▤ Summary

Merlin is a local-first, deterministic byte-exact deduplication engine designed to optimize context for large language model inference and other text-heavy pipelines. It uses a SIMD-friendly open-addressing flat hash set with xxHash3-64 to achieve high-throughput, lossless deduplication and reports sustained processing speeds up to 8.7 GB/s. Empirical results show input reductions from about 13.9% in low-redundancy datasets to over 71% in highly redundant pipelines while preserving absolute data fidelity. The paper also documents integration via the Model Context Protocol (MCP) for secure, zero-network-interception deployment across IDEs and autonomous agents, and an implementation plus open-source community release is available.

▦ Linked Companies