Reduce token usage by letting coding agents fetch only the specific code they need
Imagine an agent exploring a codebase without wasting tokens, focusing only on the essentials. XERJ automatically indexes code, documentation, logs, and PDFs from a specified folder into a searchable format. It supports the Elasticsearch API directly, allowing existing clients and dashboards to connect without modification.
Reference coding to reduce token consumption
When an agent uses grep to find files, it must load the entire content into the context, resulting in high token consumption. XERJ solves this by returning only indexed paths. In controlled experiments, it reduced output tokens by 2.7 times compared to grep-based agents and by 26 times compared to memory-based agents, while maintaining the same 16/16 resolution rate. In real development environments, users reported approximately 5 times token savings. This effect is more pronounced with internal or recent code than with popular public libraries that models already know.
Automatic indexing and diverse data support
A single xerj autoindex command scans all files in a folder, detects their types, and creates an index. No schema writing or pipeline configuration is required, and it indexes code symbols and line numbers using tree-sitter. It supports various formats including CSV, JSON, SQLite, PDF, DOCX, Jupyter notebooks, and Unity projects, creating a separate index for each dataset. Re-indexing processes only changed files to improve efficiency.
Elasticsearch compatibility and benchmarks
XERJ implements the Elasticsearch REST API, allowing the direct use of Kibana or official client libraries. It supports key features such as indexing, documents, bulk, search, aggregation, kNN, scroll, and re-indexing. In benchmarks compared to Elasticsearch 8.13.4, it recorded 55 wins, 26 ties, and 4 losses, with indexing throughput 1.72 times higher and disk usage 1.61 times smaller. The only disadvantage is read p99 latency under high write load. The consistency test suite passes 1366 out of 1369 cases.
Security model and installation
The development --insecure flag disables TLS and API key authentication, making any caller with port access a superuser. This is suitable for single-user use on local development machines, and new nodes bind to 127.0.0.1 by default. When authentication is enabled, the key must be passed via the XERJ_API_KEY environment variable or the --api-key option. Installation is a one-liner: curl -fsSL https://xerj.org/get | sh. It supports x86-64 and arm64 architectures on Linux, macOS, and Windows. It is a single binary written in Rust with no external dependencies.
Pre-use considerations
XERJ is distributed under the Apache 2.0 license, and community contributions are made by submitting short field reports. Using neural network-based semantic search requires pre-placing model files locally, but the default vocabulary embedder works offline. You must read the security model documentation and understand the risks of --insecure mode before using it. Recognize that it is an independent engine with API compatibility, not a reproduction of Elasticsearch's internal implementation, and consider the limitations of the scroll API (finite snapshot) by using search_after for large result sets.