Run all agent models through a single open-source inference engine
Instead of hosting models for search, document conversion, safety checks, and translation on separate servers, you can process them all on a single cluster. SIE is an inference engine that bundles over 100 open-source models required for agent tasks into one API. Models are loaded into memory only when a request arrives and are automatically removed when unused, ensuring efficient resource management.
Model configuration by agent task
SIE categorizes and provides models based on the specific task types an agent performs. Search and reranking include bge-m3, SPLADE, ColBERT, and Qwen3-reranker, while converting PDFs or Office documents to Markdown uses LightOnOCR, GLM-OCR, MinerU, PaddleOCR-VL, and Docling. Structured data extraction relies on GLiNER2, GLiForm, and Qwen3 series models, text classification uses the GLiClass series, and content safety judgments are handled by Qwen3Guard, Granite Guardian, and OPIR models. Additionally, Qwen3 series LLMs support auxiliary tasks such as executing agent loops, translating over 400 languages, image understanding, and speech recognition.
OpenAI-compatible API and integration ecosystem
When migrating from existing OpenAI API code to SIE, you can use endpoints like /v1/embeddings and /v1/chat/completions as-is, reducing migration overhead. Integration guides are provided for major agent frameworks like LangChain, LlamaIndex, Haystack, DSPy, and CrewAI, as well as vector databases such as Chroma, Qdrant, Weaviate, and LanceDB. It also supports offloading documents, images, and structured outputs from MCP (Mode Context Protocol) clients to the SIE cluster to reduce agent token costs.
From local execution to production deployment
During development, you can run it directly on a laptop by executing pip install "sie-server[local]" and then sie-server serve in a Python 3.12 environment. In NVIDIA GPU environments, you can launch a CUDA 12-based server via Docker images, with separate images provided for general use, Transformers 5, and SGLang vision extraction to prevent model dependency conflicts. For production, you can deploy a load balancing gateway, KEDA-based auto-scaling (including zero scaling), and Grafana dashboards together using Kubernetes and Helm charts. Terraform modules for Alibaba Cloud ACK, AWS EKS, Azure AKS, and GCP GKE are also maintained separately.
Model loading methods and performance characteristics
Models download weights and load into memory on the first call, skipping the download for subsequent calls. When serving multiple models simultaneously, an LRU (Least Recently Used) policy is applied to evict older models from memory. Inference latency varies depending on the model type, task type, hardware specifications, and batch size. Typed decision models like Laya and GLiNER2.5-Decide process multiple questions in a single forward pass, accounting for probabilistic dependencies between questions. The Laya series adjusts probabilities using the provided temperature value, whereas Laya-multilingual returns raw softmax probabilities without adjustment.
Pre-deployment checks
SIE is released under the Apache 2.0 license, allowing commercial use. Local execution requires Python 3.12, and GPU inference requires an NVIDIA CUDA 12 environment. While Apple Silicon Macs support running generative models via MLX, some vision OCR models are only available in specific Docker images on Linux GPU environments. SIE collects anonymous usage data such as version, OS, architecture, and GPU type, but does not collect IP addresses or request data; collection can be disabled with the SIE_TELEMETRY_DISABLED=1 environment variable. Detailed settings for each model and sie_id must be checked in the configuration files within the packages/sie_server/models/ directory.
