Feature guide

Memory and throughput

• Prefix caching reuses shared prompt prefixes.
• Quantization reduces weight or KV cache memory.
• Speculative decoding can reduce decode latency.
• Deployment optimization explains scheduler and memory controls.

Model customization

• LoRA adapters serve one or more adapters with a base model.
• Tool calling parses model output into API tool calls.
• Reasoning and tool parsers select a parser that matches the model.
• Sampling and structured output constrains generated JSON or text.

Input types

• Multimodal inputs cover image, audio, and video models.
• Encoder-decoder models support sequence-to-sequence tasks.
• Pooling runners provide embeddings, classifications, rewards, and scores.

Operations

• Observability covers logs, metrics, and health checks.
• aphrodite bench measures latency, throughput, startup, and live-server load.
• Production deployment covers lifecycle and readiness.