Sonar

Serve Hugging Face models with an OpenAI-compatible API or a Python API.

Install

curl -fsSL https://sonar.dphn.ai/install.sh | bash

The installer detects the platform and accelerator. It can create a Python environment and install a release or nightly build.

If you do not know which workflow applies, start with Choose a path.

Install Sonar Choose an installation procedure for your accelerator.

Start an API server Serve a model through OpenAI-compatible HTTP endpoints.

Use the Python API Run local and batched inference with the LLM class.

Choose parallelism Select a layout for one GPU, one node, or multiple nodes.

Optimize a deployment Tune memory use, batching, startup time, and request latency.

Check model support Read model and quantization tables generated from the source tree.