Directory › Inference runtimes › TabbyAPI

TabbyAPI

Inference runtimes ✓ verified 🔒 local-only AGPL-3.0 ★ 1,275
GitHub ↗

TabbyAPI is a self-hostable server for ExLlamaV2 model inference, compatible with OpenAI's FastAPI framework. It allows users to run the quantized model on their local devices. The tool provides fast and efficient inference capabilities.

Who it's for: Developers and researchers who require a self-hostable AI inference solution for ExLlamaV2 models can use TabbyAPI.

Strengths

  • FastAPI OpenAI-compatible
  • Supports ExLlamaV2 quantized model inference
  • Self-hostable

Trade-offs

  • Limited to local-only data storage
  • No specific information on its performance or scalability

Data locality

Runs fully offline — no data leaves your machine.