Inference runtimes

Run open LLMs locally via a simple CLI and OpenAI-compatible REST API; pulls and serves GGUF models

C/C++ LLM inference engine for GGUF models, runs on CPU/GPU across platforms

High-throughput, memory-efficient inference and serving engine for LLMs

Desktop app and bindings to run local LLMs privately on consumer hardware

OpenAI-compatible API to run LLM, image and audio models locally, no GPU required

Gradio web UI for running and chatting with local LLMs across many backends

Open-source offline ChatGPT-alternative desktop app running local models

Fast serving framework for LLMs and vision-language models

Apple-silicon array framework for on-device ML including local LLM inference

Distribute and run an LLM as a single cross-platform executable file

Single-file llama.cpp-based runner with a UI for local text and story generation

Hugging Face production toolkit for deploying and serving LLMs

FastAPI OpenAI-compatible server for ExLlamaV2 quantized model inference

Desktop app to discover, download and run local LLMs with a GUI and local server