Home › Best AI tools › AI model serving and deployment tools
AI model serving and deployment tools
14 inference engines and platforms for serving models in production. Inference engines and orchestration tools for running models behind an API on your own or cloud GPUs. Compare throughput features, hardware support and Kubernetes integration.
How to read this page. The tools are grouped by category and are not ranked. Pricing labels come from vendors' own pages when we could confirm them. Always verify plans and limits on the vendor's site.
Chatbots & Assistants (1)
| Tool | What it does | Pricing | Link |
|---|---|---|---|
| Ollama | Runs open language models locally through a command line and local API. | See site | Official site |
LLM APIs & Model Providers (4)
| Tool | What it does | Pricing | Link |
|---|---|---|---|
| LocalAI | Open-source, OpenAI-compatible API for running models locally. | Free | Official site |
| MLC LLM | Universal LLM deployment engine for running models on many devices. | Free | Official site |
| Modal | Serverless cloud for running AI workloads and model inference. | Freemium | Official site |
| TensorRT-LLM | NVIDIA open library for high-performance large language model inference. | Free | Official site |
AI Developer Tools (9)
| Tool | What it does | Pricing | Link |
|---|---|---|---|
| BentoML | Open-source framework and cloud for packaging and serving AI models. | See site | Official site |
| dstack | Open-source orchestration for AI training and inference across clouds. | See site | Official site |
| KServe | Open-source model serving on Kubernetes. | Free | Official site |
| llama.cpp | Open-source C/C++ library for running language models efficiently on local hardware. | Free | Official site |
| NVIDIA Dynamo-Triton | NVIDIA open-source inference serving software (formerly Triton Inference Server) for deploying AI models at scale. | See site | Official site |
| Ray | Open-source framework for scaling Python and AI workloads. | Free | Official site |
| SGLang | Open-source fast serving framework for large language and vision-language models. | Free | Official site |
| SkyPilot | Open-source framework for running AI workloads on any cloud or Kubernetes. | See site | Official site |
| vLLM | Open-source high-throughput inference engine for large language models. | Free | Official site |