vLLM

What is vLLM?

vLLM is an open source inference and serving engine for large language models. It is maintained by the vLLM project and is designed to run models efficiently on GPUs, with an OpenAI-compatible server for applications. The problem it solves is serving cost and throughput: teams can host open models themselves and handle more concurrent requests on the same hardware instead of paying per token to a hosted API.

Key features

• PagedAttention. The engine manages key-value cache memory in pages, which reduces waste and allows more concurrent sequences.
• Continuous batching. Requests are batched dynamically as they arrive, improving GPU utilisation for chat and generation workloads.
• OpenAI-compatible server. A built-in HTTP server exposes chat and completion endpoints that work with OpenAI-style clients.
• Tensor and pipeline parallelism. Large models can be split across multiple GPUs and nodes.
• Quantisation support. Models can run in lower precision formats to reduce memory use and increase throughput.
• Prefix caching. Shared prompt prefixes can be cached to speed up repeated system prompts and multi-turn conversations.
• Broad model support. vLLM supports many open model architectures and Hugging Face checkpoints.

Who is vLLM for?

ML platform and infrastructure teams that run open models on their own GPUs, and companies that need control over model weights, data and latency. It suits teams serving chat, coding or batch generation at scale, organisations with privacy or compliance requirements that prevent sending data to a hosted API, and researchers benchmarking model performance. It requires GPU operations experience and is not a managed service.

Use cases

• Serving an open chat model behind an internal API.
• Running batch generation for summaries or classifications.
• Hosting a coding model for an internal developer assistant.
• Benchmarking throughput and latency across GPU configurations.
• Keeping sensitive prompts and data inside a private infrastructure.

Pricing

vLLM is free and open source under the Apache License 2.0. There is no paid plan from the project. The real cost is the GPU hardware or cloud capacity you run it on, plus the engineering time to operate the service. See the official vLLM documentation.

Getting started

1. Install vLLM in a Python environment with a supported GPU.
2. Start the OpenAI-compatible server with a model ID.
3. Send a chat completion request to the local endpoint.
4. Add authentication and a reverse proxy before exposing it to users.

pip install vllm
vllm serve Qwen/Qwen3-8B

Alternatives

• Ollama runs open models locally with a simple command line and API.
• LM Studio is a desktop app for running local models.
• OpenRouter provides a hosted API across many model providers.

FAQ

Is vLLM free?

Yes. vLLM is open source under the Apache License 2.0 with no licence fee. You pay for the GPUs and infrastructure it runs on.

What is vLLM used for?

vLLM is used to serve large language models efficiently. It provides high-throughput inference with an OpenAI-compatible API for chat, completion and batch workloads.

vLLM vs Ollama?

Ollama is aimed at running models locally with a simple developer experience. vLLM is aimed at production serving with high concurrency, multi-GPU support and advanced memory management. Ollama is easier to start with, while vLLM is built for throughput at scale.

Does vLLM work with OpenAI clients?

Yes. vLLM includes an OpenAI-compatible server, so clients that use the OpenAI API format can point at the local endpoint.

Can vLLM run on multiple GPUs?

Yes. It supports tensor parallelism and pipeline parallelism to split large models across multiple GPUs and nodes.