Contenu en anglaisCette ressource est actuellement disponible uniquement en anglais. La traduction vers d'autres langues est prévue dans une future mise à jour.

Resources/Landscape Guide

Open Source AI

Open Source AI Landscape 2026

The definitive guide to open source AI in 2026. Frontier models, training frameworks, inference servers, fine-tuning techniques, vector databases, and orchestration tools — with practical guidance on choosing the right stack for your use case.

11 Sections

45 min read

Code Examples Included

March 2026

Why Open Source AI Matters

In 2022, GPT-3.5 was widely considered unreachable by the open source community. The gap felt insurmountable. By 2026 the picture is dramatically different: Llama 4 Maverick is competitive with frontier closed models on most benchmarks, DeepSeek-R1 challenges OpenAI o1 on mathematical reasoning, and the open source ecosystem has produced specialized models that outperform closed equivalents in narrow domains.

For enterprises and developers, this means genuine choice for the first time. Open-weight models are no longer a fallback; they are often the first choice.

Data Privacy

The model runs entirely on your infrastructure. Your data never leaves your environment — critical for healthcare, legal, finance, and any regulated industry.

Cost at Scale

A single A100 cluster replaces $X/token API costs at volume. At 10M+ requests per month, self-hosted models typically deliver 5–20× cost reduction.

Customization

Fine-tune on your domain, your tone, your data. Closed APIs give you prompt engineering; open weights give you full model control.

The One Real Challenge

Operational burden. Self-hosting a model means you own infrastructure provisioning, model updates, monitoring, capacity planning, and incident response. Closed APIs outsource all of that. The question is never "is open source better?" — it's "do you have the engineering capacity to operate it reliably?"

The Open Source AI Stack

graph TB
  A["Foundation Models (Llama 4, Mistral Large 3, Qwen 2.5, DeepSeek R2)"] --> B["Fine-tuned / Instruction-tuned Variants"]
  B --> C["Inference Server (vLLM / TGI / Ollama)"]
  C --> D["Orchestration Layer (LangChain, LlamaIndex, CrewAI)"]
  D --> E["Application (RAG, Agents, Chatbot, Code assistant)"]

Frontier Open-Weight Models

The landscape as of early 2026. MMLU scores are indicative — always benchmark on your specific task before selecting a model for production.

Model	Org	Params	Context	License	MMLU	Best For
Llama 4 Maverick	Meta	400B (MoE)	1M	Llama 4	87.5	Frontier-competitive, multimodal
Llama 4 Scout	Meta	109B (MoE)	10M	Llama 4	79.6	Long context, efficient MoE
Llama 4 Behemoth	Meta	2T (MoE, preview)	256K	Llama 4	92.0	Maximum capability (teacher model)
Mistral Large 3	Mistral	123B	128K	MRL	84.0	Enterprise, European compliance
Mistral Small 3	Mistral	24B	128K	Apache 2.0	81.0	Efficient, permissive license
DeepSeek-R1	DeepSeek	671B (MoE)	128K	MIT	90.8	Reasoning, math, code
DeepSeek-R1-Distill-70B	DeepSeek	70B	128K	MIT	86.7	Efficient reasoning
Qwen2.5 72B	Alibaba	72B	128K	Qwen License	86.6	Multilingual, coding
Qwen2.5-Coder 32B	Alibaba	32B	128K	Apache 2.0	—	Code generation
Gemma 2 27B	Google	27B	8K	Gemma	75.2	Compact, well-optimized
Phi-4	Microsoft	14B	16K	MIT	84.8	Small but surprisingly capable

Understanding Licenses

Apache 2.0

Most Permissive

Most permissive for commercial use. Grants patent rights, allows modification and redistribution. Mistral favors this for their flagship models.

MIT

Very Permissive

Extremely permissive, minimal restrictions. DeepSeek releases under MIT, making their models among the most liberally licensed frontier models.

Llama 4

Conditional

Permissive for most commercial use, but requires a license agreement for products/services with > 700M monthly active users. Same conditions as Llama 3.

Open Weight vs Open Source

Key Concept

Important distinction: "open weight" means model weights are available, but training code and data may not be. True open source (like Mistral) releases both.

Specialized Models

General-purpose models are only the beginning. The open-source ecosystem has produced highly capable specialist models that outperform much larger general models within their domain.

Code Generation

Qwen2.5-Coder 32B— Top open-source coder, competitive with frontier models on HumanEval and SWE-bench

CodeLlama 70B— Meta's dedicated code model, strong infilling

StarCoder2 15B— BigCode collaboration, strong multi-language support

DeepSeek-Coder-V2— MoE architecture, excellent at competitive programming

Math & Reasoning

DeepSeek-R1 variants— Chain-of-thought reasoning, strong on MATH and AIME

QwQ-32B— Alibaba's reasoning model, approaches o1 quality

Numina-Math-72B— Competition math specialist, fine-tuned on competition data

Multimodal (Vision + Text)

LLaVA-1.6— Lightweight, widely deployable vision-language model

Qwen2-VL 72B— State-of-the-art open multimodal, strong on documents

Phi-3.5-vision— Microsoft's compact vision model, runs on edge devices

Embeddings

nomic-embed-text-v1.5— Matryoshka embeddings, adjustable dimension

mxbai-embed-large-v1— High quality English embeddings, Apache 2.0

BAAI/bge-m3— Multilingual (100+ langs), dense + sparse + colbert

Re-Rankers

BAAI/bge-reranker-v2-m3— Multilingual cross-encoder reranker

cross-encoder/ms-marco— Industry-standard passage reranking for RAG pipelines

European Enterprise Note

For European enterprises, Mistral's models (Apache 2.0 licensed for Mistral Small 3, EU-headquartered, EU-hosted options available) are often the default choice for compliance and data sovereignty reasons. Mistral Small 3 and Mistral Large 3 offer a permissive or commercial-friendly license with a clear European provenance that satisfies many procurement and data residency requirements.

Training Frameworks

Two frameworks dominate: PyTorch and JAX. Unless you have a specific reason to choose JAX, start with PyTorch — the ecosystem, tooling, and community support are unmatched.

PyTorch

De facto standard

Dynamic computation graphs, imperative execution style, and the largest ecosystem of any ML framework. Used by Meta, Microsoft, Hugging Face, and the vast majority of the research community.

Largest model and library ecosystem

Easiest debugging (Python-native)

Best community and documentation

Native FSDP for distributed training

JAX / Flax

Research & TPUs

Google's functional ML framework with XLA compilation. Excels on TPUs, enables function transformations (grad, jit, vmap, pmap). Flax and Equinox are the leading neural network libraries built on top.

Best performance on TPUs

XLA just-in-time compilation

Functional transforms (jit, vmap, pmap)

Preferred by Google DeepMind research

Key Training Libraries

transformersby HuggingFace

Load, fine-tune, and share any model from the Hub. The central library for the open-source AI ecosystem.

trlby HuggingFace

Supervised fine-tuning (SFT), RLHF, DPO, and GRPO training loops. The standard library for alignment training.

accelerateby HuggingFace

Single abstraction layer for multi-GPU, multi-node, and mixed-precision training. Write once, run everywhere.

deepspeedby Microsoft

ZeRO optimizer stages 1/2/3, 3D parallelism (tensor, pipeline, data). Required for training very large models.

FSDPby PyTorch native

Fully Sharded Data Parallel — PyTorch's native answer to DeepSpeed ZeRO. Simpler integration, comparable performance.

SFT Training Example (trl)

python

from transformers import AutoTokenizer, AutoModelForCausalLM
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
import torch

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-4-Scout-17B-16E-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-4-Scout-17B-16E-Instruct")
dataset = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:10000]")

trainer = SFTTrainer(
    model=model,
    args=SFTConfig(
        output_dir="./sft-output",
        num_train_epochs=3,
        per_device_train_batch_size=4,
    ),
    train_dataset=dataset,
    processing_class=tokenizer,
)
trainer.train()

Fine-Tuning: LoRA & QLoRA

Full fine-tuning requires as many GPUs as pretraining — prohibitive for most teams. Parameter-efficient fine-tuning (PEFT) methods make it possible to adapt frontier models on a single GPU.

LoRA: Low-Rank Adaptation

Instead of updating all model weights, LoRA adds small adapter matrices A and B alongside frozen weight matrices. Only the adapters are trained, reducing trainable parameters by up to 10,000x for a 7B model.

W' = W + ΔW = W + AB

where A ∈ ℝ^(d×r), B ∈ ℝ^(r×k), r << min(d, k)

The rank r controls the capacity of the adapters. Typical values: 8–64. Higher rank = more capacity but more parameters. At inference time, adapters can be merged into the base model for zero overhead.

QLoRA: Quantized LoRA

QLoRA quantizes the base model weights to 4-bit NF4 (Normal Float 4), then trains LoRA adapters in bfloat16. This allows fine-tuning a 70B model on just 2× A100 80GB GPUs — something that would normally require a 16-GPU cluster. Quality loss from quantization is minimal when adapters are trained in higher precision.

LoRA Configuration (peft)

python

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=16,                    # rank — controls adapter capacity
    lora_alpha=32,           # scaling factor (alpha/r = effective LR)
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 6,815,744 || all params: 8,036,802,560 || trainable%: 0.085

Method	GPU Memory (7B)	Trainable Params	Quality	Best Use Case
Full Fine-Tuning	~112 GB	100%	Highest	When quality is paramount and GPUs are abundant
LoRA	~16 GB	0.1–1%	Near-full	Style/format adaptation, instruction tuning
QLoRA	~6 GB	0.1–1%	95–98% of LoRA	Resource-constrained fine-tuning, 70B on 2 GPUs

DoRA: Weight-Decomposed LoRA

A newer variant that decomposes weight matrices into magnitude and direction components, then applies LoRA only to the direction component. Often achieves better quality than standard LoRA at the same rank. Supported in peft via use_dora=True.

Fine-Tune vs RAG: When to Use Each

Use Fine-Tuning when:

• Adapting output style or format
• Teaching a specialized domain vocabulary
• Consistent persona or tone is required
• Task-specific instruction following

Use RAG when:

• Grounding responses in specific documents
• Knowledge base is updated frequently
• Source attribution is required
• Reducing hallucinations on factual queries

Inference Servers

Once you have a model, you need to serve it. The choice of inference server determines your throughput, latency, and operational complexity. For production workloads, vLLM is the most widely adopted choice.

Server	Language	Best For	Quantization	Streaming	License
vLLM	Python	High-throughput production	GPTQ, AWQ, GGUF	✓	Apache 2.0
TGI	Rust/Python	HuggingFace stack	bitsandbytes, GPTQ	✓	Apache 2.0
Ollama	Go	Local development	GGUF (llama.cpp)	✓	MIT
llama.cpp	C++	Edge/CPU/Apple Silicon	GGUF all levels	✓	MIT
LMDeploy	Python	Fast inference + int4	W4A16, W8A8	✓	Apache 2.0
Triton Inference Server	C++	Multi-framework prod	Backend dependent	✓	BSD

vLLM PagedAttention Explained

Traditional inference allocates KV-cache in large contiguous blocks, wasting memory and preventing batching of requests with different sequence lengths. PagedAttention treats the KV-cache like virtual memory pages — blocks are allocated on demand and shared across requests when possible. This enables continuous batching (new requests join in-flight batches) and delivers 2–4× better GPU utilization over naive serving.

Starting vLLM (OpenAI-Compatible API)

bash

# Start vLLM OpenAI-compatible server
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --port 8000

python

from openai import OpenAI

# vLLM exposes an OpenAI-compatible API
client = OpenAI(base_url="http://localhost:8000/v1", api_key="token")

response = client.chat.completions.create(
    model="meta-llama/Llama-4-Scout-17B-16E-Instruct",
    messages=[{"role": "user", "content": "Explain attention mechanisms"}],
    temperature=0.7,
    max_tokens=512,
)
print(response.choices[0].message.content)

Local Inference

For development, air-gapped environments, or personal use, local inference tools let you run models on consumer hardware without a cloud account. Ollama is the easiest entry point.

Ollama

Manages model downloads, GGUF quantization, and exposes an OpenAI-compatible local API. No Python environment required.

bash

# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh

# Pull and run models
ollama run llama4:scout         # ~23 GB GGUF Q4_K_M (MoE, efficient)
ollama run mistral-small3       # ~14 GB GGUF Q4
ollama run deepseek-r1:70b      # ~40 GB
ollama run qwen2.5-coder:7b     # Code specialist

# List downloaded models
ollama list

GGUF Quantization Levels

Format	Bits/Weight	Quality	Recommended For
Q2_K	2-bit	Low	Absolute minimum RAM
Q4_K_M	4-bit	Good	Best quality/size balance — recommended default
Q5_K_M	5-bit	Very good	When you have extra RAM to spare
Q6_K	6-bit	Excellent	Near-lossless, large RAM available
Q8_0	8-bit	Near-lossless	Development, high-RAM systems
F16	16-bit	Lossless	Maximum quality, server GPU only

Hardware Guide

Hardware	Recommended Model
MacBook M2/M3/M4 (16GB)	8B Q4_K_M
MacBook M2 Pro (32GB)	13-14B Q4_K_M
MacBook M3 Max (64GB)	70B Q4_K_M
RTX 3090 24GB	13B Q8_0 or 30B Q4
A100 80GB	70B FP16 or Llama 4 Scout Q4
2× A100 80GB	Llama 4 Maverick Q4 or 70B FP16

LM Studio

Cross-platform GUI for local models. Browse and download from HuggingFace, OpenAI-compatible local server, hardware usage monitoring. Great for non-developer users.

Jan.ai

Privacy-first desktop LLM application. 100% offline, open source (AGPL), supports Ollama-compatible models. Built for users who want zero telemetry.

Vector Databases

Vector databases are the backbone of RAG systems. The right choice depends on scale, existing infrastructure, and whether you need metadata filtering alongside vector search.

Database	Type	Scale	License	Unique Feature
pgvector	PostgreSQL extension	Medium	Apache 2.0	SQL + vectors, zero new infra
Chroma	Embedded/server	Small-Medium	Apache 2.0	Simplest API, great for prototyping
Qdrant	Rust server	Large	Apache 2.0	Payload filtering, fast
Weaviate	Go server	Large	BSD	Hybrid search, GraphQL
Milvus	C++ server	Very Large	Apache 2.0	Billion-scale, cloud-native
LanceDB	Embedded	Medium	Apache 2.0	Arrow-native, serverless

Start with pgvector

If you already run PostgreSQL, pgvector adds vector search with zero new infrastructure. It handles millions of vectors comfortably with IVFFlat or HNSW indexes — more than enough for most production RAG systems.

pgvector Quickstart

sql

-- Enable the extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create table with vector column
CREATE TABLE documents (
    id        bigserial PRIMARY KEY,
    content   text,
    embedding vector(1536)    -- dimension matches your embedding model
);

-- Create approximate nearest neighbor index (IVFFlat)
CREATE INDEX ON documents USING ivfflat (embedding vector_cosine_ops)
  WITH (lists = 100);

-- Alternatively, HNSW (better recall, slower build)
-- CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops);

-- Semantic similarity query
SELECT content, 1 - (embedding <=> '[0.1, 0.2, ...]'::vector) AS similarity
FROM documents
ORDER BY embedding <=> '[0.1, 0.2, ...]'::vector
LIMIT 5;

Orchestration & Agents

Orchestration frameworks connect your model to tools, memory, and multi-step pipelines. The landscape is crowded — choose based on your use case, not on GitHub stars alone.

Framework	GitHub Stars	Best For	Abstraction Level
LangChain	90k+	General-purpose pipelines	High
LangGraph	10k+	Stateful agent workflows	Medium
LlamaIndex	35k+	RAG-heavy applications	Medium
CrewAI	20k+	Multi-agent collaboration	High
AutoGen	30k+	Conversational multi-agents	Medium
DSPy	20k+	Prompt optimization	Low-Medium
Semantic Kernel	20k+	.NET/enterprise integration	High
Haystack	15k+	NLP pipelines, open	Medium

DSPy: Prompt Optimization

DSPy takes a different philosophy from other frameworks: instead of hand-crafting prompt templates, you define a task signature (inputs, outputs, and constraints) and a few labeled examples, then DSPy automatically optimizes the prompts using algorithms like OPRO or BootstrapFewShot. This is particularly powerful when working with smaller open-source models that are sensitive to prompt phrasing — let the optimizer find what works rather than manually iterating.

Automatic few-shot selectionPrompt instruction tuningMulti-hop reasoning chainsWorks with any LLM

Evaluation Tools

Evaluation is where most open-source AI projects fail in production. Before deploying any model, define measurable quality criteria and establish a baseline.

lm-evaluation-harness

by EleutherAI

The standard benchmark runner for open-source models. Runs MMLU, HellaSwag, ARC, WinoGrande, and 60+ other benchmarks. Used to generate Open LLM Leaderboard scores.

OpenCompass

by Shanghai AI Lab

Comprehensive evaluation platform with 100+ benchmarks, especially strong coverage of Chinese language benchmarks and Asian language models.

Ragas

by Explodinggradients

RAG-specific evaluation framework. Measures context recall, faithfulness, answer relevancy, and context precision using LLM-as-judge methodology.

DeepEval

by Confident AI

Unit-test style evaluation framework. Write evaluation assertions in Python, integrate into CI/CD, track metrics over model versions.

Evals

by OpenAI

OpenAI's evaluation format has become an industry standard. Many open-source projects adopt the same eval structure for interoperability.

HELMET

by Princeton

Holistic evaluation of long-context language models. Critical for models claiming large context windows — tests actual long-context recall and reasoning.

Ragas RAG Evaluation Example

python

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall
from datasets import Dataset

eval_data = Dataset.from_dict({
    "question": ["What is LoRA?"],
    "answer": ["LoRA adds low-rank adapter matrices to frozen weights..."],
    "contexts": [["Low-Rank Adaptation adds trainable matrices A and B..."]],
    "ground_truth": ["LoRA is a parameter-efficient fine-tuning method..."],
})

result = evaluate(
    eval_data,
    metrics=[faithfulness, answer_relevancy, context_recall]
)
print(result)
# {'faithfulness': 0.96, 'answer_relevancy': 0.89, 'context_recall': 0.92}

Choosing Your Stack

There is no universal right answer. Work through these questions in order — each answer narrows your choices significantly.

1. Privacy constraints?

If data cannot leave your infrastructure, you are on the open-source-only path by default. This immediately rules out any managed API service. Size your infrastructure first.

2. What is your request volume?

< 1K req/day: Ollama on a single machine is fine. 1K–100K/day: vLLM on a single A100 node. > 100K/day: vLLM cluster or TGI behind a load balancer. At very high volumes, the cost savings over API access pay for the infrastructure in weeks.

3. GPU RAM budget?

A rough guide: 7B model ≈ 14 GB FP16 (or 5–6 GB Q4); 13B ≈ 26 GB; 70B ≈ 140 GB FP16 (or 40 GB Q4); 405B ≈ 810 GB FP16 (or 200 GB Q4). Add 20% overhead for KV-cache. QLoRA fine-tuning needs ~1.5× inference memory.

4. What is your primary task?

General chat → Llama 4 Scout. Code generation → Qwen2.5-Coder. Reasoning/math → DeepSeek-R1. Multilingual → Qwen2.5 72B. Document Q&A → Mistral Small 3 + pgvector. Each domain has a clear winner — do not use a general model when a specialist exists.

5. Do you need fine-tuning?

Style and format changes → LoRA (fast, cheap). Domain-specific knowledge → QLoRA + SFT on your corpus. Reasoning improvement → GRPO or DPO on preference data. If the base model behavior is close enough with prompting, skip fine-tuning entirely.

Recommended Starter Stacks

Use Case	Model	Serving	Orchestration	Vector DB
Internal chatbot	Llama 4 Scout	vLLM	LangChain	pgvector
Code assistant	Qwen2.5-Coder 7B	Ollama	Claude Code	—
Document Q&A	Mistral Small 3	vLLM	LlamaIndex	Qdrant
Multi-agent workflow	Llama 4 Scout	vLLM	LangGraph	pgvector
Reasoning tasks	DeepSeek-R1-Distill 7B	Ollama/vLLM	Custom	—
Privacy-critical	Llama 4 Scout	Ollama (air-gapped)	Custom	Chroma

Need Help Choosing and Deploying Your Open Source AI Stack?

Selecting the right model and infrastructure for your use case requires balancing performance, cost, compliance, and operational maturity. We help enterprises navigate these decisions and implement open-source AI systems that are reliable, private, and cost-effective at scale.

Related Resources

Production RAG Implementation Guide

Build retrieval-augmented generation systems that work in production with open-source vector databases

AI Agent Implementation Guide

Build production agents with open-source LLMs, from architecture to deployment

LLM Cost Optimization Guide

Reduce inference costs by 70–90% through model selection, quantization, and caching strategies