DruxAI
← New search
⚡ TechnologyBeta — synthesis improving

Which open source LLM works best locally, and on what hardware?

2 views

3 modelsComplete
70%Consensus Score — 3 models mostly agreed, with some nuance. Read the breakdown.
🔴Semantic Entropy — Fragmented · 2 distinct meanings across 3 responsesH=58%
70%
consensus
Mostly aligned
How much the models agreed on this answer

High agreement — the answer is well-supported across models.

Models agree on

  • ✓Llama 3.1 8B and Mistral 7B are top choices for low-end hardware (4-8GB VRAM/CPU)
  • ✓Qwen2.5 14B and Gemma 2 27B are ideal for mid-range GPUs (12-24GB VRAM)
  • ✓Llama 3.1 70B/Qwen2.5 72B are best for high-end hardware (24GB+ VRAM/Apple Silicon)
  • ✓GGUF (Q4_K_M) is the recommended quantization format
  • ✓Ollama and llama.cpp are preferred tools for deployment

Points of disagreement

  • ~Coding model preference: DeepSeek-Coder vs. Qwen2.5-Coder
  • ~Apple Silicon's role (primary focus vs. secondary mention)
  • ~Inclusion of Llama 3.3 70B (GLM 5.3) vs. other models omitting it

The best open-source LLM for local use depends on your hardware, but here's a distilled guide based on current consensus and top-performing models:

Key Recommendations by Hardware Tier

  1. ·

    Low-end (4-8GB VRAM or CPU-only):

    • ·Top pick: 4-bit quantized Llama 3.1 8B (GGUF Q4_K_M) or Mistral 7B v0.3
    • ·Tools: llama.cpp (CPU) or Ollama for simplicity
    • ·Performance: 5-15 tokens/sec on CPU, usable for chat; GPU allows faster inference.
  2. ·

    Mainstream (12-24GB VRAM):

    • ·Best balance: Qwen2.5 14B (Q5_K_M) or Gemma 2 27B (Q4_K_M)
    • ·Coding focus: Qwen2.5-Coder 14B or DeepSeek-Coder-V2 Lite 16B
    • ·Hardware: RTX 3060 12GB, 3080, or 4090 (24GB for 32B models like Qwen2.5 32B).
  3. ·

    High-end (24GB+ VRAM or Apple Silicon):

    • ·Top-tier: Llama 3.1 70B or Qwen2.5 72B (Q4_K_M)
    • ·Apple M1/M2 Ultra: 70B models run efficiently due to unified memory.

Shared Tools and Formats

  • ·Quantization: GGUF (Q4_K_M is the sweet spot for quality/size).
  • ·Frameworks: llama.cpp (CPU/GPU flexibility), Ollama (user-friendly), or vLLM (GPU-optimized).

Divergences and Nuances

  • ·Coding models: DeepSeek-Coder variants are highlighted by some, while others prioritize Qwen2.5-Coder.
  • ·Apple Silicon: Some emphasize its unique advantage for 70B models, while others focus on NVIDIA GPUs.
  • ·Emerging models: GLM 5.3 mentions Llama 3.3 70B, which others omit (possibly a versioning discrepancy).

Quick Decision Guide

HardwareBest ModelUse Case
8GB VRAM/CPULlama 3.1 8B (Q4)General chat, light tasks
12-16GB VRAMQwen2.5 14B (Q5)Balanced performance
24GB+ VRAMLlama 3.1 70B (Q4)Frontier capabilities

For new users, start with Ollama (ollama run llama3.1:8b) and scale up based on hardware. The field evolves rapidly, but these recommendations balance current performance and accessibility.

Follow-ups

You just saw open-source models answer

Want GPT-5, Claude, Gemini & more on the same question?

Sign in free to run any question against frontier models — side by side, same synthesis, honest comparison.

GPT-5Claude SonnetGemini 2.5 ProGrokDeepSeek R1Perplexity Sonar
Free models only · sign in for premium