<![CDATA[Local LLMs]]>Megathread for local, open source, and small large language models

]]>https://forum.objects.foundation/topic/10/local-llmsRSS for NodeSun, 12 Jul 2026 22:40:35 GMTFri, 03 Jul 2026 21:21:04 GMT60<![CDATA[Reply to Local LLMs on Sun, 12 Jul 2026 16:11:35 GMT]]>Google Gemma 4

Hugging Face: https://huggingface.co/collections/google/gemma-4
GitHub: https://github.com/google-gemma
Website: https://ai.google.dev/gemma/docs/core
Announcement: https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/, https://blog.google/innovation-and-ai/technology/developers-tools/introducing-gemma-4-12B/

Developer: Google DeepMind
Released: 26B A4B and 31B: April 2026 / 12B Unified: June 2026
Variants: 12B Unified, 26B A4B, 31B
Parameters: 12B Unified: 11.95B / 26B A4B: 25.2B total, 3.8B active / 31B: 30.7B
Context: 256,000 tokens
Architecture: 31B: Dense / 26B A4B: MoE, 8 active of 128 experts plus 1 shared / 12B: Unified encoder-free dense, multimodal input projected directly into the decoder
License: Apache 2.0
Modalities: 12B: Text + Image + Audio / 26B A4B and 31B: Text + Image
Runs on: Laptop, consumer GPU/workstation class
Formats: GGUF via Ollama/LM Studio, safetensors via Hugging Face, Docker
On disk: 12B: 7GB Q4, estimated / 26B A4B: 15GB Q4, estimated / 31B: 18GB Q4, estimated

]]>https://forum.objects.foundation/post/77https://forum.objects.foundation/post/77Sun, 12 Jul 2026 16:11:35 GMT<![CDATA[Reply to Local LLMs on Sun, 12 Jul 2026 16:02:19 GMT]]>Google Gemma 4 Effective

Hugging Face: https://huggingface.co/collections/google/gemma-4
GitHub: https://github.com/google-gemma
Website: https://ai.google.dev/gemma/docs/core
Announcement: https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/

Developer: Google DeepMind
Released: April 2026
Variants: E2B, E4B
Parameters: E2B: 2.3B effective, 5.1B with embeddings / E4B: 4.5B effective, 8B with embeddings
Context: 128,000 tokens
Architecture: Dense, hybrid local sliding window + global attention, Per-Layer Embeddings for on-device efficiency
License: Apache 2.0
Modalities: Text + Image + Audio
Runs on: Smartphone, Laptop, Edge device
Formats: GGUF via Ollama/LM Studio, safetensors via Hugging Face
On disk: E2B: 1.4GB Q4, estimated / E4B: 2.7GB Q4, estimated

]]>https://forum.objects.foundation/post/76https://forum.objects.foundation/post/76Sun, 12 Jul 2026 16:02:19 GMT<![CDATA[Reply to Local LLMs on Sun, 12 Jul 2026 15:41:53 GMT]]>PrismML Bonsai Image 4B

Hugging Face: https://huggingface.co/prism-ml/bonsai-image-binary-4B-mlx-1bit, https://huggingface.co/prism-ml/bonsai-image-ternary-4B-mlx-2bit
GitHub: https://github.com/PrismML-Eng/Bonsai-Image-Demo
Website: https://prismml.com
Announcement: https://prismml.com/news/bonsai-image-4b

Developer: PrismML
Released: May 2026
Parameters: 4B, transformer trunk
Architecture: MMDiT diffusion transformer, base architecture FLUX.2 Klein 4B, 25 blocks: 5 double-stream + 20 single-stream
Variants: Binary, 1-bit / Ternary, 2-bit
License: Apache 2.0
Modalities: Text-to-Image
Runs on: Smartphone, Laptop
Formats: MLX 1-bit, MLX 2-bit, Gemlite 1-bit/2-bit for CUDA, safetensors
On disk: Binary: 0.93GB transformer, 3.42GB total deployment payload / Ternary: 1.21GB transformer

]]>https://forum.objects.foundation/post/75https://forum.objects.foundation/post/75Sun, 12 Jul 2026 15:41:53 GMT<![CDATA[Reply to Local LLMs on Sun, 12 Jul 2026 15:26:31 GMT]]>Qwen3.6

Hugging Face: https://huggingface.co/Qwen/Qwen3.6-27B, https://huggingface.co/Qwen/Qwen3.6-35B-A3B
GitHub: https://github.com/QwenLM/Qwen3.6
Website: https://qwen.ai/

Developer: Alibaba Cloud, Qwen team
Released: April 2026
Parameters: 27B / 35B total, 3B active
Context: 262,144 tokens, extensible to 1,010,000
Architecture: Dense / MoE, 256 experts, 8 routed + 1 shared active
License: Apache 2.0
Modalities: Text + Vision
Runs on: Laptop, 24GB+ unified memory minimum
Formats: GGUF, MLX 4-bit, community quantizations
On disk: 27B: 16GB Q4_K_M, estimated / 35B-A3B: 21GB Q4_K_M, estimated

]]>https://forum.objects.foundation/post/74https://forum.objects.foundation/post/74Sun, 12 Jul 2026 15:26:31 GMT<![CDATA[Reply to Local LLMs on Sun, 12 Jul 2026 15:05:13 GMT]]>Liquid AI LFM2-24B-A2B

Hugging Face: https://huggingface.co/LiquidAI/LFM2-24B-A2B
GitHub: https://github.com/Liquid4All/cookbook
Website: https://www.liquid.ai
Announcement: https://www.liquid.ai/blog/lfm2-24b-a2b

Developer: Liquid AI
Released: February 2026
Parameters: 24B total, 2.3B active
Context: 32,768 tokens
Architecture: MoE, 40 layers: 30 conv + 10 attention, 64 experts, top-4 routing
License: LFM Open License v1.0
Modalities: Text
Runs on: Laptop, 14GB+ unified memory minimum, 32GB+ recommended
Formats: GGUF Q4_K_M, Q5_K_M, Q6_K, safetensors, ONNX
On disk: 14.44GB Q4_K_M / 16.93GB Q5_K_M / 19.58GB Q6_K

]]>https://forum.objects.foundation/post/72https://forum.objects.foundation/post/72Sun, 12 Jul 2026 15:05:13 GMT<![CDATA[Reply to Local LLMs on Sun, 12 Jul 2026 14:53:22 GMT]]>Liquid AI LFM2.5-VL-450M

Hugging Face: https://huggingface.co/LiquidAI/LFM2.5-VL-450M
Website: https://www.liquid.ai
Announcement: https://www.liquid.ai/blog/lfm2-5-vl-450m

Developer: Liquid AI
Released: April 2026
Parameters: 450M
Context: 32,768 tokens
Architecture: hybrid gated-convolution + grouped-query attention LM backbone, LFM2.5-350M, with SigLIP2 NaFlex 86M vision encoder
License: LFM Open License v1.0
Modalities: Text + Vision, bounding box / object detection support
Runs on: Smartphone, Laptop, Edge device
Formats: GGUF, ONNX, MLX 4-bit/5-bit/6-bit/8-bit/bf16, safetensors
On disk: 270MB Q4, estimated

]]>https://forum.objects.foundation/post/71https://forum.objects.foundation/post/71Sun, 12 Jul 2026 14:53:22 GMT<![CDATA[Reply to Local LLMs on Sun, 12 Jul 2026 14:34:01 GMT]]>IBM Granite 4.1 3B

Hugging Face: https://huggingface.co/ibm-granite/granite-4.1-3b
GitHub: https://github.com/ibm-granite/granite-4.1-language-models
Announcement: https://huggingface.co/blog/ibm-granite/granite-4-1

Developer: IBM
Released: April 2026
Parameters: 3B
Context: 512,000 tokens
Architecture: dense decoder-only transformer
License: Apache 2.0
Modalities: Text
Runs on: Smartphone, 2GB at 4-bit, estimated, Laptop, Edge device
Formats: GGUF Q4_K_M, Q5_K_M, Q6_K, Q8_0, MLX/Core ML via conversion
On disk: 2.2GB Q4_K_M, estimated

]]>https://forum.objects.foundation/post/70https://forum.objects.foundation/post/70Sun, 12 Jul 2026 14:34:01 GMT<![CDATA[Reply to Local LLMs on Sun, 12 Jul 2026 14:46:56 GMT]]>Apple Foundation Model 3 Core Advanced

Framework: Apple Foundation Models
Docs: https://developer.apple.com/documentation/FoundationModels

Developer: Apple
Released: June 2026, WWDC
Parameters: 20B total, 1-4B active
Context: 4,096 tokens
Architecture: Sparse MoE with Instruction-Following Pruning
License: Proprietary, OS-bundled
Modalities: Text + Image + Audio
Runs on: Smartphone, Laptop, 12GB+ RAM

]]>https://forum.objects.foundation/post/69https://forum.objects.foundation/post/69Sun, 12 Jul 2026 14:46:56 GMT<![CDATA[Reply to Local LLMs on Sun, 12 Jul 2026 14:12:15 GMT]]>Apple Foundation Model 3 Core

Docs: https://developer.apple.com/documentation/FoundationModels

Developer: Apple
Released: June 2026, WWDC
Parameters: 3B
Context: 4,096 tokens
Architecture: Dense
License: Proprietary, OS-bundled
Modalities: Text
Runs on: Smartphone, Laptop

]]>https://forum.objects.foundation/post/68https://forum.objects.foundation/post/68Sun, 12 Jul 2026 14:12:15 GMT<![CDATA[Reply to Local LLMs on Sun, 12 Jul 2026 13:51:46 GMT]]>Liquid AI LFM2.5

Hugging Face: https://huggingface.co/LiquidAI
Website: https://www.liquid.ai
Announcement: https://www.liquid.ai/blog/lfm2-5-230m, https://www.liquid.ai/blog/lfm2-5-350m-no-size-left-behind, https://www.liquid.ai/blog/lfm2-5-1-2b-thinking-on-device-reasoning-under-1gb, https://www.liquid.ai/blog/lfm2-5-8b-a1b

Developer: Liquid AI
Released: Jan-Jun 2026, rolling family
Parameters: LFM2.5-230M: 230M / LFM2.5-350M: 350M / LFM2.5-1.2B-Thinking: 1.2B / LFM2.5-8B-A1B: 8B total, 1B active
Context: 230M/350M/1.2B: 32K / 8B-A1B: 128K
Architecture: hybrid gated-convolution + grouped-query attention; dense for 230M/350M/1.2B, MoE for 8B-A1B
License: LFM Open License v1.0
Modalities: Text
Runs on: 230M/350M: Smartphone, Laptop, Edge device / 1.2B-Thinking: Smartphone, Laptop / 8B-A1B: Laptop, 8GB+ unified memory
Formats: GGUF Q4_K_M, MLX 4-bit, ONNX
On disk: 230M: 153MB / 350M: 250MB est. / 1.2B-Thinking: 720MB est. / 8B-A1B: 4.8GB est.

]]>https://forum.objects.foundation/post/67https://forum.objects.foundation/post/67Sun, 12 Jul 2026 13:51:46 GMT<![CDATA[Reply to Local LLMs on Sun, 12 Jul 2026 14:25:09 GMT]]>

From PrismML:

PrismML Bonsai 1-bit

Hugging Face: https://huggingface.co/prism-ml/Bonsai-8B-mlx-1bit, https://huggingface.co/prism-ml/Bonsai-4B-mlx-1bit, https://huggingface.co/prism-ml/Bonsai-1.7B-mlx-1bit
GitHub: https://github.com/PrismML-Eng/Bonsai-demo
Website: https://prismml.com
Announcement: https://prismml.com/news/bonsai-8b

Developer: PrismML
Released: March 2026
Parameters: 8B: 8.19B / 4B: 4.0B / 1.7B: 1.7B
Context: 8B: 65,536 / 4B: 32,768 / 1.7B: 32,768
Architecture: Qwen3 dense base, GQA, SwiGLU MLP, RoPE, RMSNorm; 8B: 36 layers, 32Q/8KV heads / 4B: 36 layers, 32Q/8KV heads / 1.7B: 28 layers, 16Q/8KV heads
License: Apache 2.0
Modalities: Text
Runs on: Smartphone, Laptop
Formats: MLX 1-bit g128, GGUF Q1_0_g128
On disk: 8B: 1.28GB MLX, 1.15GB GGUF / 4B: 0.63GB MLX, 0.57GB GGUF / 1.7B: 0.27GB MLX, 0.24GB GGUF

]]>https://forum.objects.foundation/post/20https://forum.objects.foundation/post/20Sun, 12 Jul 2026 14:25:09 GMT