If you use Claude or ChatGPT heavily for coding, you know the feeling: you’re deep in a debugging session, firing off prompts one after another, and somewhere in the back of your mind you’re thinking “how much is this costing me right now?”
That mental tax is real — and it changes how you use AI. You start batching questions. You hold back from asking “dumb” things. You self-censor prompts that feel too long. That’s the opposite of what a coding assistant is supposed to do.
Running a model locally flips that. No API meter. No usage anxiety. Ask whatever, as many times as you want. Google released Gemma 4 on April 2, 2026 — and it’s probably the most practical open-weight model you can run on your own machine right now. Let’s set it up the right way.
What Is Gemma 4, Really?
What Is Gemma 4, Really?Gemma 4 is Google DeepMind’s fourth generation of open-weight models, built on the same research foundation as Gemini 3. It comes in four sizes, and the whole family is released under the Apache 2.0 license — meaning you can use it commercially, modify it, and redistribute it without restrictions. That’s a notable upgrade from earlier Gemma releases.
📊 Gemma 4 Model Lineup — Pick the Right One for Your Hardware
| Model | RAM (4-bit quant) | Context | Best For |
|---|---|---|---|
| E2B | ~4 GB | 128K | Old laptops, budget setups |
| E4B ⭐ Recommended | ~5.5–6 GB | 128K | Most laptops (16GB+ RAM) |
| 26B A4B (MoE) | ~16–18 GB | 256K | Mac mini/Studio, 24GB+ |
| 31B Dense | ~20+ GB | 256K | Workstations, high-end Macs |
⚠️ The 26B is a Mixture-of-Experts model: all 26B parameters load into memory, but only ~4B activate per inference. Fast, but still needs the RAM to load the full model.
The Setup (5 Minutes, Seriously)
Ollama is the easiest way to run local models. It handles model downloads, the inference server, and the OpenAI-compatible API endpoint automatically. Here’s the complete setup for Mac:
Install Ollama
brew install –cask ollama
Pull Gemma 4 (choose your size)
ollama pull gemma4:e4b# For Mac mini / 24GB+ — ~16GB download
ollama pull gemma4:26b# Verify download
ollama list
Run and test it
ollama run gemma4:e4b
# Or chat in your browser via the REST API:
curl http://localhost:11434/api/chat -d ‘{
“model”: “gemma4:e4b”,
“messages”: [{“role”:”user”,”content”:”explain this function”}]
}’
🍎 Apple Silicon Users:
Ollama v0.19+ automatically uses Apple’s MLX framework on M-series chips — no manual setup needed. This gives you noticeably faster inference compared to older versions. Make sure you’re on the latest Ollama before you benchmark anything.
Stop the “Cold Start” Problem
By default, Ollama unloads the model from memory after 5 minutes of inactivity. For a coding session where you switch to the browser and come back, that means a slow reload every single time. Fix it in two steps:
Step 1 — Tell Ollama to keep the model loaded indefinitely:
export OLLAMA_KEEP_ALIVE=”-1″# Reload your shell
source ~/.zshrc
Step 2 — Keep it warm in memory with a launchd ping (Mac):
cat << ‘EOF’ > ~/Library/LaunchAgents/com.ollama.preload.plist
<?xml version=”1.0″ encoding=”UTF-8″?>
<!DOCTYPE plist PUBLIC “-//Apple//DTD PLIST 1.0//EN”
“http://www.apple.com/DTDs/PropertyList-1.0.dtd”>
<plist version=”1.0″>
<dict>
<key>Label</key><string>com.ollama.preload</string>
<key>ProgramArguments</key>
<array>
<string>/opt/homebrew/bin/ollama</string>
<string>run</string>
<string>gemma4:e4b</string>
<string></string>
</array>
<key>StartInterval</key><integer>300</integer>
<key>RunAtLoad</key><true/>
</dict>
</plist>
EOFlaunchctl load ~/Library/LaunchAgents/com.ollama.preload.plist
This sends an empty prompt every 5 minutes, keeping Gemma 4 resident in memory. Your Mac will use a bit more RAM as a trade-off — but no more cold starts mid-session.
Use It Everywhere — The OpenAI-Compatible API
This is the part people underestimate. Ollama exposes Gemma 4 at http://localhost:11434/v1 using the same format as the OpenAI API. That means any tool designed for OpenAI can talk to your local model with zero code changes — just swap the base URL.
🔌 Tools That Connect to Ollama Out of the Box
Continue (VS Code)
Inline code completions + chat in your editor
OpenCode / Aider
Terminal-based coding agents
Open WebUI
ChatGPT-like browser interface for Ollama
LangChain / LlamaIndex
Plug into your Python projects directly
Connecting to Continue (VS Code extension):
{
“models”: [{
“title”: “Gemma 4 (Local)”,
“provider”: “ollama”,
“model”: “gemma4:e4b”
}]
}
Honest Assessment: What It’s Actually Good At
After the HN community got their hands on Gemma 4, the verdict was nuanced. Here’s a realistic breakdown:
✅ Where Gemma 4 Holds Its Own
- Explaining code you’re reading for the first time
- Writing boilerplate and repetitive code blocks
- Generating utility functions and helper methods
- Refactoring a single file (stays coherent within ~128K context)
- Debugging straightforward errors with clear stack traces
- Writing tests for already-written functions
- Translating between languages (Python ↔ JS, SQL ↔ code)
⚠️ Where It Struggles
- Multi-file refactors that require understanding the whole codebase
- Deep architectural decisions and complex system design
- Subtle edge-case bugs that require domain expertise
- Tool calling in some setups — especially right after launch
- Anything where Claude Sonnet / GPT-4o meaningfully outperforms 8B models
⚡ The Tool Calling Issue — Read This Before You Rage Quit
Multiple users reported consistent tool call failures right after Gemma 4’s launch — not because the model can’t do it, but because quantization implementations shipped with bugs. Google’s Ollama page confirms tool calling is supported, but early GGUF builds from third parties had broken implementations. If tool calls fail, make sure you’re on the latest Ollama and using the official gemma4:latest tag rather than community quants. And if you’re using reasoning mode — try disabling it in your config. Reasoning mode can break expected tool-call formatting.
The Memory Mistake Everyone Makes
The most common setup failure is pulling too big a model for your RAM. One developer shared that running the 26B on a 24GB Mac mini took ~17GB — leaving barely 7GB for macOS itself and everything else running. The moment a second request came in, the system froze.
Quick RAM Rule of Thumb:
Model size at 4-bit quantization + at least 6–8 GB overhead for your OS and other apps.
→ 16GB Mac? Stick with E4B (~6GB model)
→ 24GB Mac mini? 26B works but runs tight. E4B is more comfortable.
→ 36GB Mac? 26B is your sweet spot. 31B is possible.
Check how well your GPU is being used after loading the model:
ollama ps
# Good output:
gemma4:e4b 100% GPU ← Full GPU acceleration
# Bad output:
gemma4:26b 14% CPU 86% GPU ← CPU spillover = slow
If you see CPU layers, you’ve exceeded your GPU memory. Either switch to a smaller model or a lower quantization.
The Smarter Play: Hybrid Routing
Here’s how experienced developers who care about both cost and quality actually use local models: they don’t pick one or the other. They route by task complexity.
The win isn’t replacing Claude. The win is not needing Claude for the 70% of prompts you used to send it out of habit.
One Last Thing: Timing Matters
The open model community moves fast — sometimes too fast. Gemma 4 shipped on April 2, and within days there were reports of broken quantizations, failed tool calls, and unreliable outputs from third-party GGUF builds.
Good news: as of mid-April, Unsloth has patched their GGUF builds and Google updated the official chat template. If you’re setting this up now, you’re landing in a much more stable place than day-one adopters. Just keep two things in mind:
- Always use the official Ollama model tag (
gemma4:e4borgemma4:latest) rather than community re-uploads, unless they’re from a trusted source like Unsloth - Keep Ollama updated. The MLX performance improvements and caching fixes only apply to v0.19 and later
TL;DR — Save This for Later
- Gemma 4 + Ollama is genuinely practical for everyday coding assistance
- E4B is the right pick for most 16GB laptops (~6GB at 4-bit)
- Set
OLLAMA_KEEP_ALIVE="-1"to prevent cold starts - It won’t replace frontier models for hard problems — route smart, not all-in
- If tool calling fails, disable reasoning mode and update Ollama first
- 100% local means your code never leaves your machine. That matters.
Got questions about running local models? Drop them in the comments — I test this stuff constantly and try to answer everything.
