{
  "schemaVersion": "1.0",
  "entity": "BlogPosting",
  "title": "Edge AI: Running Small Language Models on Raspberry Pi and Phones",
  "description": "A hands-on guide to running small language models like Gemma, Phi, and Qwen on a Raspberry Pi or phone with Ollama, GGUF quantization, and no cloud API.",
  "author": "vd",
  "datePublished": "2026-08-09T00:00:00.000Z",
  "dateModified": "2026-08-09T00:00:00.000Z",
  "tags": [
    "AI",
    "EdgeComputing",
    "Raspberry Pi",
    "Ollama",
    "Local AI"
  ],
  "aeoDirectAnswers": [
    {
      "question": "Why run a language model on a Raspberry Pi instead of the cloud?",
      "answer": "Cloud AI APIs are fast, capable, and cheap per request. So why bother with a $80 computer that struggles to keep up? Three reasons keep coming up when I talk to people building this way: privacy, latency behavior, and cost at scale. **Privacy is the one that actually matters for most hobbyists and small teams.** If your device is reading private messages, medical notes, or anything from a home camera feed, sending that text to a third-party API means trusting their retention policy, their breach history, and their business model. Running inference locally means the data never leaves the device. That's not a marketing claim, it's a property of the architecture: there's no network call to intercept, log, or subpoena. **Latency behaves differently, not necessarily better.** A cloud API round-trip depends on your internet connection, the provider's current load, and network jitter. Local inference has none of that variability, but it's also usually slower in raw throughput on Pi-class hardware compared to a cloud GPU. What you gain is predictability: the same prompt takes roughly the same time whether your Wi-Fi is good or terrible, because there's no Wi-Fi involved in the inference step at all."
    },
    {
      "question": "What is a small language model, and how is it different from the big ones?",
      "answer": "A small language model is, unsurprisingly, defined mostly by parameter count. There's no official cutoff, but in practice the models people call \"SLMs\" in 2026 sit somewhere between 270 million and roughly 8 billion parameters. Compare that to frontier cloud models, which run from tens of billions into the trillions. Fewer parameters means less capability per query, but the relationship isn't linear. A 3B model in 2026 handles tasks that would have needed a 30B model three years ago, because training techniques (better data curation, distillation from larger teacher models, and architecture tweaks) have improved faster than raw parameter counts have grown. SLMs aren't \"the cloud model but worse.\" They're a different design target: models trained specifically to be useful within a tight memory and compute budget, not scaled-down versions of something bigger. That distinction matters when you pick one. A 3B model that's been trained and tuned for instruction-following and summarization will usually beat a naively-shrunk 7B model at the tasks it was designed for, even though it's smaller on paper."
    },
    {
      "question": "Which SLM families are worth running on edge hardware right now?",
      "answer": "Here's where things stand across the major families as of this writing, based on official model cards and Ollama's library listing: | Family | Edge-relevant sizes | License | Notes | |---|---|---|---|"
    },
    {
      "question": "How does quantization make these models fit on a Pi or phone?",
      "answer": "Here's the part that actually makes edge AI possible: quantization. A language model's weights are, in their original trained form, usually stored as 16-bit floating point numbers (FP16 or BF16). A 3-billion-parameter model at FP16 needs roughly 6GB of RAM just to hold the weights, before you account for the memory needed during inference itself. That doesn't leave much room on an 8GB Pi. Quantization reduces the precision of those weights, storing each one in fewer bits. The GGUF format used by llama.cpp (and, under the hood, by Ollama) supports several quantization levels, and the naming convention tells you roughly what to expect: **Q8_0**: 8-bit weights. Close to the original quality, with a measured perplexity increase of around 0.01 compared to full precision in most reported testing. Costs you almost double the size of Q4 for that fidelity."
    },
    {
      "question": "How do I set up Ollama on a Raspberry Pi?",
      "answer": "Ollama is the simplest on-ramp for this, mostly because the install process is identical to a desktop Linux machine. You're not compiling anything by hand unless you want to."
    },
    {
      "question": "How do I call a local model from my own application code?",
      "answer": "Interactive chat is useful for testing, but the reason to put a model on a Pi is usually so that some other program can talk to it. Ollama serves an HTTP API at http://localhost:11434/api by default, and it's the same interface whether you're on a Pi, a laptop, or a server. The two endpoints you'll actually use are /api/generate for one-shot completions and /api/chat for multi-turn conversations. Both take a model field, an optional stream boolean (streaming is on by default), an options object for runtime generation settings, and a keep_alive duration controlling how long the model stays resident in memory afterwards. Setting \"stream\": false is the first change most people want on a Pi, because a script that only needs the finished text is easier to write against one JSON object than against a sequence of partial ones."
    },
    {
      "question": "How do I run a small language model on a phone?",
      "answer": "Phones are a different environment. There's no systemd, no apt package manager, and app store policies gate what you can install. Two paths dominate right now."
    },
    {
      "question": "How do I choose the right model for my device class?",
      "answer": "I get asked this more than anything else, so here's the framework I actually use when someone asks \"will X model run on Y device.\" !Device class decision sketch showing Pi 4, Pi 5, mid-range phone, and flagship phone with matching model sizes *Image Prompt: A premium hand-drawn sketch note style illustration on a cozy cream paper background. Four small hand-drawn devices in a row: a small single-board computer labeled with a \"4GB\" sticky note doodle, a slightly bigger board labeled \"8GB\", a phone doodle labeled \"mid-range\", and a larger sleeker phone doodle labeled \"flagship\", each connected by a hand-drawn arrow to a small stacked-block icon sized proportionally to represent model size (smallest block under the first device, largest block under the last), warm graphite lines, playful pencil-sketch shading, soft pastel yellow and blue highlight accents, faint notebook grid lines in the background. No text, no watermark. Square 1:1.*"
    },
    {
      "question": "What should I check first when inference stalls or slows down?",
      "answer": "When a setup that used to work starts hanging, crashing, or crawling, the cause on Pi-class hardware is almost always one of three things: context length, temperature, or a model that keeps getting unloaded between calls. Check them in that order before you go looking for a bug. **Context length quietly competes with the weights.** Ollama uses a context window of 4096 tokens by default, changeable with the OLLAMA_CONTEXT_LENGTH environment variable on the server or the num_ctx option on an individual request. Raising num_ctx toward a model's advertised maximum is the fastest way I know to turn a model that loaded comfortably into one that pushes the board into swap, because the cache for that context comes out of the same RAM pool as the weights. If something worked yesterday and doesn't today, ask what changed about your prompt length before you blame the model. **Sustained generation is a thermal workload, not a bursty one.** Raspberry Pi boards progressively throttle the Arm cores once the SoC passes 80C, and at 85C both the Arm cores and the GPU get throttled back. A typical desktop workload spikes and recovers; token generation holds the cores busy for as long as the response takes, which is exactly the profile that reaches those limits. vcgencmd measure_temp reports the current SoC temperature, and vcgencmd get_throttled returns a bit pattern where 0x4 means throttling is happening right now and 0x40000 means it has happened since boot. Bit 0x1 is worth reading too: it flags undervoltage, which on a Pi 5 usually points at a marginal USB-C supply rather than anything wrong with the model. Raspberry Pi recommends a 5V 5A supply for the Pi 5, though a 3A one is fine if you aren't hanging power-hungry peripherals off the USB ports."
    },
    {
      "question": "What can't small models on edge hardware actually do well?",
      "answer": "I'd be doing you a disservice if I only listed the wins. There are real limits, and pretending otherwise sets people up to build something that quietly disappoints them in production. **Context length gets expensive fast.** Even models advertising 32K or 128K token context windows need proportionally more RAM to actually use that window during inference on a Pi. In practice, I keep working context on Pi-class hardware well under the model's advertised maximum, because the RAM cost of a long context competes directly with the model weights for the same limited pool. **Multi-step reasoning degrades before raw recall does.** A 3B model can usually tell you a fact it was trained on. Ask it to chain four logical steps together and you'll see it lose the thread more often than a larger cloud model would. If your use case genuinely needs multi-step reasoning, an SLM is often the wrong tool regardless of how well you quantize it."
    },
    {
      "question": "Can a Raspberry Pi actually run a useful language model, or is it just a demo?",
      "answer": "It depends on the task. For classification, short summarization, and single-turn question answering, a Pi 5 with an 8GB model like Gemma 3's 4B variant at Q4_K_M is genuinely useful, not just a proof of concept. For extended multi-turn conversation or complex reasoning, the limits show up quickly and you'll want to manage your expectations accordingly."
    },
    {
      "question": "What's the difference between GGUF and the original model files from Hugging Face?",
      "answer": "The original files from a model's Hugging Face repository are usually in safetensors format at full precision (FP16/BF16), meant for GPU training and inference frameworks like PyTorch. GGUF is a format specifically designed for llama.cpp-style CPU and edge inference, and it's typically where quantization happens, converting those full-precision weights into the 4-bit, 5-bit, or 8-bit formats discussed above."
    },
    {
      "question": "Do I need internet access after setting up a model on my Pi or phone?",
      "answer": "No, once the model file is downloaded, inference runs entirely on the device with no network calls required. This is the core privacy argument for edge AI. You'll only need connectivity again if you want to pull a different model or update Ollama itself."
    },
    {
      "question": "Why does my model run fine on a Pi 5 but crash or hang on a Pi 4?",
      "answer": "Almost always RAM. The Pi 4's top configuration (8GB on some boards, though many are 2GB or 4GB) leaves much less headroom once you subtract what the OS and any other running services need. If a model that runs on a Pi 5 hangs on a Pi 4, check free -h for swap usage before assuming it's a software bug; a smaller model or more aggressive quantization is usually the actual fix."
    },
    {
      "question": "Is it better to build my own llama.cpp binary instead of using Ollama?",
      "answer": "For most people, no. Ollama wraps llama.cpp with model management, an API server, and a simpler CLI, and the performance difference on Pi-class hardware is rarely worth the extra setup complexity. Building llama.cpp directly makes more sense if you need a specific compile-time optimization flag for your exact CPU, or you're deploying to a device Ollama doesn't officially support yet. ---"
    },
    {
      "question": "What to Read Next",
      "answer": "How to Install Gemma 4 Locally with Ollama: the same Ollama workflow on desktop hardware, a good place to build familiarity before moving to a Pi How to Run Google's Gemma 4 Locally on Your Phone: the full Google AI Edge Gallery walkthrough referenced above Gemma 4 on Edge Devices: Android, Raspberry Pi, and IoT Applications: a broader look at multimodal edge deployment beyond text-only chat"
    }
  ],
  "semanticFactualBody": "A Raspberry Pi 5 with 8GB of RAM can run a 3-billion-parameter language model at a few tokens per second, entirely offline, for the price of a nice dinner. That sentence would have sounded like a joke three years ago. It isn't anymore. This guide covers which small language models (SLMs) actually run on a Pi or a phone today, how quantization trades accuracy for speed, and the exact commands to get a model answering questions on hardware that fits in your palm. I run three of these setups in my own apartment right now: a Pi 5 doing intent classification for a home automation script, a Pi 4 (don't laugh, it still works for tiny models) running a note-summarizer, and a two-year-old Android phone running a fully offline chat assistant for flights where I don't trust airport Wi-Fi. None of them talk to the internet after the model download finishes. --- Prerequisites Before you start, you need: A Raspberry Pi 5 with at least 8GB RAM (4GB works for the smallest models, but you'll be tight on headroom), running 64-bit Raspberry Pi OS A microSD card or, better, an NVMe/USB SSD boot drive (model files are multiple gigabytes, and slow storage makes load times painful) SSH or direct terminal access to the Pi For the phone side: an Android device with at least 6GB RAM, or an iPhone with A16/A17-class silicon or newer, plus Wi-Fi for the one-time model download Basic comfort with a Linux terminal. Nothing exotic, but you will type commands If you haven't set up Ollama on a normal desktop"
}