Why I Run Language Models on My Own Machine
Cloud AI is convenient, until you can't use it. Student data, unpublished research, industry IP, and institutional compliance all push toward running LLMs on your own hardware, and open models are now capable enough to do it.
I work across academia and industry: research projects, teaching, and applied AI development with partners who handle sensitive data and intellectual property. In every one of those contexts, there are things I cannot send to a cloud AI service, including student data with ethics restrictions, unpublished research, industry IP under NDA, and proprietary datasets. Some of that material has to stay on approved infrastructure.
Local language models solve this. I run them on my own hardware, so nothing leaves my machine, and the output is good enough for everyday writing and coding. I started running them because of privacy constraints. Now I also use them offline and when developing AI tools.
For those of us who work with LLMs and AI systems daily, running models locally is old news. But I’m constantly struck by how many colleagues and students don’t even know it’s possible. They assume you need a server farm or a cloud subscription. You don’t. Any modern Mac with Apple Silicon, most gaming PCs, even the new generation of mini PCs with shared CPU/GPU memory can run capable models.
The Privacy and Compliance Case
If you’re an academic, you deal with sensitive data constantly: student submissions, unpublished manuscripts, ethics applications, grant reviews, participant data. Cloud AI services process your inputs on remote servers. Their privacy policies have improved, and many now say they won’t train on your data. But “we promise not to” is different from “we can’t.” The data still leaves your machine and sits on someone else’s infrastructure.
I use cloud AI tools every day for work that doesn’t involve sensitive material. But when I’m working with student data or unpublished research, local models are the responsible choice. The data never leaves my computer, so I don’t have to rely on a provider’s privacy policy.
University ethics boards increasingly have explicit policies about what can and can’t be sent to cloud AI services. Ethics approvals for human participant research specify how data must be stored and processed, and consent agreements rarely anticipated cloud AI. Data-sharing agreements restrict where shared datasets can be processed. Government contracts, industry NDAs, and classified projects can require processing to stay on approved infrastructure. Running a model on approved local hardware keeps the data where these agreements require it to be.
Working on the Go
I travel a lot: long-haul flights, conference venues with overloaded wifi, hotels with patchy connections. If your AI workflow depends entirely on cloud services, you’re stuck the moment connectivity disappears. With local models, I keep working regardless. Some of my most productive sessions have been on 14-hour flights with a capable local model running on my laptop, no internet required.
The Industry and Applied Case
Part of my work involves developing AI-powered systems (agent-based and LLM-based tools) for industry partners and government stakeholders, on applied research projects spanning defence, training systems, and operational support. These projects share constraints that rule cloud AI out or make it hard to use.
- Industry partners need assurance that their IP and proprietary data won’t leave the network. When you’re working with someone’s competitive advantage, “we’ll use a cloud API” isn’t a conversation that goes well.
- Security is non-negotiable on some projects, which involve classified or restricted information that cannot leave approved infrastructure.
- Systems that have to keep running when the internet drops can’t depend on a cloud API.
- Some applications need edge deployment, running on-device in the field without an internet connection.
These requirements mean you design for local capability from the start, choosing models that run efficiently on constrained hardware and building pipelines that don’t assume cloud access.
The Development and Cost Case
Even without privacy constraints, cost is a practical reason to run models locally. If you’re building tools that embed LLMs (agents, RAG pipelines, custom workflows), you need a model you can hit thousands of times without worrying about API credits. A local model costs nothing per request.
I use local models for prototyping agent workflows, testing integrations before pointing at production APIs, and letting students experiment freely in the lab without usage limits. Once a pipeline works locally, I swap in a cloud model for production if I need the extra capability. But the development loop happens on my machine, at zero marginal cost.
What’s Good Enough Locally Now
A year ago, local models were a compromise, and you gave up output quality for the privacy and cost benefits. The open models released since then have changed that.
Google’s Gemma 3 models are a standout. The 4B model gives good results for something that runs on 8GB of RAM, and the 12B version handles everyday tasks well. OpenAI released GPT-OSS, its first open-weight language models since GPT-2, with a 20B variant that runs in about 16GB of memory. Qwen 3, Llama 3.2, Phi-4, and Mistral round out a field where there’s a strong option at every size point. These models do well at summarisation, drafting, code generation, data analysis, and question answering.
Ollama makes running them trivial: install it, pull a model, and you’re running in under five minutes. Just ollama pull gemma3 and go. Ollama now includes a built-in UI as well, so you don’t need the command line if you prefer not to use it. LM Studio is another excellent option. It’s a desktop app where you can download and chat with models. It’s a great starting point for students and colleagues who aren’t comfortable with the terminal. See our Local LLM Setup Guide for the full model comparison table and installation walkthrough for both.
For tasks that demand more capability, such as complex reasoning, long-form writing, or careful analysis, I still use cloud models. I run the everyday tasks above on local models. When the data can’t leave my machine or I’m offline, a local model is the only option that works.
Where to Start
If you want to try running models locally, I’ve written a step-by-step guide that covers installation, model selection, hardware requirements, and how to connect local models to your development tools.
Read the full guide: Local LLM Setup Guide
Start with Ollama or LM Studio and a small model. With a reasonably modern laptop and five minutes, you can have it running today. Then try it on a task you’d normally send to a cloud model.
The best AI setup isn’t all-cloud or all-local. It’s knowing when each one matters, and having both options available. Local models give you privacy and independence at zero marginal cost. Cloud models give you maximum capability. Use both, and you’re covered.
Michael Richardson Professor, School of Psychological Sciences Faculty of Medicine, Health and Human Sciences Macquarie University
AI Disclosure: I wrote this article with help from AI tools, including Claude. The ideas and the workflow described here are my own. The AI helped me draft and edit the text. I use AI throughout my research and teaching, and I encourage others to do the same. Using AI well is a skill to be learned openly.
The AI models used here, like all current LLMs, were trained on vast quantities of text written by others, largely without explicit consent. Every output they produce draws on the ideas and language of countless researchers and writers. Their labour makes these tools possible, and I want to acknowledge it even where no individual can be credited.