Running CPU-Optimized Large Language Model Inference in a Container
2026-11-17
Introduction
Running large language models locally has become increasingly practical with CPU-optimized inference engines.
The llama-modchip project enables running open-weight Mixture-of-Experts models like DeepSeek-V4-Flash (284B parameters) on systems with 64GB of RAM without requiring a discrete GPU.
Combined with OpenCode, an open-source AI coding agent, developers can create a fully local, privacy-preserving development environment.
Containerizing this setup with Podman provides isolation, reproducibility, and the ability to run without root privileges.
Container Setup
I wrote a simple Dockerfile that packages both llama-modchip and OpenCode together in a container.
The container runs as a non-root user, matching your host UID and GID to avoid permission issues with mounted volumes. Bind-mount your workspace directory to persist code and model weights outside the container.
Running the Stack
Inside the container, install a model using the interactive setup script. The script recommends suitable models based on available system RAM.
Access a chat UI in a browser on your Podman host. The llama-modchip gateway and OpenCode each offer a chat server.
Conclusion
This containerized approach provides a reproducible, rootless environment for local LLM inference and AI-assisted coding. The combination of llama-modchip's CPU optimizations and OpenCode's agent capabilities enables fully offline development workflows.
For more details on llama-modchip's quantization techniques and benchmarks, see the project repository and the Hacker News discussion.