Running CPU-Optimized Large Language Model Inference in a Container

2026-11-17

Introduction

Running large language models locally has become increasingly practical with CPU-optimized inference engines.

The llama-modchip project enables running open-weight Mixture-of-Experts models like DeepSeek-V4-Flash (284B parameters) on systems with 64GB of RAM without requiring a discrete GPU.

Combined with OpenCode, an open-source AI coding agent, developers can create a fully local, privacy-preserving development environment.

Containerizing this setup with Podman provides isolation, reproducibility, and the ability to run without root privileges.

Container Setup

I wrote a simple Dockerfile that packages both llama-modchip and OpenCode together in a container.

The container runs as a non-root user, matching your host UID and GID to avoid permission issues with mounted volumes. Bind-mount your workspace directory to persist code and model weights outside the container.

Running the Stack

Inside the container, install a model using the interactive setup script. The script recommends suitable models based on available system RAM.

Access a chat UI in a browser on your Podman host. The llama-modchip gateway and OpenCode each offer a chat server.

Conclusion

This containerized approach provides a reproducible, rootless environment for local LLM inference and AI-assisted coding. The combination of llama-modchip's CPU optimizations and OpenCode's agent capabilities enables fully offline development workflows.

For more details on llama-modchip's quantization techniques and benchmarks, see the project repository and the Hacker News discussion.