Building rig-code: Lessons from Local-First AI Agents and Async Orchestration
Over the past few years, my role has increasingly intersected with the operational realities of deploying AI systems. While at EUROPEAN PATENT OFFICE (EPO), I managed the delivery of AI tools designed to navigate vast, complex document landscapes. The paramount concern there was data sovereignty and rigorous governance. Every model invocation, every piece of data processed, had to fit within strict legal and architectural boundaries. Similarly, during my time at ORANGE POLAND, leading a technical project team for a major telecommunications client, we grappled with the latency and cost implications of cloud-dependent AI services. These experiences planted a seed that eventually grew into rig-code: a local-first autonomous AI coding agent built on Rust, Tokio, and Ollama.
The core idea is simple but powerful: what if you could have an AI pair programmer that doesn't send your proprietary codebase to a remote API? rig-code is a tool I built to answer that question. It leverages the async runtime of Tokio to orchestrate interactions with locally-running LLMs via Ollama, creating a responsive, private, and customizable coding assistant. The project became a sandbox for applying principles I've honed in governance and delivery to a deeply technical domain.
Why Local-First? Lessons from the Field
The decision to go local-first wasn't just about privacy (though that's a huge driver). At GLS IT SERVICES, while managing the AI systems integration, we saw how unpredictable network latency could derail real-time developer tools. An agent that needs to wait for a cloud API response every few keystrokes feels sluggish. By running models locally via Ollama, rig-code achieves sub-second response times for many tasks. The trade-off is model capability; a local Llama 3 or Mistral model won't match GPT-4's raw power on every benchmark. However, for targeted coding tasks—explanations, refactoring suggestions, boilerplate generation—a well-prompted smaller model is often sufficient and dramatically more usable.
Rust and Tokio: The Architecture for Orchestration
Why Rust? My background in C++ (often used in high-performance engine projects) gave me an appreciation for memory safety and performance. For a tool that needs to manage multiple async processes—watching files, parsing code, interacting with the LLM, and presenting a CLI interface—Rust's ownership model prevents entire classes of bugs. Tokio is the backbone. It allows rig-code to seamlessly manage concurrent tasks. For instance, while the agent is synthesizing a response from the LLM, it can simultaneously index a new file that was just created or updated in your project directory, all without blocking.
This async architecture was directly inspired by challenges faced in earlier projects. At KNAPPSCHAFT, delivering AI projects on time meant meticulously managing dependent services. In rig-code, the "services" are internal modules. The Tokio runtime acts like a robust internal project manager, scheduling work (like LLM inference, file I/O, and user input handling) efficiently across threads. It's a microcosm of the orchestration skills needed to manage cross-functional teams, but applied to software components.
Concrete Trade-offs and Iterative Delivery
A key lesson from my time as an AI Project Delivery Manager is that you must build in increments. rig-code started as a simple wrapper: read a file, send its content to Ollama with a prompt, print the response. From there, we (the "team" being myself and the codebase) iterated. We added conversation history to maintain context, project-wide indexing for better suggestions, and configurable model parameters.
A critical trade-off we encountered was state management. Maintaining a persistent, in-memory representation of the project structure for fast querying consumed significant RAM, especially for large codebases. The alternative—relying on the file system for every query—was slower but more resource-efficient. The solution was a hybrid approach: a lightweight cache for frequent operations, with fallbacks to disk, mirroring the performance-cost balancing acts I've navigated in cloud deployments on AWS and Azure.
The Outcome and Future Roadmap
rig-code today is a functional tool that demonstrates a viable architecture for private AI-assisted development. It uses Cargo for its build system (which, by the way, is a fantastic model of software engineering itself), and its core is structured around the tokio::spawn and tower::Service abstractions for clean concurrency and service composition.
More than the code, the project reinforced a professional belief: the best tools often come from applying cross-domain expertise. The governance mindset from EPO informs how I think about data flow. The delivery discipline from KNAPPSCHAFT shapes how I prioritize features. And the technical problem-solving from ORANGE POLAND guides the architectural trade-offs.
If you're exploring local AI tooling, my advice is to start small, define your core constraints (privacy, latency, cost), and be ruthless in your prioritization. The technology is there—Rust, Tokio, and projects like Ollama are mature and powerful. The real challenge, as always, is thoughtful engineering and clear-eyed management of trade-offs.