Introduction

Building an AI agent that lives entirely on your own hardware is appealing, but it quickly runs into a wall: the largest model that fits on a laptop may not be powerful enough for complex planning. This post describes how one developer bridged that gap by connecting a local autonomous agent to a decentralized GPU network, without surrendering control over file access, tool execution, or final approvals.

The project, called zkzkAgent, started as a local Linux assistant powered by Ollama. It entered the Decentralize AI Hackathon hosted by HackerNoon and Nosana, and this update documents the second phase: giving the agent the option to offload heavy reasoning to a much larger model running on distributed compute.

What Happened

zkzkAgent is built with LangGraph and uses separate nodes to classify requests, answer questions, generate plans, and execute tools. It includes about two dozen tools for working with files, applications, networks, processes, packages, and deployments. Destructive actions such as deleting files or installing packages are blocked until the user explicitly types "yes."

By default, the agent runs a 2-billion-parameter model locally. That model is fast and fits in memory, but it cannot handle the same workload as a 27-billion-parameter model. The developer faced a common choice: stay with a small local model or send requests to a centralized AI provider. Instead, he integrated Nosana, an open-source GPU cloud, as a third path.

The integration followed one strict rule: only model inference calls travel to the decentralized network. The graph logic, routing, confirmation gates, and all tool execution remain on the local machine. A remote model can suggest removing a file, but only the local computer can act on that suggestion, and only after manual approval.

Configuration is handled through environment variables, including an API key, base URL, and model identifier. Because Nosana exposes an OpenAI-compatible chat endpoint, the agent connects through LangChain's ChatOpenAI client. The rest of the agent interacts with a generic language-model object, so it does not need to know whether the backend is local Ollama or a remote GPU.

To validate the setup, the developer asked the agent to build a calculator web application with a FastAPI backend and an HTML, CSS, and JavaScript frontend. Using the 27-billion-parameter model on Nosana, the agent produced a nine-step plan, received approval, and then executed ten tool calls in sequence. It created the project folder, wrote the source files, and opened the result in VS Code. A follow-up request renamed the application across the title, heading, backend, and documentation without touching the calculator logic.

The finished app ran locally with uvicorn and responded to API requests successfully. Across the entire test, eighteen model calls went to the decentralized inference endpoint and returned successful responses, while every file write and command execution stayed on the developer's machine. The generated calculator code avoided unsafe evaluation, parsing math expressions through Python's ast module and allowing only a limited set of operators.

Why This Matters

Most discussions about local AI present a false binary: either run a small model on your hardware or upload everything to a centralized provider. This project demonstrates a middle ground. Sensitive or private work can remain on a local model, while complex planning can borrow power from a distributed GPU network.

The approach also preserves accountability. Because tools execute locally and destructive operations require human confirmation, a remote model never gains direct access to the user's system. Token usage is streamed with the final response chunk, which makes billing transparent and based on actual consumption.

For developers working on modest hardware, this pattern opens the door to larger reasoning models without abandoning a local-first philosophy.

Key Takeaways

  • A local AI agent can offload model inference to a decentralized GPU network while keeping tool execution and file access on-premises.
  • The integration used an OpenAI-compatible API and LangChain, requiring minimal changes to the existing agent graph.
  • Human-in-the-loop safeguards remained intact: destructive tools still require explicit approval before running.
  • A 27-billion-parameter model successfully planned and built a working calculator web app, while all code was written and executed locally.
  • Eighteen inference calls returned successful responses, demonstrating reliable connectivity to the decentralized backend.

Conclusion

The experiment shows that local agents do not have to stay small forever. By separating the model layer from the execution layer, developers can choose where each part of a request runs based on privacy, cost, and capability.

The author plans to publish benchmarks comparing the local 2-billion-parameter model against the larger remote model, explore hybrid routing so that simple conversations stay local while planning uses the GPU network, and add automatic fallback if the decentralized endpoint becomes unreachable. He also intends to tighten safety around shell commands and let the agent fix bugs in its own generated code.

The agent and the calculator it built are available on GitHub, along with the full run log from the Nosana test.