Back to blog

Local LLMs and Self-Hosted AI Infrastructure: A Practical Guide

By seokchol hong

Introduction

Cloud AI services are convenient, but they come with limits around cost, data privacy, and dependence on internet connectivity. Local LLMs and self-hosted AI infrastructure offer an alternative. With tools such as Ollama and Supabase, teams can run models on their own servers and use AI without sending data to external providers.

1. Ollama: The Standard Way to Run Local LLMs

Ollama makes it easy to run a wide range of open source LLMs locally. It provides a simple interface for pulling and running models, much like Docker does for containers.

Installation and Basic Use

# macOS/Linux
curl -fsSL https://ollama.com/install.sh | sh

# Download and run a model
ollama pull llama3
ollama run llama3

Major Supported Models

  • Llama 3 by Meta: one of the most popular open source LLMs
  • Qwen 3 by Alibaba: strong multilingual support
  • Mistral: efficient lightweight models
  • GLM-4 by Zhipu AI: optimized for Chinese and English bilingual use
  • Phi by Microsoft: small but high-performing models

Combined with Open Code

If you connect Ollama to Open Code (oh-my-opencode), you can build a fully free AI coding environment. There is no API cost at all, which makes it ideal for learning and experimentation.

Performance and Hardware Requirements

  • 7B models: around 8 GB RAM, can run on a general GPU
  • 13B models: around 16 GB RAM, mid-tier GPU recommended
  • 70B models: 64 GB+ RAM and a high-end GPU such as an RTX 4090 or better
  • Apple Silicon Macs can run local models efficiently through Metal acceleration

2. Supabase: Backend Infrastructure for AI

Supabase is an open source alternative to Firebase built on PostgreSQL. It is especially useful for AI app development.

AI Integration

  • pgvector: PostgreSQL extension for vector search, suitable as the vector database for RAG systems
  • Edge Functions: serverless functions for AI pipelines
  • Realtime: real-time data synchronization
  • Row Level Security: per-user access control

Supabase MCP

With Supabase MCP connected to Claude Code, PostgreSQL can be managed through natural language. Commands like "Create a users table and add a unique constraint on email" become possible. Through 32 tools, it can handle schema design, table creation, query execution, and RLS configuration.

3. MiroFish: A Collective Intelligence Simulation Engine

MiroFish is an unusual AI tool that simulates thousands of virtual humans from a single document. It can be used for market research, public opinion simulation, and prediction of product reactions.

MCP vs. CLI

MiroFish can also be used in two ways:

  • MCP: run simulations in natural language from Claude Desktop or Claude Code
  • CLI: control it directly from the terminal with commands

4. Pros and Cons of Self-Hosting

Advantages

  • Data sovereignty: corporate data does not have to be sent to external servers
  • Predictable cost: fixed hardware cost instead of variable API billing
  • Offline use: AI functionality remains available without internet access
  • Customization: freedom to fine-tune models and optimize prompts

Disadvantages

  • Upfront investment: high-performance hardware such as GPUs is expensive
  • Operational burden: updates, server maintenance, and security are on you
  • Performance limits: local models may underperform compared with top commercial models such as GPT-5 and Claude 4.5
  • Required expertise: installation, configuration, and optimization need technical skill

Recommended Scenarios

  • Enterprises where data privacy is critical, such as healthcare, finance, and legal
  • High-volume workloads where API cost is a concern
  • Domain-specific systems that require fine-tuning
  • Learning and experimentation

Closing

Local LLMs and self-hosted AI infrastructure are both an alternative to and a complement for cloud AI. A practical approach is to start with Ollama, experiment with open source models, build vector-backed RAG systems with Supabase, and then adopt a hybrid strategy when needed. You do not need to do everything locally, but having a local option is a meaningful strategic asset for both cost control and data security.

Back to blog