Run LLMs locally

Run LLMs (Large language model) locally

List of LLMs software

Ollama

• Docker support: Available
• Supported OS: Win , Mac & Linux
• User Interface: CLI (Command-line interface)
• Support API endpoint: Available
• Supported embedding models:
• License: OSS, MIT

LM Studio - Discover, download, and run local LLMs

• Docker support: N/A
• Supported OS: Win , Mac & Linux
• User Interface: GUI
• Support API endpoint: Available
• Supported embedding models:
• License: Mixed; Apache License 2.0 for lmstudio.js. But Terms of Use mentioned "using the Services for your personal, non-commercial purposes."

janhq/jan: Jan is an open source alternative to ChatGPT that runs 100% offline on your computer. Multiple engine support (llama.cpp, TensorRT-LLM)

• Docker support: Available
• Supported OS: Win , Mac & Linux
• User Interface: CLI (Command-line interface)
• Support API endpoint: Available
• Supported embedding models:
• License: GAPL

GPT4All

• Docker support: N/A
• Supported OS: Win , Mac & Linux
• User Interface: GUI
• Support API endpoint: Available
• Supported embedding models: "SBert and Nomic Embed Text v1 & v1.5"
• License: OSS, MIT

oobabooga/text-generation-webUser Interface: A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.

• Docker support: Available
• Supported OS: Win , Mac & Linux
• User Interface: GUI
• Support API endpoint: Available
• Supported embedding models:
• License: OSS, GNU v.3

PrivateGPT

• Docker support: Available
• Supported OS: Win , Mac & Linux
• User Interface: GUI
• Support API endpoint: Available
• Supported embedding models:
• License: OSS, Apache v.2

AnythingLLM | The ultimate AI business intelligence tool

• Docker support: Available
• Supported OS: Win , Mac & Linux
• User Interface: GUI
• Support API endpoint:
• License: Mixed, OSS, MIT

LLM: A CLI utility and Python library for interacting with Large Language Models (llm · PyPI)

• Docker support:　The `LLM` package might be supported if installed in a Python Docker Container. However, it has not been actually tested.　
• Supported OS:
• User Interface: CLI
• Support API endpoint: N/A
• Supported embedding models:
• License:

MLC LLM

• Docker support: Not available
• Supported OS: Win , Mac & Linux
• User Interface: CLI
• Support API endpoint: Available
• Supported embedding models:
• License: OSS, Apache License Version 2.0

(left blank intentionally)

* Docker support: Available or not
* Supported OS: {{Win}}, {{Mac}} & {{Linux}}
* User Interface: GUI or CLI
* Support API endpoint: Available or not
* Supported embedding models:
* License: Mixed, OSS or copyright

Further reading

• 5 easy ways to run an LLM locally | InfoWorld
• Troyanovsky/Local-LLM-Comparison-Colab-UI: Compare the performance of different LLM that can be deployed locally on consumer hardware. Run yourself with Colab WebUI.