Editing Run LLMs locally

Run LLMs (Large language model) locally

== List of LLMs software ==
[https://ollama.com/ Ollama]<ref>[https://blog.miniasp.com/post/2024/03/04/Useful-tool-Ollama 介紹好用工具：Ollama 快速在本地啟動並執行大型語言模型 | The Will Will Web]</ref>
* Docker support: Available<ref>[https://ollama.com/blog/ollama-is-now-available-as-an-official-docker-image Ollama is now available as an official Docker image · Ollama Blog]</ref> {{Gd}}
* Supported OS: {{Win}}, {{Mac}} & {{Linux}}
* User Interface: GUI
* Support API endpoint: Available<ref>[https://github.com/ollama/ollama?tab=readme-ov-file#rest-api ollama/ollama: Get up and running with Llama 3, Mistral, Gemma, and other large language models.]</ref>
* License: OSS, MIT<ref>[https://github.com/ollama/ollama/blob/main/LICENSE ollama/LICENSE at main · ollama/ollama]</ref>

[https://lmstudio.ai/ LM Studio - Discover, download, and run local LLMs]
* Docker support: N/A {{exclaim}}
* Supported OS: {{Win}}, {{Mac}} & {{Linux}}
* User Interface: GUI
* Support API endpoint: Available<ref>[https://lmstudio.ai/docs/local-server Local LLM Server | LM Studio]</ref>
* License: Mixed; Apache License 2.0 for lmstudio.js<ref>[https://github.com/lmstudio-ai/lmstudio.js/blob/main/LICENSE lmstudio.js/LICENSE at main · lmstudio-ai/lmstudio.js]</ref>. But Terms of Use<ref>https://lmstudio.ai/terms</ref> mentioned "using the Services for your personal, non-commercial purposes."

[https://github.com/janhq/jan janhq/jan: Jan is an open source alternative to ChatGPT that runs 100% offline on your computer. Multiple engine support (llama.cpp, TensorRT-LLM)]
* Docker support: Available
* Supported OS: {{Win}}, {{Mac}} & {{Linux}}
* User Interface: CLI (Command-line interface)
* Support API endpoint: Available
* License: GAPL<ref>[https://github.com/janhq/jan/blob/dev/LICENSE jan/LICENSE at dev · janhq/jan]</ref>

[https://gpt4all.io/index.html GPT4All]
* Docker support: N/A
* Supported OS: {{Win}}, {{Mac}} & {{Linux}}
* User Interface: GUI
* Support API endpoint: Available<ref>[https://docs.gpt4all.io/gpt4all_python.html Generation - GPT4All Documentation]</ref>
* License: OSS, MIT<ref>[https://github.com/nomic-ai/gpt4all/blob/main/LICENSE.txt gpt4all/LICENSE.txt at main · nomic-ai/gpt4all]</ref>

[https://github.com/oobabooga/text-generation-webui oobabooga/text-generation-webUser Interface: A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.]
* Docker support: Available
* Supported OS: {{Win}}, {{Mac}} & {{Linux}}
* User Interface: GUI
* Support API endpoint: Available<ref>[https://github.com/oobabooga/text-generation-webui/wiki/12-%E2%80%90-OpenAI-API#examples 12 ‐ OpenAI API · oobabooga/text-generation-webui Wiki]</ref>
* License: OSS, GNU v.3<ref>[https://github.com/oobabooga/text-generation-webui/blob/main/LICENSE text-generation-webui/LICENSE at main · oobabooga/text-generation-webui]</ref>

(left blank intentionally)
<pre>
* Docker support: Available or not
* Supported OS: {{Win}}, {{Mac}} & {{Linux}}
* User Interface: GUI or CLI
* Support API endpoint: Available or not
* License: Mixed, OSS or copyright
</pre>

== Further reading ==
* [https://www.infoworld.com/article/3705035/5-easy-ways-to-run-an-llm-locally.html 5 easy ways to run an LLM locally | InfoWorld]
* [https://github.com/Troyanovsky/Local-LLM-Comparison-Colab-UI Troyanovsky/Local-LLM-Comparison-Colab-UI: Compare the performance of different LLM that can be deployed locally on consumer hardware. Run yourself with Colab WebUI.]

== References ==
<references />

[[Category: Tool]]
[[Category: Artificial intelligence]]
[[Category: Generative AI]]