Ollama provides a way to run AI models on your PC with around 16 GB ram. It is slow but it works. Need to try connecting with eGPU as my mac supports Thunderbolt 4.
Ollama AI
https://ollama.com/
ollama run llama2
Uncensored models
ollama run llama2-uncensored
ollama run nous-hermes-llama2
ollama run wizard-vicuna
ollama run codellama. //for java, python, c++ etc
ollama list
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway --name ollama-webui --restart always ghcr.io/ollama-webui/ollama-webui:main
ollama webui
- cd /Users/[]/AI/ollama-webui/backend
2. sh start.sh
Open chrome with debug port
cd /Applications/Google Chrome.app/Contents/MacOS
./Google\ Chrome --remote-debugging-port=9222
TextGen
Manual installation text generation webui using Conda
https://github.com/oobabooga/text-generation-webui
Recommended if you have some experience with the command-line.
0. Install Conda
https://docs.conda.io/en/latest/miniconda.html
On Linux or WSL, it can be automatically installed with these two commands (source):
curl -sL "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" > "Miniconda3.sh"
bash Miniconda3.sh
1. Create a new conda environment
conda create -n textgen python=3.11
conda activate textgen
2. Install Pytorch
System | GPU | Command |
Linux/WSL | NVIDIA | pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121 |
Linux/WSL | CPU only | pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu |
Linux | AMD | pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.6 |
MacOS + MPS | Any | pip3 install torch torchvision torchaudio |
Windows | NVIDIA | pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121 |
Windows | CPU only | pip3 install torch torchvision torchaudio |
The up-to-date commands can be found here: https://pytorch.org/get-started/locally/.
For NVIDIA, you may also need to manually install the CUDA runtime libraries:
conda install -y -c "nvidia/label/cuda-12.1.0" cuda-runtime
3. Install the web UI
git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
pip install -r <requirements file according to table below>
Requirements file to use:
GPU | CPU | requirements file to use |
Apple | Intel | requirements_apple_intel.txt |
conda activate textgen
cd text-generation-webui
python server.py
