AI notes

Ollama provides a way to run AI models on your PC with around 16 GB ram. It is slow but it works. Need to try connecting with eGPU as my mac supports Thunderbolt 4. 

 

Ollama AI

https://ollama.com/

ollama run llama2

Uncensored models

ollama run llama2-uncensored

ollama run nous-hermes-llama2

ollama run wizard-vicuna

ollama run codellama. //for java, python, c++ etc

ollama list

http://127.0.0.1:11434/

https://ollamahub.com/  

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway --name ollama-webui --restart always ghcr.io/ollama-webui/ollama-webui:main

ollama webui 

1. cd /Users/[]/AI/ollama-webui/backend

 2. sh start.sh

1. http://localhost:8080/

Open chrome with debug port

cd /Applications/Google Chrome.app/Contents/MacOS

./Google\ Chrome --remote-debugging-port=9222

TextGen

Manual installation text generation webui using Conda

https://github.com/oobabooga/text-generation-webui

Recommended if you have some experience with the command-line.

0. Install Conda

https://docs.conda.io/en/latest/miniconda.html

On Linux or WSL, it can be automatically installed with these two commands (source):

curl -sL "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" > "Miniconda3.sh"

bash Miniconda3.sh

1. Create a new conda environment

conda create -n textgen python=3.11

conda activate textgen

2. Install Pytorch

System	GPU	Command
Linux/WSL	NVIDIA	pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
Linux/WSL	CPU only	pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
Linux	AMD	pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.6
MacOS + MPS	Any	pip3 install torch torchvision torchaudio
Windows	NVIDIA	pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
Windows	CPU only	pip3 install torch torchvision torchaudio

The up-to-date commands can be found here: https://pytorch.org/get-started/locally/.

For NVIDIA, you may also need to manually install the CUDA runtime libraries:

conda install -y -c "nvidia/label/cuda-12.1.0" cuda-runtime

3. Install the web UI

git clone https://github.com/oobabooga/text-generation-webui

cd text-generation-webui

pip install -r <requirements file according to table below>

Requirements file to use:

GPU	CPU	requirements file to use
Apple	Intel	requirements_apple_intel.txt

conda activate textgen

cd text-generation-webui

python server.py