TDM 19000: Setting Up Ollama Models

Setting up the $SCRATCH Directory

To begin working with Ollama, we first need to make a symbolic link to where the Ollama files should be stored on Anvil.

By default, Ollama will write data to the ~/.ollama location in your home directory (the tilde symbol at the upper-left-hand-side of your keyboard refers to your home directory). BUT the data should actually be stored in you $SCRATCH/.ollama directory.

So we need to create a symbolic link that tells Ollama to connect to and store in the scratch space.

In a new Terminal window on Anvil, run these lines (one by one) to create an Ollama symbolic link - this only needs to be done once - to set up your environment:

# create a new .ollama in $SCRATCH
mkdir  -p  $SCRATCH/.ollama

# create a symbolic link to your personal directory
ln  -s  $SCRATCH/.ollama  ~

# If you make a mistake during this process, you can remove the symbolic link to the .ollama directory and try again; the `rm` stands for `remove` and the `-rf` is a recursive force, i.e., throwing everything in that directory away.  Please use this with caution.  In particular, be use to NOT put a space between the tilde and the directory location.
rm  -rf  ~/.ollama

Without the symbolic link, Ollama would download large models into your personal home directory. You would very quickly hit the storage limit, and the downloads would likely fail.

With the symbolic link, large Ollama models live in scratch storage - which is meant to contain much more storage, and often temporary files. (For comparison: Your home directory can contain 25 GB of information, and your scratch directory can contain 100 TB of data. In other words, your scratch directory holds 4000 times as much storage as your home directory.)

Very helpful information about symbolic links and Ollama in general on Anvil can be found in the example book page on LLMs.

Downloading an Ollama Model (we will start with llama3.2)

We are going to be working with LLMs, using Ollama as the framework that will us to run, manage, and deploy these models. There are some pre-existing models for Ollama that can be found on the official Ollama site. For this project, we will select Meta’s llama3.2. This collection is optimized for speed and local usage.

In a Terminal window (leaving the one running the server alone!) type:

/anvil/projects/tdm/bin/ollama pull llama3.2

This says to download the llama3.2 LLM to your ~/.ollama directory - but it is actually downloading to the $SCRATCH/.ollama location because of the symbolic link we created in the previous question!

To view information about anything you have downloaded (and to confirm that the download worked in the first place), you can use:

%%bash
/anvil/projects/tdm/bin/ollama list

…​in your notebook to display your current downloads.

Choosing the Number of Cores

If we were to try to use llama3.2 directly, bad things would happen. The models that Ollama uses expect to use all CPU cores on the server, but we only have access to a certain number of CPU cores on a node. Ollama does not know this by default, so we have to tell it to use a smaller number of CPU cores (also referred to as 'threads') so the models do not take hours to run.

To simplify and shorten the process, create a small new model based on the llama3.2 model - but using only 2 CPU threads.

In your notebook:

Create a file called mymodel. You create this file from just a few important lines:

  • cat > file: write text into a file

  • ~/mymodel: the file name in the home directory

  • << HERE and HERE: starts and ends the heredoc - this lets you write multiple lines into the file until the word HERE appears again

Contents of the file

  • FROM [model]: use the base model Llama3.2

  • PARAMETER num_thread [thread_count]: run the model using 2 CPU threads

Since we are creating a file that will define a model that uses 2 threads, we can use them all as following:

%%bash
cat > ~/mymodel << HERE
FROM llama3.2
# using 2 threads!
PARAMETER num_thread 2
HERE

Run this in your notebook to display the contents of your mymodel text file:

%%bash
cat ~/mymodel

We created the configuration file mymodel which contains settings for how a new model should be built. Now, create a new model named llama3.2-2, and build this custom version of llama3.2 based on what is contained within your mymodel file.

%%bash
# take the contents of `mymodel`, create a new model, call it 'llama3.2-2'
/anvil/projects/tdm/bin/ollama create llama3.2-2 -f ~/mymodel