TDM 19000 - Retrival-Augmented Generation

Overview about RAG

A Retrieval-Augmented Generation (RAG) system is a way to improve AI responses by searching a set of documents, finding relevant information from them, and then using that information to generate an applicable answer. Responses from LLMs using RAG search for this information before creating an output, so its answers are based on real data you’ve inputted rather than just what it has been previously trained to know.

LLMs cannot be directly trained on text. First, the text must be converted to a vector-format using embeddings - we experienced a bit of this towards the end of the previous project. BUT it can be computationally intensive to make the conversion from a lot of text into numbers. It is best to store the vetors in a database that we can easily come back to later.

In this case, we will use the vector database Milvus - to get a deeper understanding of how Milvus works, read the Wikipedia page here.

The Milvus database belongs in the SCRATCH directory on Anvil - similar to large Ollama files, Milvus takes up too much memory to be stored in our home directories.

In your notebook, start by directing your Milvus URI to run from your SCRATCH directory:

import os

URI = f"{os.getenv('SCRATCH')}/milvus_demo.db"
collection_name = "[first_name]_[last_name]_test_collection" # change to YOUR name

print("Using Milvus DB at:", URI)
print("Using the collection:", collection_name)

If you are, for some reason, needing to remove your entire Milvus database and start over, you can delete it. To do this fully, you must remove the database AND the database lock file. This can be accomplished by running this command in the Terminal:

rm  $SCRATCH/milvus_demo.db  $SCRATCH/.milvus_demo.db.lock

Removing this database should be unnecessary. You should probably only need to do this if you are getting errors, for instance, when creating or updating the vector store.

We will begin working with an Ollama server. Run /anvil/projects/tdm/bin/ollama serve in a new Terminal window.

# show what port your Ollama server is running on!

with open(f"/dev/shm/ollama.{os.getuid()}") as hostfile:
    hostline = [line.rstrip() for line in hostfile]
os.environ["OLLAMA_HOST"] = hostline[0]
print(os.environ["OLLAMA_HOST"])