TDM 19000: Setting Up CLIP Model and Processor

To get started, we will need to download CLIP - a model that comes from OpenAI. This model learns to connect images and text; it can take an image, and tell you what string of text it is similar to. Similar to some other models we’ve worked with, the CLIP model has some pretty sizable files (far too big for our home directory), so we will have to redirect them to the much larger SCRATCH directory:

# In the Terminal !

# Make a .cache directory
mkdir  -p  ~/.cache

# Create the target directory
# (Hugging Face is the platform that hosts OpenAI's CLIP models)
mkdir  -p  $SCRATCH/huggingface

# Create a symbolic link
ln  -s  $SCRATCH/huggingface  ~/.cache/huggingface

In your notebook, run:

%%bash
ls -l ~/.cache/huggingface

…to verify that the huggingface cache is being redirected to SCRATCH. The output should look something like this:

lrwxrwxrwx 1 x-mdw x-cis220051 32 Jul 20 14:15 /home/x-mdw/.cache/huggingface -> /anvil/scratch/x-mdw/huggingface

From here, we need to get the CLIP model and processor:

CLIPModel - allows us to encode images
CLIPProcessor - handles resizing, normalization, and format conversion for images.

from transformers import CLIPModel
from transformers import CLIPProcessor

# loads a pretrained CLIP model from Hugging Face
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")

processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

When importing CLIPProcessor, you may get a long warning message. This is ok to ignore!