Large Language Models Experiments Using Google Colab
In addition to using LLM APIs from OpenAI, Google, etc. you can also run smaller LLMs yourself, either locally on your laptop if you have enough memory (and optionally a good GPU), or in the cloud. We saw in the chapter on Ollama the easiest way to run local models. Here we use the Hugging Face transformers library directly, which gives you access to thousands of open models and is the usual starting point for experiments like fine-tuning. When I need a GPU that my laptop does not have, I usually run these experiments in the cloud using either Google Colab or a leased GPU server.
We will use the small Qwen/Qwen3-0.6B model from Hugging Face. It is small enough (about 1.2 GB) to run on a laptop CPU or a free Colab instance, and it is an instruct model: it has been trained to follow instructions in a chat format. When you want better results and have a GPU available, change the model name to a larger model.
Google Colab directly supports only the Python and R languages. We can use Hy by using the %%writefile test.hy cell magic to write the contents of a cell to a local file, in this case Hy language source code, and then running that file in another cell with !hy test.hy. A notebook for this example has three cells:
1 !pip install hy transformers accelerate
1 %%writefile test.hy
2 (import transformers [pipeline])
3 ... the rest of the Hy code listed below ...
1 !hy test.hy
The first time you run the last cell, the model files are downloaded and cached on disk, so later runs start much faster.
If you have a laptop that can run this example, you can also run it locally. There is no example directory for this chapter; save the following code in a file test.hy and let uv install the dependencies in a temporary environment:
1 uv run --with hy --with transformers --with torch --with accelerate hy test.hy
Here is the code example:
1 (import sys hy)
2 ;; Hy replaces sys.executable with its own launcher. Restore the real Python
3 ;; interpreter for libraries (like PyTorch) that start Python helper processes.
4 (setv sys.executable (getattr hy "sys_executable" sys.executable))
5
6 (import transformers [pipeline])
7
8 (setv generator (pipeline "text-generation"
9 :model "Qwen/Qwen3-0.6B"
10 :dtype "auto"
11 :device-map "auto"))
12
13 (setv messages
14 [{"role" "user"
15 "content" "Sam bought a new sports car and wanted to see Mary. Continue the story in three sentences. /no_think"}])
16
17 (setv result (generator messages :max-new-tokens 200 :do-sample True :temperature 0.7))
18 ;; The pipeline returns the whole conversation; the last message is the
19 ;; model's reply. Qwen3 models may start with an (empty) <think> section.
20 (setv reply (get result 0 "generated_text" -1 "content"))
21 (print (.strip (get (.split reply "</think>") -1)))
The pipeline function downloads the model and its tokenizer and wraps them in an object that takes care of converting text to tokens and back. Passing a list of chat messages (instead of a plain string) makes the pipeline format the prompt using the model’s chat template. The :device-map “auto” option uses a GPU when one is available. Qwen3 models can “think” step by step before answering; the /no_think at the end of the prompt asks the model to skip that for this simple request.
The first lines of the program deserve an explanation. When a program is started with the hy command, Hy sets Python’s sys.executable to the hy launcher and saves the real Python interpreter in hy.sys_executable. That is usually harmless, but some libraries start helper processes by running sys.executable with Python code; PyTorch’s multiprocessing support does this. Without the fix, the helper process hands Python code to Hy and you will see an error like NameError: name ‘from’ is not defined (or, if you start the program with python -m hy, a warning that a resource_tracker process “died unexpectedly”). Setting sys.executable back to the real interpreter avoids both problems; the getattr default makes the line harmless when the module is imported from a Python program instead. The chapter Async and Concurrent Hy looks at the same issue in more depth for process pools.
The generated text varies for each run. Here is example output:
1 Sam drove his sports car to the park, where Mary was waiting. She waved him over and asked him to join her for lunch. As they sat together, Sam shared his excitement about the car and how it would bring him closer to his friends.
Small models like this one are useful for experiments and for simple, narrow tasks, but they make more mistakes than large hosted models. Macs with Apple Silicon and 16G or more of shared memory can run models with a few billion parameters, and machines with 64G or more of memory can run much larger models and even fine-tune smaller ones. While it is certainly simpler to use APIs from OpenAI and other vendors there are privacy and control advantages to running self-hosted models.