GPU Setup & Cloud

Learning Objectives

• Verify local GPU availability using nvidia-smi and PyTorch's CUDA API
• Configure Google Colab with a T4 GPU for free cloud-based experiments
• Benchmark matrix multiplication on CPU vs GPU and measure the speedup
• Estimate the largest model that fits in your VRAM using the fp16 rule of thumb

The Problem

Most lessons in phases 1-3 run fine on CPU. But once you start training CNNs, transformers, or LLMs (phases 4+), you need GPU acceleration. A training run that takes 8 hours on CPU takes 10 minutes on GPU.

You have three options: local GPU, cloud GPU, or Google Colab (free).

The Concept

Your options:

1. Local NVIDIA GPU
   Cost: $0 (you already have it)
   Setup: Install CUDA + cuDNN
   Best for: Regular use, large datasets

2. Google Colab (free tier)
   Cost: $0
   Setup: None
   Best for: Quick experiments, no GPU at home

3. Cloud GPU (Lambda, RunPod, Vast.ai)
   Cost: $0.20-2.00/hr
   Setup: SSH + install
   Best for: Serious training, large models

s0-gpu-dispatch

Build It

Option 1: Local NVIDIA GPU

Check if you have one:

nvidia-smi

Install PyTorch with CUDA. On Linux the default wheel ships with CUDA. On Windows, copy the CUDA command from the selector at pytorch.org.

pip install torch torchvision torchaudio

Then check that PyTorch sees the GPU:

import torch

print(f"CUDA available: {torch.cuda.is_available()}")
print(f"CUDA version: {torch.version.cuda}")
if torch.cuda.is_available():
    print(f"GPU: {torch.cuda.get_device_name(0)}")
    print(f"Memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")

Option 2: Google Colab

1. Go to colab.research.google.com
2. Runtime > Change runtime type > T4 GPU
3. Run !nvidia-smi to verify

Upload notebooks from this course directly to Colab.

Option 3: Cloud GPU

For Lambda Labs, RunPod, or Vast.ai:

ssh user@your-gpu-instance

pip install torch torchvision torchaudio
python -c "import torch; print(torch.cuda.get_device_name(0))"

No GPU? No problem.

Most lessons work on CPU. The ones that need GPU will say so and include Colab links.

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using: {device}")

Build It: GPU vs CPU benchmark

import torch
import time

size = 5000

a_cpu = torch.randn(size, size)
b_cpu = torch.randn(size, size)

start = time.time()
c_cpu = a_cpu @ b_cpu
cpu_time = time.time() - start
print(f"CPU: {cpu_time:.3f}s")

if torch.cuda.is_available():
    a_gpu = a_cpu.to("cuda")
    b_gpu = b_cpu.to("cuda")

    torch.cuda.synchronize()
    start = time.time()
    c_gpu = a_gpu @ b_gpu
    torch.cuda.synchronize()
    gpu_time = time.time() - start
    print(f"GPU: {gpu_time:.3f}s")
    print(f"Speedup: {cpu_time / gpu_time:.0f}x")

Exercises

1. Run the benchmark above and compare CPU vs GPU times
2. If you don't have a GPU, run it on Google Colab and compare
3. Check how much GPU memory you have and estimate the largest model you can fit (rule of thumb: 2 bytes per parameter for fp16)

Hints

Exercise 1. Run the code from the Build It section as it is, then write down the CPU time, the GPU time, and the speedup line. Run it a second time; the first GPU run includes start-up cost, so the second numbers are the ones to compare.

Exercise 2. In Colab, select Runtime > Change runtime type > GPU, then paste the benchmark into a cell. Run !nvidia-smi in another cell first; it must show a GPU name before the benchmark result means anything.

Exercise 3. Read the total memory with torch.cuda.get_device_properties(0).total_memory / 1e9, as in code/gpu_check.py. Divide the memory in bytes by 2 to get the parameter count for fp16, then keep a margin, because activations and the CUDA context also use memory.

Key Terms