torch.OutOfMemoryError: CUDA out of memory. Tried to allocate …

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB means the next allocation does not fit in the free memory of the GPU. The message also reports the GPU's total capacity, how much memory PyTorch has allocated and how much it has reserved but not used. Make each step need less memory, or stop holding memory you no longer use.

Read the numbers in the message

• allocated by PyTorch: memory used by live tensors.
• reserved by PyTorch but unallocated: cached blocks PyTorch keeps for reuse. A large value here points to fragmentation.
• If the GPU's free memory is small but PyTorch's own usage is low, another process holds the memory; check with nvidia-smi.

Fixes, from most to least effective

# 1. smaller batches, same effective batch size with gradient accumulation
accum = 4
for i, (x, y) in enumerate(loader):          # loader with batch_size / accum
    with torch.autocast("cuda", dtype=torch.bfloat16):   # 2. mixed precision
        loss = model(x.cuda()).loss / accum
    loss.backward()
    if (i + 1) % accum == 0:
        optimizer.step(); optimizer.zero_grad(set_to_none=True)
    running += loss.item()                   # 3. keep a number, not the graph

# 4. evaluation without autograd buffers
with torch.no_grad():
    preds = model(x_val.cuda())

Fragmentation

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

Expandable segments let the caching allocator grow existing blocks instead of needing one large contiguous block. torch.cuda.memory_summary() prints a detailed breakdown when you need to see where memory goes.