Tensors · dynamic neural networks

Record the forward pass. Replay it backward.

PyTorch builds neural networks using and replaying a tape recorder. Tensors live on the CPU or the GPU; every differentiable operation you run is recorded, and backward() plays the tape in reverse.

Get started Learn the basics

Forward
records

x @ w+ b.relu().sum()

Mm
Backward0Add
Backward0Relu
Backward0Sum
Backward0

Backward
replays

record →← replay

x = torch.randn(4, 3)
w = torch.randn(3, 2, requires_grad=True)
b = torch.zeros(2, requires_grad=True)

y = (x @ w + b).relu().sum()   # recorded as it runs
y.backward()                   # replayed in reverse
print(w.grad.shape)            # torch.Size([3, 2])

Python first

When you execute a line of code, it gets executed. There isn’t an asynchronous view of the world — and the stack trace points to exactly where your code was defined.

A new tape every run

Control flow decides what gets recorded · runs illustrative

h = x @ w + b
h = h.relu() if h.sum() > 0 else h.tanh()
loss = h.sum()
loss.backward()

Run 1 · branch taken

MmAddReluSum

The tape holds ReluBackward0.

Run 2 · other branch

MmAddTanhSum

The tape holds TanhBackward0 instead — no graph to rebuild.

Change the way your network behaves arbitrarily, with zero lag or overhead.From the README, on reverse-mode auto-differentiation

torchA tensor library like NumPy, with strong GPU support

torch.autogradTape-based automatic differentiation for every differentiable tensor operation

torch.nnNeural networks, deeply integrated with autograd

torch.multiprocessingPython multiprocessing with memory sharing of tensors

torch.utilsDataLoader and other utilities