Imagine a sketchbook where you can erase and redraw every line as you go, rather than a stone tablet you have to chisel out perfectly before showing anyone. PyTorch is the sketchbook of deep learning — it lets researchers and developers build, test, and reshape their ideas on the fly, which is exactly why it has become the tool of choice for so much of today’s cutting-edge artificial intelligence work.

In plain terms, PyTorch is a free, open-source software library for building and training neural networks — layered, brain-inspired systems that learn patterns from examples rather than being told the rules directly. It is written primarily in Python, the same widely-used, easy-to-read programming language behind tools like NumPy and Pandas, with performance-critical pieces written in C++ underneath for speed.

Where the Name Comes From

The name splits neatly into two halves. “Torch” refers to Torch, an older open-source machine learning library written in the Lua programming language, which PyTorch’s GPU-accelerated backend code is built upon. “Py” simply signals that this newer version wraps that same powerful backend inside a friendly, Python-first interface. Put together, “PyTorch” means roughly “Torch, but for Python developers” — and that simple shift in programming language is a big part of why it caught on so quickly.

A Brief Origin Story

PyTorch was originally developed by Facebook’s AI Research lab, commonly known as FAIR, and the team first released it publicly in 2016, with the project made fully open source the following year, in 2017. As Facebook restructured into Meta, stewardship of the project moved with it, and in 2022 PyTorch was handed over to the newly created PyTorch Foundation, which itself operates under the much larger, vendor-neutral Linux Foundation. That move was deliberate: it gives the wider community — not just one company — a fair, shared say in the project’s future direction.

Other major technology companies have since joined in supporting the project. IBM, for instance, became a premier member of the PyTorch Foundation in 2023, having already collaborated on efforts to make training huge AI models with billions of parameters more efficient and to bring down the cost of checkpointing — the process of periodically saving a model’s progress during long training runs.

Built for Flexibility From the Ground Up

PyTorch was designed around one central idea: data scientists should be able to run and test small pieces of their code in real time, rather than waiting for an entire program to be fully written and compiled before seeing any results. For the very large models used in modern computer vision and natural language processing, that old wait-and-see approach could mean losing hours just to discover a single typo. PyTorch’s real-time style makes it an excellent platform for rapid prototyping and dramatically speeds up the often-frustrating process of debugging.

Deep learning’s enormous power comes from something researchers call representation learning: rather than being handed a fixed set of rules, a deep learning model learns the best way to represent a problem at the very same time it learns to solve it. This is the single biggest difference between deep learning and older, classical machine learning techniques — and PyTorch became one of the principal tools that made experimenting with representation learning genuinely practical for everyday developers.

Removing the Wait Between Idea and Result

Before PyTorch’s style of working became common, building deep learning models with the available mathematical libraries was a slow, often frustrating affair, sometimes requiring tens or hundreds of lines of code just to accomplish the simplest tasks. The emphasis in those earlier tools was squarely on raw flexibility and computational speed, not on the everyday experience of the person typing the code. PyTorch closed that gap by letting practitioners write models in a style that felt like ordinary Python, drop in a debugger breakpoint exactly where needed, and inspect every tensor mid-calculation, just as they would with a NumPy array.

A Favourite of the Research Community

PyTorch’s flexibility and approachability have made it the dominant choice in academic and industrial research alike. Between 2020 and 2024, the majority of machine learning research teams that disclosed their tooling reported using PyTorch. Every major AI research lab — OpenAI, DeepMind, Meta, and NVIDIA among them — relies on it for everything from diffusion image-generation models and large language models to recommender systems and robotics.

From Lab Notebook to Production Line

What makes PyTorch’s story particularly interesting is how thoroughly it has closed the old gap between “tools researchers love” and “tools companies can actually ship.” A model first sketched out in a research notebook using PyTorch can, with the right supporting tools, be carried all the way through to a production system serving millions of real users — without the team needing to rewrite their work in an entirely different framework along the way.

At its heart, every PyTorch model is doing the same basic thing: turning data into numbers, passing those numbers forward through a chain of simple mathematical steps to make a guess, checking how wrong that guess was, and then nudging itself backward through the same chain to do a little better next time.

Step One — Turning Data Into Numbers

Machine learning is, in a real sense, a game played in two parts: first, turn whatever data you have — a photo, a sentence, a spreadsheet of house prices — into a representative set of numbers; second, build or choose a model that can learn the most useful pattern hiding inside those numbers. PyTorch stores every one of those numbers inside a structure called a tensor, which we will look at closely in the next section.

Step Two — The Forward Pass

Once data is sitting inside tensors, a neural network processes it through a series of layers. Picture a row of small mathematical functions, each one a “neuron” — every neuron takes its input, multiplies it by a number called a weight, adds another number called a bias, and passes the result onward. Data flows forward through every layer of the network, from the very first input layer to the final output, producing a prediction at the end. This entire journey is called the forward pass.

Step Three — Measuring the Mistake

Once the network produces a guess, a loss function compares that guess to the true, correct answer and produces a single number describing how wrong the guess was. A common choice for predicting numbers (rather than categories) is Mean Squared Error, which squares the gap between the guess and the true value — punishing big mistakes far more harshly than small ones — and then averages that penalty across every example in the dataset.

Step Four — Backpropagation, the Backward Pass

This is where PyTorch’s most celebrated feature comes in. Once the loss has been calculated, the network needs to work out exactly how much each individual weight and bias contributed to that error, so it knows which way to nudge each one. This process, called backpropagation, walks backward through the very same chain of operations used in the forward pass, using a calculus rule called the chain rule to calculate each parameter’s share of the blame — values known as gradients.

Step Five — The Optimizer Updates the Model

Finally, an optimizer takes those freshly calculated gradients and nudges every weight and bias a small step in the direction that should reduce the error, scaled by a setting called the learning rate. Too large a learning rate and the model can overshoot wildly; too small, and training crawls along far too slowly. This entire five-step cycle — forward pass, loss, backward pass, gradient, optimizer step — repeats over and over, often thousands of times, until the model’s predictions become reliably good.

In any machine learning system, even one working with sounds, images, or words, all data eventually has to be represented as numbers. In PyTorch, that representation always takes the form of a tensor — the single most fundamental data structure in the entire library.

A tensor is best understood as a multi-dimensional array of numbers — a sort of mathematical bookkeeping device that can hold anything from a single value to an enormous, many-layered grid of values. The word “tensor” is really just a generic umbrella term that includes several more familiar mathematical objects as special cases.

Tensors vs. NumPy Arrays

If you have ever used NumPy, PyTorch tensors will feel immediately familiar — they function almost identically to NumPy’s ndarray structures. The crucial difference is hardware: while a NumPy array can only run its calculations on a computer’s central processing unit (CPU), a PyTorch tensor can also run on a graphics processing unit (GPU), which is built to perform huge numbers of simple calculations in parallel and can dramatically speed up the heavy number-crunching that deep learning demands.

import torch

# a 1D tensor (vector)
x = torch.tensor([1.0, 2.0, 3.0])

# a 2D tensor (matrix) of zeros
y = torch.zeros((3, 3))

# move a tensor to the GPU, if one is available
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
x = x.to(device)

What Tensors Actually Carry

Tensors do far more than just hold a model’s inputs and outputs. They also store a model’s parameters — the weights and biases that get adjusted during training — and, crucially, they can carry the information needed for automatic differentiation: the process that calculates gradients without a human ever writing out the underlying calculus by hand. Setting requires_grad=True on a tensor tells PyTorch to start tracking every operation performed on it, specifically so those gradients can be computed later.

PyTorch organises almost everything a developer needs into a small handful of cooperating modules. Understanding these few building blocks unlocks the ability to construct nearly any kind of neural network without writing a single mathematical formula by hand.

A module, in PyTorch terms, is a self-contained unit that performs a specific task — and modules can contain other modules nested inside them, the same way a chapter of a book can contain smaller sections. This nesting is what allows enormous, elaborate networks to still be saved, loaded, and moved between machines as a single, named object.

Defining a Network as a Python Class

Almost every custom PyTorch model is written as a Python class that inherits from torch.nn.Module. Inside that class, the constructor (__init__) defines which layers exist, and a separate forward() method defines exactly how data should flow through those layers on its way to a prediction.

import torch
import torch.nn as nn

class NeuralNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(10, 16)
        self.fc2 = nn.Linear(16, 8)
        self.fc3 = nn.Linear(8, 1)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = torch.sigmoid(self.fc3(x))
        return x

model = NeuralNetwork()

Here, nn.Linear(10, 16) creates a fully connected layer that accepts 10 input numbers and produces 16 output numbers, automatically creating and initialising the weights and biases needed to do so. Functions such as torch.relu and torch.sigmoid are activation functions — small mathematical adjustments applied between layers that allow the network to learn far more interesting, non-linear patterns than plain multiplication and addition alone ever could.

Autograd: The Tape Recorder for Maths

PyTorch’s automatic differentiation system works rather like a tape recorder. As operations are carried out on a tensor with gradient tracking switched on, autograd records each one onto an internal tape — formally, a directed acyclic graph, or DAG, where the leaves are the input tensors and the roots are the output tensors. When .backward() is called, PyTorch plays that tape back in reverse, applying the chain rule from calculus at every single step to work out each parameter’s gradient.

x = torch.tensor(2.0, requires_grad=True)
y = x ** 2 + 3 * x + 1

y.backward()  # triggers autograd
print(x.grad)  # tensor(7.) — the gradient of y with respect to x

Putting the Optimizer to Work

Once gradients have been calculated, an optimizer applies them. A typical training step looks the same across nearly every PyTorch project: clear out any old gradients, run the forward pass, calculate the loss, run the backward pass, and let the optimizer update the weights.

criterion = nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

optimizer.zero_grad()         # 1. clear old gradients
outputs = model(inputs)       # 2. forward pass
loss = criterion(outputs, targets)  # 3. compute loss
loss.backward()               # 4. backward pass
optimizer.step()              # 5. update weights

If there is one single feature most responsible for PyTorch’s popularity, it is this: the dynamic computation graph. Understanding what that phrase means — and why it matters so much — explains most of the difference in feel between PyTorch and many of its rivals.

What a Computation Graph Is

A computation graph is simply a way of mapping out how data flows between the different mathematical operations in a system. Picture it as a flowchart: each node performs one operation, and the connecting arrows show how results pass from one operation into the next. Every modern deep learning framework represents its neural networks this way internally.

Static vs. Dynamic: The Crucial Difference

The key question is: when does that flowchart actually get drawn? A static computation graph must be fully defined and locked in place before any real data is ever processed — the entire architecture has to be planned out and compiled in advance, rather like an architect’s blueprint that must be finished before a single brick is laid. A dynamic computation graph, the kind PyTorch uses, takes the opposite approach: the graph is built fresh, on the fly, line by line, at the exact moment the code actually runs. This is often called a “define-by-run” system.

Why This Makes Such a Practical Difference

Because the graph is built using ordinary Python as it runs, a PyTorch model can use standard if statements, for loops, and recursion exactly as any other Python program would — a model’s architecture can even change depending on the specific input it receives. Older static-graph frameworks instead required learning special replacement constructs, such as a framework’s own version of a loop or conditional, which often made the code that actually ran look quite different from the code that had been written, complicating debugging considerably.

x = torch.tensor(2.0, requires_grad=True)
y = x ** 2 + 3 * x + 1  # the computation happens immediately

y.backward()
print(x.grad)  # tensor(7.)

This flexibility is especially valuable for models with dynamic flow control, such as recurrent networks that process sequences of varying length, or convolutional networks that need to accept images of different sizes. Building that kind of variable-shaped architecture is notoriously difficult with a rigidly static graph, but entirely natural with PyTorch’s dynamic one.

The Debugging Payoff

Perhaps the most appreciated benefit is what this means for everyday debugging. Because the code that gets written is exactly the code that runs, a developer can drop a standard Python breakpoint directly into a model’s forward pass and inspect every tensor’s value at that exact moment — the same familiar workflow used for debugging any other Python program, with tools like pdb or an IDE’s built-in debugger working exactly as expected.

Reading about PyTorch only goes so far — let us build something real. We will create a tiny model that predicts a person’s salary from just two clues: their age and their years of work experience.

Step 1 — Install & Confirm

PyTorch installs through pip, Python’s standard package manager. A simple CPU-only setup is enough for small projects like this one.

pip install torch torchvision torchaudio

import torch
print(torch.__version__)
print("CUDA available?", torch.cuda.is_available())

Step 2 — Prepare the Data

Real machine learning projects usually have thousands of examples; this toy version uses just nine, purely to illustrate the mechanics. Note how the inputs are normalised — rescaled to a similar range — which helps training proceed far more smoothly.

ages = torch.tensor([40, 22, 55, 30, 45, 60, 35, 28, 50], dtype=torch.float32)
experience = torch.tensor([10, 2, 16, 6, 12, 18, 8, 4, 14], dtype=torch.float32)
salaries = torch.tensor([140, 89, 181, 113, 154, 194, 127, 104, 167], dtype=torch.float32)

inputs = torch.stack((ages, experience), dim=1)
inputs = (inputs - inputs.mean(dim=0)) / inputs.std(dim=0)
targets = (salaries / 100).unsqueeze(1)

Step 3 — Build the Model

A single linear layer is enough for this simple example — no hidden layers required. It automatically creates the weights and bias the model needs.

import torch.nn as nn

model = nn.Linear(in_features=2, out_features=1)

Step 4 — Choose a Loss Function and Optimizer

Since this is a regression task — predicting a continuous number rather than a category — Mean Squared Error is the natural loss function, paired here with stochastic gradient descent as the optimiser.

import torch.optim as optim

loss_fn = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=0.05)

Step 5 — The Training Loop

This is the five-step cycle from the previous section, now written out as runnable code, repeated across many epochs — full passes through the training data.

num_epochs = 50

for epoch in range(num_epochs):
    predictions = model(inputs)             # 1. forward pass
    loss = loss_fn(predictions, targets)    # 2. compute loss
    optimizer.zero_grad()                   # 3. clear old gradients
    loss.backward()                         # 4. backward pass
    optimizer.step()                        # 5. update weights

    if (epoch + 1) % 5 == 0:
        print(f"Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}")

Step 6 — Evaluate the Result

Finally, the trained model’s predictions can be compared against the true salaries to see how well it learned.

import numpy as np

predicted = np.round(model(inputs).detach().numpy() * 100)
print("Predicted:", predicted.flatten())
print("Actual:   ", salaries.numpy())

# Predicted: [139.  89. 181. 111. 153. 195. 125. 106. 167.]
# Actual:    [140.  89. 181. 113. 154. 194. 127. 104. 167.]

Scaling Up to a Real Network

Swapping that single linear layer for a deeper network with hidden layers and activation functions costs very little extra code, because the surrounding training loop barely changes at all.

class SalaryNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(2, 8)
        self.fc2 = nn.Linear(8, 1)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        return self.fc2(x)

model = SalaryNet()  # the training loop above stays exactly the same

The core torch library is only the foundation. Around it has grown a wide constellation of companion libraries, each purpose-built for a different domain of deep learning, plus a set of tools specifically designed to carry a model from a research notebook into a real, deployed product.

Domain-Specific Libraries

From Research Prototype to Production

Dynamic graphs are wonderful for experimentation, but production systems often benefit from the speed and predictability of a more fixed, optimised structure. PyTorch bridges that gap with several dedicated tools.

Distributed and Large-Scale Training

For models too large to fit comfortably on a single graphics card — increasingly common with today’s massive AI systems — PyTorch provides dedicated tools for spreading the work across many devices at once. Distributed Data Parallel (DDP) efficiently replicates a model across multiple GPUs while splitting up the training data between them, and Fully Sharded Data Parallel (FSDP) goes a step further, splitting the model’s own parameters across devices so that even individual models too large for one GPU’s memory can still be trained.

A Thriving Open Community

Beyond the official tools, an enormous and active developer community continually builds on top of PyTorch. Libraries like GPyTorch and BoTorch extend it into specialised areas such as Bayesian optimisation, while the official PyTorch forums and a very active GitHub presence give newcomers somewhere reliable to ask questions and find working examples.

Deep learning’s appetite for raw computing power is enormous, and the single biggest lever for taming training time is choosing the right hardware. PyTorch was built from the start to take full advantage of specialised processors designed exactly for this kind of work.

Why GPUs Changed Everything

A typical central processing unit (CPU) contains only a handful of cores, each one very good at handling a wide variety of different tasks one after another. A graphics processing unit (GPU), in contrast, contains hundreds or even thousands of simpler cores, all built to perform the same kind of calculation simultaneously. Since a neural network is essentially built from enormous numbers of nearly identical, repeated calculations, this parallel design maps onto deep learning extraordinarily well, providing a dramatic speed-up over CPU-only training.

How PyTorch Talks to the Hardware

PyTorch wraps low-level GPU code in a clean, simple Python interface, but still allows direct access to the underlying hardware when that extra level of control is genuinely needed. It supports several different hardware backends: CUDA for NVIDIA graphics cards, the most widely used option in deep learning; ROCm for AMD hardware; and MPS for Apple Silicon chips, which lets models train directly on the GPU built into modern Mac laptops and desktops — something that remains noticeably harder to achieve without workarounds in some rival frameworks.

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f'Using device: {device}')

model = model.to(device)
inputs = inputs.to(device)

This single pattern — checking for a GPU and moving both the model and the data onto whichever device is available — appears in virtually every serious PyTorch project, since it lets the exact same code run efficiently whether it happens to be executing on a laptop with no graphics card or a server packed with several of them.

Tensor Cores and Mixed Precision

Modern NVIDIA GPUs include specialised circuitry called Tensor Cores, purpose-built to accelerate exactly the kind of matrix multiplication that dominates deep learning workloads. PyTorch can also make use of Automatic Mixed Precision (AMP), a technique that performs some calculations using lower-precision numbers to save memory and increase speed, while carefully preserving full precision wherever accuracy would otherwise suffer.

Multi-GPU and Cloud Training

For genuinely large-scale work, PyTorch’s native support for distributing computation across multiple GPUs — and even across multiple separate machines — means a project is rarely permanently capped by the hardware sitting on one desk. Cloud platforms including Amazon Web Services, Google Cloud, and Microsoft Azure all offer ready-made environments for running PyTorch on powerful GPU hardware by the hour, without anyone needing to own that hardware outright.

PyTorch is not just a teaching tool or a research curiosity. It quietly powers some of the most widely used AI systems on the planet, working behind the scenes at companies whose products hundreds of millions of people touch every single day.

Companies Behind the Curtain

By Application Domain

Stepping back from individual companies, PyTorch’s use cases cluster into a few broad domains, each one playing to a particular strength of the framework.

Education and Research

PyTorch’s gentle, Pythonic learning curve has also made it a favourite teaching tool. Stanford University uses its flexibility to research new algorithmic approaches efficiently, and platforms like Udacity build entire courses for training new AI practitioners around it. For learners, that means the skills picked up while following a beginner tutorial transfer almost directly into the same tools used at the frontier of AI research.

PyTorch’s popularity is well earned, but no tool is without trade-offs. A clear-eyed look at both sides makes for far better decisions about when to reach for it.

The Deployment Gap

The single most consistent criticism of PyTorch concerns what happens after a model is trained. A model that works beautifully in a notebook still has to be wrapped in a serving layer, containerised, given GPU drivers and autoscaling rules, and monitored for latency and errors — a substantial chunk of engineering work that has very little to do with machine learning itself. Tools like TorchServe, ONNX, and a growing number of third-party deployment platforms exist specifically to ease this transition, but the experience is still generally considered less turnkey than some rival ecosystems offer.

When PyTorch Is the Right Choice

• You are doing research or rapid prototyping where the ability to inspect and modify a model mid-run matters most.
• Your data is images, audio, or text — exactly where PyTorch’s ecosystem and pretrained models are richest.
• You want to use cutting-edge pretrained models from Hugging Face or similar hubs with minimal friction.
• You are learning deep learning for the first time and want code that reads like ordinary Python.

When to Consider Alternatives

• You need mature, battle-tested mobile deployment out of the box — TensorFlow Lite remains more polished here.
• Your data is simple and tabular — classical machine learning tools like scikit-learn are often a better fit.
• Your team already has deep investment in another framework’s production tooling and switching costs would outweigh the benefits.

PyTorch’s biggest rivals are TensorFlow and its companion high-level API, Keras. All three remain excellent, actively developed tools — the right pick depends heavily on the specific job at hand and on how a team prefers to work.

Where Keras Fits Into This Picture

It is worth remembering that Keras itself is not strictly a rival to PyTorch in the way TensorFlow is — it is a high-level API that historically ran on top of TensorFlow, prioritising simplicity and approachability over low-level control. Since the release of Keras 3, it has also become possible to run Keras code on top of PyTorch itself as one of several interchangeable backends, which has blurred the old lines between these tools even further.

Common Misconceptions Worth Retiring

A few persistent but outdated beliefs are still repeated in online discussions. One holds that PyTorch simply cannot handle serious production environments — yet Meta itself runs dedicated PyTorch-based serving infrastructure handling billions of daily interactions. Another claims TensorFlow remains rigid and painful to debug — a fair criticism of its earliest versions, but one that the eager-execution mode introduced years ago has substantially addressed. Frameworks evolve quickly, and reviews written even a year or two ago can already be out of date.

The Industry-Wide Trend

Industry data published in the O’Reilly Technology Trends report for 2025 noted a 28% decline in TensorFlow content usage, framing it as a sign that PyTorch has increasingly won the attention of AI developers. That said, TensorFlow’s mature deployment ecosystem, especially for mobile and web applications, still gives it a genuine edge for certain production scenarios — meaning the most accurate summary remains that both frameworks are thriving, simply in different parts of the AI development lifecycle.

Less than a decade after its first public release, PyTorch has moved from a promising research tool to a foundational pillar of the entire AI industry — and its trajectory shows no sign of slowing down.

Recent and Upcoming Developments

Remaining Challenges

• The Deployment Gap: Despite real progress with TorchServe and ONNX, moving a trained model into a fully production-ready, monitored, autoscaling system still typically requires more manual engineering than some competing ecosystems demand.
• Resource Demands: Training today’s largest models remains extremely computing-intensive, concentrating cutting-edge work among organisations with access to substantial GPU budgets.
• Mobile Maturity: PyTorch Mobile remains genuinely useful but less polished and more manual to configure than some longer-established mobile deployment tools.
• Keeping Pace With Rapid Change: Frequent API updates, while generally improving the framework, can occasionally break existing code, requiring ongoing maintenance from teams running older PyTorch projects.

Sources & References