Understanding Trainable Layers in PyTorch

When building neural networks, one concept that often confuses beginners is the idea of a trainable layer. A trainable layer is any layer that contains parameters that are updated during training using backpropagation.

What Makes a Layer Trainable?

A layer is trainable if it contains parameters such as weights or biases.

During training the following steps happen:

The model makes a prediction
The loss function measures the error
Gradients are calculated
The optimizer updates parameters

Only layers with parameters can be updated. These are called trainable layers.

The Simplest Test

Ask this question:

Does this layer have weights or bias?

Yes → Trainable
No → Not trainable

Common Trainable Layers in PyTorch

Layer	Why it is Trainable
nn.Linear	Contains weight and bias
nn.Conv2d	Contains convolution kernels
nn.BatchNorm	Contains learnable scale and shift
nn.Embedding	Contains embedding matrix

Example:

self.fc = nn.Linear(10,5)

This layer internally contains:

weight : (5,10)
bias   : (5)

These parameters are updated during training.

Non-Trainable Layers

Some layers perform operations but do not learn parameters.

Layer	Reason
ReLU	Applies max(0,x)
Sigmoid	Mathematical transformation
Dropout	Random masking
MaxPool	Selects maximum value
Flatten	Changes tensor shape

Example:

x = torch.relu(x)

ReLU simply applies the function max(0,x). It contains no parameters and therefore is not trainable.

Important Rule When Building Models

All trainable layers must be defined inside __init__ and referenced using self.

class Model(nn.Module):

    def __init__(self):
        super().__init__()

        self.fc1 = nn.Linear(2,4)
        self.fc2 = nn.Linear(4,1)

    def forward(self,x):

        x = torch.relu(self.fc1(x))
        x = self.fc2(x)

        return x

PyTorch discovers parameters through:

model.parameters()

Common Beginner Mistake

Defining trainable layers inside the forward function:

def forward(self,x):

    fc = nn.Linear(2,4)

    return fc(x)

This is incorrect because:

New weights are created every forward pass
The optimizer never sees them
The model never learns

Mental Model

Think of a neural network as a factory:

Trainable layers → machines with adjustable knobs
Activation functions → fixed rules

Training adjusts the knobs to reduce prediction error.

Final Takeaway

Trainable layers are the parts of a neural network that contain parameters and get updated during training.

If a component has no parameters, it is not trainable — even if it is called a layer.