This track explains how a model learns from data.
The goal is to understand:
- what parameters are,
- how predictions are produced,
- how error becomes loss,
- how gradients describe parameter changes,
- how gradient descent updates parameters,
- how this extends from one weight to many weights,
- how chain rule and backpropagation train neural networks,
- and how modern optimizers automate these updates at scale.
This track should be understood first manually, then connected to NumPy and later PyTorch.
- What is a model?
- Input
- Output
- Prediction
- Target / actual value
- Parameter
- Weight
A model is a mathematical system that receives input and produces an output.
A very simple model can be:
input × weight = prediction
The weight controls the model's behavior.
Learning means:
changing the weight so predictions become closer to the correct targets.
- Prediction error
- Prediction − target
- Positive error
- Negative error
- Squared error
- Loss
- Why loss is different from raw error
- Mean loss across multiple examples
- Mean Squared Error intuition
Prediction
↓
Compare with target
↓
Error
↓
Loss
Raw error tells us:
how far and in which direction the prediction differs from reality.
Loss gives the training process a numerical score representing how badly the model is performing.
- Initial parameter guess
- Make prediction
- Calculate error
- Calculate loss
- Change parameter
- Repeat
- Observe whether loss improves
Understand that learning does not mean the model suddenly discovers an answer.
It repeatedly:
predicts
↓
measures mistake
↓
changes parameters
↓
predicts again
- What is a learning rate?
- Why parameter updates need a step size
- Small learning rate
- Large learning rate
- Overshooting intuition
- Slow convergence intuition
The gradient determines:
which direction and how strongly the parameter should move.
The learning rate determines:
how much of that suggested movement we actually take.
Conceptually:
gradient = direction and sensitivity
learning rate = step-size control
Basic intuition completed.
Deeper study still required.
- Basic derivative intuition
- Derivative as rate of change
- Relationship between parameter change and loss change
- Functions in depth
- Slope
- Slope of a curve
- Derivative notation
- Derivative of simple functions
- Power rule intuition
- Derivative of squared loss
- Derivative with respect to a parameter
Understand:
If I change this parameter slightly, how will the loss change?
This is the mathematical question that allows learning to become systematic.
- Basic gradient intuition
- Gradient sign
- Gradient magnitude
- Gradient becoming smaller near a good parameter
- Gradient calculation for one weight
- Gradient calculation across multiple samples
- Gradient as derivative of loss with respect to parameter
- Gradient vector
- Gradient with many parameters
- Partial derivatives
Positive gradient:
increasing the parameter increases loss
So gradient descent tends to move the parameter downward.
Negative gradient:
increasing the parameter decreases loss
So gradient descent tends to move the parameter upward.
Near a good solution:
gradient → smaller
because small parameter changes have less effect on loss.
- Gradient descent intuition
- Parameter update
- Learning rate × gradient
- Subtracting the gradient
- Repeating updates
- Convergence intuition
- Gradient descent from scratch
- Gradient descent on multiple training samples
- Loss landscape
- Local minima intuition
- Plateaus
- Divergence
- Oscillation
- Learning-rate sensitivity
Conceptually:
new parameter
=
old parameter
-
learning rate × gradient
Why subtract?
Because the gradient points toward increasing loss.
Gradient descent moves in the opposite direction.
- NumPy arrays
- Multiple input values
- Multiple targets
- Vectorized predictions
- Multiple errors
- Multiple squared losses
- Mean loss
- Mean gradient
- Batch parameter update
The model can learn a relationship such as:
y = 2x
from multiple training examples.
It begins from an incorrect weight and repeatedly reduces mean loss until the learned weight approaches the correct value.
The model is no longer learning from one observation at a time.
It can evaluate:
many observations
↓
many errors
↓
mean loss
↓
mean gradient
↓
one parameter update
- Whole dataset used for one gradient calculation
- Mean gradient across observations
- Formal definition of batch gradient descent
- Batch size
- Full-batch training
- Benefits
- Limitations
Understand why using the whole dataset produces one overall update direction.
Later compare this with:
- Stochastic Gradient Descent
- Mini-batch Gradient Descent
Next major milestone.
- One input with multiple parameters
- Multiple inputs
- Multiple weights
- Bias
- Weight vector
- Input vector
- Prediction from multiple weighted inputs
- Separate gradient for each parameter
- Gradient vector
Current simple model:
x × w
Next:
x1w1 + x2w2 + x3w3 + bias
This is the bridge from:
simple learner
to:
neuron
- Function with multiple variables
- Hold other variables constant
- Derivative with respect to one parameter
- One gradient per parameter
- Partial derivative notation
- Gradient vector from partial derivatives
If a model has:
w1
w2
w3
bias
we need to answer separately:
How does loss change if w1 changes?
How does loss change if w2 changes?
How does loss change if w3 changes?
How does loss change if bias changes?
Those answers form the gradient vector.
- Scalar gradient vs gradient vector
- Parameter vector
- One derivative per parameter
- Direction in parameter space
- Updating multiple parameters simultaneously
Single parameter:
one parameter
→
one gradient
Many parameters:
many parameters
→
many partial derivatives
→
gradient vector
- Loss as a function of parameters
- One-dimensional loss curve
- Two-dimensional parameter surface
- High-dimensional loss landscape
- Valley
- Minimum
- Saddle point intuition
- Plateau intuition
Understand gradient descent geometrically.
A real neural network does not have one visible curve.
It has a loss function over potentially millions or billions of parameter dimensions.
Requires deeper derivative understanding.
- Function inside another function
- Composition of functions
- Local derivative
- Multiplying effects through a chain
- Chain rule intuition
- Chain rule notation
- Chain rule in a learning model
A model may have:
input
↓
weighted calculation
↓
activation
↓
prediction
↓
loss
A weight affects loss indirectly through every intermediate operation.
The chain rule allows us to calculate that total influence.
- Operation nodes
- Values flowing forward
- Dependencies
- Forward computation
- Reverse traversal
- Local derivatives
- Gradient flow
x ──┐
× ── prediction ── error ── square ── loss
w ──┘
The graph records how the final loss depends on earlier values.
This becomes the foundation of backpropagation and autograd.
- What backpropagation actually means
- Forward pass
- Loss calculation
- Backward pass
- Chain rule through graph
- Gradient with respect to each parameter
- Reusing intermediate derivatives
- Backprop through one neuron
- Backprop through one layer
- Backprop through multiple layers
Forward:
inputs
↓
predictions
↓
loss
Backward:
loss
↓
how much each operation contributed
↓
how much each parameter contributed
↓
gradients
Then:
gradients
↓
optimizer
↓
updated parameters
- Why manual derivatives do not scale
- Computational graph recording
- Reverse-mode automatic differentiation
- Autograd intuition
- Difference between symbolic differentiation and autodiff
- Why backpropagation is efficient for neural networks
PyTorch:
requires_grad
.backward()
.grad
should only become convenient automation after the manual process is understood.
- Basic implementation
- Formal understanding
Uses:
entire dataset
→
one gradient
→
one update
- One training example per update
- Noisy gradients
- Faster updates
- Advantages
- Disadvantages
- Small subset of dataset per update
- Batch size
- Why modern neural networks commonly use mini-batches
- GPU efficiency
- Gradient noise
Conceptually:
dataset
↓
small batches
↓
gradient for each batch
↓
parameter update
- Epoch
- Batch
- Training step
- Iteration
- Relationship between dataset size and batch size
- Multiple epochs
Replace arbitrary while loops with structured training loops.
Conceptually:
Epoch 1
Batch 1
Batch 2
Batch 3
Epoch 2
Batch 1
Batch 2
Batch 3
- Basic parameter update
- SGD optimizer
- Why gradients can oscillate
- Velocity intuition
- Accumulating direction
- Momentum update
- Gradient magnitude tracking
- Adaptive step sizes
- Momentum component
- Adaptive learning rates
- First moment
- Second moment
- Why Adam is widely used
- Adam trade-offs
- Too small
- Too large
- Stable range
- Learning-rate schedules
- Warmup
- Vanishing gradients
- Exploding gradients
- Gradient clipping
- Initialization
- Poor initialization
- Xavier initialization intuition
- He initialization intuition
- Training loss
- Validation loss
- Training accuracy
- Generalization
- Underfitting
- Overfitting
- Memorization
- Model capacity
Understand:
Low training loss does not automatically mean the model learned something useful.
A model must also perform well on data it did not train on.
- Weight decay
- L1 intuition
- L2 intuition
- Dropout
- Early stopping
- Data augmentation concept
- Regularization vs optimization
Before entering the Neural Network track:
- Multiple weights
- Vectors
- Dot product
- Bias
- Partial derivatives
- Gradient vector
- Chain rule basics
Current:
prediction = input × weight
Next:
prediction =
input vector · weight vector
+
bias
Then:
weighted sum
↓
activation
↓
neuron output
After backpropagation is understood manually:
Manual:
prediction
↓
loss
↓
gradient calculation
↓
parameter update
PyTorch:
forward
↓
loss
↓
backward
↓
optimizer step
Topics to unlock:
-
requires_grad - computational graph
-
.backward() -
.grad -
zero_grad() - optimizer
-
optimizer.step()
These are useful but should not block progress.
- Numerical derivative approximation
- Finite differences
- Jacobian intuition
- Hessian intuition
- Convex vs non-convex optimization
- Saddle points
- Momentum geometry
- Adaptive optimizers
- Second-order methods overview
- Floating-point precision
- Gradient precision
- Overflow
- Underflow
- Mixed precision intuition
- Input
- Weight
- Prediction
- Error
- Loss
- Repeated parameter update
- NumPy dataset
- Multiple samples
- Mean Squared Error
- Gradient
- Learning rate
- Gradient descent
- Weight convergence
Build a learner with:
- multiple inputs
- multiple weights
- one prediction
- separate gradients
Build:
- weighted sum
- bias
- activation
- prediction
Build:
- forward pass
- loss
- computational graph
- chain rule
- gradients
- parameter update
Build:
- dataset
- batch size
- epoch loop
- mini-batch gradient descent
Compare:
- Gradient Descent
- SGD
- Momentum
- Adam
Track:
- convergence speed
- loss behavior
- stability
Completed path:
Model
↓
Parameter
↓
Prediction
↓
Error
↓
Loss
↓
Learning Rate
↓
Gradient
↓
Gradient Descent
↓
Multiple Training Samples
↓
Mean Loss
↓
Batch Gradient
Current transition:
Single Weight
↓
Multiple Weights
↓
Gradient Vector
↓
Chain Rule
↓
Backpropagation
↓
First Neuron
Priority order:
- Multiple parameters / weights
- Vector representation of parameters
- Derivatives in deeper detail
- Partial derivatives
- Gradient vector
- Chain rule
- Computational graphs
- Backpropagation
- First neuron
This track is complete at the foundation level when the following statement is fully understandable:
A model makes predictions using parameters. A loss function measures prediction quality. Backpropagation uses the chain rule to efficiently calculate how every trainable parameter contributed to that loss. An optimizer then uses those gradients to update the parameters so future predictions improve.
At that point, neural-network training should no longer feel like a hidden process.