← Back to Research
Research Note · Deep Learning & Neural Networks

Residual Learning — More Doesn’t Always Mean Good

Published: Sep 2026•7 min read•Author: Chetraj Jaishi
ResNetDeep LearningPaper NotesResidual LearningOptimization
Abstract: A personal research note taken from the famous research paper "Deep Residual Learning for Image Recognition" (He et al., the ResNet paper) — exploring why deeper networks suffer from the degradation problem and how residual shortcut connections solved it.
Research Note from a Famous Research PaperAnnotated Reading
This is a research note that I took from a famous research paper.
Source: "Deep Residual Learning for Image Recognition" (He et al., the ResNet paper)

✦WHY I'M READING THIS — THE PROBLEM IT SOLVES

Basic idea in deep learning: more layers = network can learn more complex stuff (low-level edges -> mid-level shapes -> high-level objects). So naturally you'd think stacking more layers always makes a model better.
Timeline of models getting deeper:
•
2012 — AlexNet — 8 layers
•
2014 — VGG — 19 layers
•
2014 — GoogleNet — 22 layers
But turns out that's not true past a point. This is called the DEGRADATION PROBLEM:
•
Expectation: more layers = more accuracy
•
Reality: after a certain depth, accuracy stops improving and starts getting WORSE
•
Important: this isn't overfitting, it's not even a memory/hardware issue. Even training error (accuracy on data it's already seen) goes up with more layers.
Why does this happen? Because as the signal (gradient) used to teach the network passes through too many layers, it gets "lost" or degraded. This is basically the vanishing/exploding gradient problem — the teaching signal becomes too small or too large and the network just stops learning properly.
So the real question the paper is solving: how do you make networks deeper WITHOUT the training getting worse?

✦TERMS I NEEDED TO KNOW FIRST

•
Deep Neural Network: AI that uses many stacked "layers" of calculation to find patterns (like spotting a cat in a photo)
•
Depth: number of layers stacked — like floors in a skyscraper, more floors = more depth
•
Identity mapping: a layer just passes its input straight to the next layer, unchanged — like a relay racer passing the baton without messing with it
•
Shortcut connection: an extra path that lets data skip over one or more layers, so original info doesn't get lost
•
Residual mapping / residual function: instead of learning the WHOLE answer, a layer only learns the "leftover" (the residual) — the small difference needed on top of what the previous layer already figured out
•
ImageNet / CIFAR-10: standard "test libraries" of images researchers use to benchmark how good a model is at recognizing objects
•
SGD (Stochastic Gradient Descent): the main method used to train the network / minimize its mistakes

✦WHAT THEY ACTUALLY DID — THE FIX

Traditional approach: every layer tries to learn a completely new mapping from scratch.
X_new = f(X)
Basically like every layer writing an entire essay from zero, one after another.
ResNet's approach: instead of learning a totally new answer H(x), let the layer learn just the DIFFERENCE (residual) between the input and the answer.
output = F(x) + x
Here "x" is what the network already knows (passed through the shortcut), and "F(x)" is the small adjustment/residual that this layer adds.
Think of it like: first layer adds one sentence. Second layer looks at what the first layer produced and only adds ITS change — and if that change turns out useless, it can basically output zero and let the next layer take over, without breaking the flow. Nothing gets lost because the shortcut keeps carrying the original info forward.
This is done using "shortcut connections" — extra paths that skip over layers and let the identity mapping happen, so the signal doesn't degrade even in a very deep network.
Result: they managed to build ResNets that reached 152 layers — 8x deeper than earlier models like VGG — while actually being LESS complex / computationally lighter.

✦PLAIN NETWORKS VS RESNETS — WHAT THEY COMPARED

They first tested plain networks (normal stacked layers, no shortcuts) vs ResNets (same idea but with shortcut connections).
Findings:
•
Plain 34-layer network actually performed WORSE than a plain 18-layer network (proves the degradation problem is real)
•
With ResNet, this completely reversed — the deeper 34-layer ResNet outperformed the 18-layer one
•
So the shortcuts were the actual thing letting depth translate into better accuracy, which wasn't possible before

✦BOTTLENECK DESIGN — HOW THEY MADE IT EFFICIENT

To go even deeper (ResNet-50, ResNet-101, ResNet-152) without running out of memory or making training painfully slow, they redesigned the block using a "bottleneck":
Instead of 2 layers per block, they used 3:
1.
1×1 conv layer — shrinks the data (reduces dimensions)
2.
3×3 conv layer — actually processes it
3.
1×1 conv layer — expands it back to original size
This keeps computation cheap while still letting the network go very deep.
Side note: ResNet layers tend to have smaller responses overall — meaning they stay closer to zero and don't distort/disturb the data as much as plain layers do. Makes sense since each layer is only learning a small residual adjustment, not a whole new representation.

✦RESULTS / WHY THIS MATTERED

•
Classification: ResNets aren't just good for classification (naming what's in an image).
•
Generalization: They generalize really well — way better at object detection and segmentation (outlining objects) too.
•
Major Benchmark Jump: ResNet-101 boosted object detection scores by ~28% compared to the previous industry standard (VGG-16) — a huge jump.

✦TL;DR

•
Problem: deeper networks should be smarter but instead got harder to train and less accurate (degradation problem, caused by signal loss through many layers).
•
Fix: instead of each layer learning a full new mapping, make it learn just the residual (difference) on top of a shortcut-carried input → output = F(x) + x.
•
Result: could train networks 8x deeper (152 layers) that were both more accurate AND less complex, and this residual idea generalized beyond classification to detection and segmentation too.

Key Research Takeaways

  • The Degradation Problem: As depth increases, plain networks degrade in training accuracy due to signal/gradient degradation — not overfitting or compute bottlenecks.
  • Residual Formulation: Learning residual differences F(x) = H(x) − x on top of identity shortcuts (output = F(x) + x) allows optimal gradient backpropagation.
  • 8x Deeper Yet Computationally Lighter: Scaled up to 152 layers while maintaining lower FLOPs via 3-layer bottleneck blocks (1×1 → 3×3 → 1×1).
  • Cross-Domain Generalization: Surpassed prior state-of-the-art across ImageNet, CIFAR, object detection (+28%), and semantic segmentation.