09 - Convolutional Neural Networks (CNN)

Class: CSCE-421

Notes:

Convolutions

Fully Connected Layer

input ($1\times 3072$) -> $Wx$ ($10\times 3072$) -> activation ($1\times 10$)

1 number: the result of taking a dot product between a row of W and the input (a 3072-dimensional dot product)

Convolution Layer

• $32\times 32\times 3$ image
• $5\times 5\times 3$ filter
• Filters always extend the full depth of the input volume
• Convolve the filter with the image i.e. "slide over the image spatially, computing dot products"

• 1 number: the result of taking a dot product between the filter and a small $5\times 5\times 3$ chunk of the image (i.e. $5^{\ast }{5}^{\ast }3=75$-dimensional dot product + bias)

Notes:

• For example if you have 3 images, you will apply one different filter to each one then we add them together
• In early days this technique has been used!
• More filters = more calculations -> your model is slower
• Most times we do full connections: each output is connected to each input, which means in this example we have 3 channels but that is not required.
• 3 images -> 3 filters -> 1 output slice

• convolve (slide over all spatial locations)
• activation map: $28\times 28\times 1$

Notes:

• Remember that a different filter generates a different output
• All the output slices are generated independently
• All output slices are completely independent
• Applying a filter will yield completely independent slices

• consider a second, green filter
• activation maps: $28\times 28\times 1$

Notes:

• We want to use this convolution to get some features of the input image!
• Things to talk about on (will be on the exam)
  • If you know the size of the input, if you know the size of the filter, how can you calculate the size of the output?

Notes:

• This operation of convolution is by far the most important one
• It will be a network with many layers but the convolutional network will be by far the most important one
• If you want 6 output slices, you need to have
  • To generate one slice you need a 5x5x3 filter
  • To generate 6 you need (5x5x3)x6

• Preview: ConvNet is a sequence of Convolution Layers, interspersed with activation functions.

• one filter => one activation map
• example $5\times 5$ filters (32 total)
• We call the layer convolutional because it is related to convolution of two signals:

- elementwise multiplication and sum of a filter and the signal (image)

Notes:

• You want to have filters that may relate to the input image so you can account for features of the image or some rotations
• Convolution = elementwise multiplication and sum of a filter and the signal (image)

Notes:

• The image does not have to be a square but in most cases kernels are just squares (rectangle filters are not commonly used)
• How to calculate the output image size if we have a 32x32 input image with a 5x5 filter?
• Note size = 28 = 32 - 5
• The size is the determined by:
  • size of input - size of filter + 1

Checkpoint 1 (convolution)

1. The Fully Connected Layer (The Old Way) Imagine you have a tiny color image that is 32 pixels wide, 32 pixels high, and has 3 color channels (Red, Green, Blue). In total, that is $32\times 32\times 3=3072$ numbers. In a Fully Connected (FC) Layer, the network takes all 3072 numbers and stretches them out into one massive, flat line. To generate just one output number, the network uses a filter that is exactly the same size: 3072 weights. It multiplies every single pixel by a corresponding weight and adds them all together (a 3072-dimensional dot product).

• The Problem: This means the network is trying to look at the entire image at once to find a pattern. It is computationally exhausting and ignores the fact that images are composed of localized features (like a small edge or a single eye).

2. The Convolution Layer (The New Way) Instead of looking at the whole image at once, a Convolutional Layer looks at small, local patches.

• The Input: Your image is a 3D block of numbers: $32\times 32\times 3$.
• The Filter (Kernel): We create a smaller 3D block to act as our filter, for example, $5\times 5\times 3$.
  • Crucial Rule: A filter's spatial dimensions (width and height) are small ($5\times 5$), but its depth must always match the depth of the input. Because the image has 3 color channels, the filter must also have 3 channels so it can process Red, Green, and Blue simultaneously.
• The Dot Product: You place this $5\times 5\times 3$ filter in the top-left corner of the image. The filter contains 75 weights ($5\times 5\times 3=75$). You multiply these 75 weights by the 75 image pixels they are touching, add them all up, and add a single "bias" number ($w^{T}x+b$). This entire operation produces just 1 number.

3. Activation Maps (Output Slices) Next, you slide this filter across the image, doing this 75-dimensional dot product at every single spatial location. When you finish sliding the filter across the entire image, you will have created a brand new, flat 2D grid of numbers. This new grid is called an Activation Map (or output slice).

• If your filter was learning to find "red vertical edges," this activation map will have high numbers wherever a red vertical edge existed in the original image, and low numbers everywhere else.

4. Multiple Filters = Multiple Slices One filter can only look for one specific type of feature. But to understand an image, we need to find horizontal edges, diagonal edges, green spots, etc.

• To do this, we simply use multiple independent filters.
• If you use 6 different $5\times 5\times 3$ filters, you will slide each of them across the image independently. This will generate 6 different activation maps.
• You then stack these 6 flat maps on top of each other. Your new output is now a 3D block of numbers with a depth of 6!

5. The Output Size Calculation (Basic) Your notes introduce the fundamental formula for figuring out the size of your new activation map: size of input - size of filter + 1.

• Why? Imagine a ruler that is 32 inches long (your input). You have a 5-inch sliding window (your filter). If you place the window at the very beginning (inch 1 to 5), you can slide it exactly 27 times before it hits the right edge (inch 28 to 32).
• Therefore, the window can be placed in exactly 28 unique positions ($32-5+1=28$). This is why sliding a $5\times 5$ filter across a $32\times 32$ image results in a $28\times 28$ activation map.

A closer look at spatial dimensions:

• $7\times 7$ input (spatially)
• assume $3\times 3$ filter
• => $5\times 5$ output

Notes:

• Roughly the output will be half the size of the input (if you do not consider the bottom)
• Remember so far we had:
  • size = size of input - size of filter + 1
• Size = 7 - 3 + 1
  • It will be 5, which is not
• So you need to do:
  • size = ((input size - filter size)/ 2) + 1

Notes:

• This is why the pixels on the borders are not treated fair in comparison to the pixels in the center
• So what have we done about this?

Stride

• The stride is a parameter that determines how many pixels the filter (or kernel) moves across the input image at a time. Think of it as the "step size" of the sliding window as it scans the data.
  • When a convolutional layer processes an image, it doesn't just look at the whole thing at once; it slides a small filter over the pixels.
    • Stride of 1: The filter moves one pixel at a time. this captures the most detail but results in a larger output.
    • Stride of 2: The filter jumps two pixels at a time. This skips every other pixel, effectively "jumping" across the image.
• Why Adjust the Stride?
  • Adjusting the stride allows you to control two main factors in your model:
    1. Output Dimensions: A larger stride reduces the spatial dimensions (height and width) of the output volume. This is often used as an alternative to pooling layers to downsample the data.
    2. Computational Efficiency: Because a larger stride results in fewer total "steps" for the filter to take, it reduces the number of calculations required, speeding up the processing time.

In practice: Common to zero pad the border

Notes:

• Every time you apply a filter, your output size is reduced by some amount

  • This will limit the number of convolutional layers that you can apply, because each layer will receive a smaller slice

• Now what if your input has been increased by 1 pixels in each direction?

• This is equivalent to apply a padding

• This will not affect the output size

  • If you do the padding and apply convolution, you will never decrease the size of the slice

• And this will help because it will somehow mitigate this bothering effect were border pixels were treated unfairly

• In conclusion, the size of your output will be affected by:

  • The size of the input
  • The size of your filter
  • The stride
  • The padding?

• Note that in most cases stride will be equal to 1 unless mentioned explicitly

e.g. input $7\times 7$
$\mathbf{3}\mathbf{\times }\mathbf{3}$ filter, applied with stride 1
pad with 1 pixel border => what is the output?

• $\mathbf{7}\times 7$ output! in general

Common to see CONV layers with stride 1, filters of size FxF, and zero-padding with (F-1)/2. (will preserve size spatially)

• e.g. $F=3\Rightarrow$ zero pad with 1
• F = 5 ⇒ zero pad with 2
• F=7=> zero pad with 3

Notes:

• After applying convolution, padding on the input does not affect the size of the output

• Remember back to... E.g. $32\times 32$ input convolved repeatedly with $5\times 5$ filters shrinks volumes spatially!
• (32->28->24 ...). Shrinking too fast is not good, doesn't work well.

Checkpoint 2 (spatial dimensions)

To understand this section, we need to address two major problems that happen when we slide a filter over an image: the shrinking problem and the border problem.

1. The Shrinking Problem When you slide a filter (like a $3\times 3$ box) across an image, the filter must stay completely inside the image boundaries. Because the filter cannot hang off the edge, you cannot place the center of the filter over the very first pixel. As a result, your output is always smaller than your input. If you have a $7\times 7$ image and a $3\times 3$ filter, sliding it one pixel at a time will only yield 5 valid placements, giving you a $5\times 5$ output. If you build a "Deep" Neural Network and apply this convolution repeatedly (e.g., $32\to 28\to 24\to 20$), your image will rapidly shrink down to nothing before the network has a chance to learn complex patterns.

2. What is Stride? "Stride" is the step size of your sliding window.

• Stride of 1: You slide the filter $1$ pixel at a time. This captures the maximum amount of detail but yields a larger output feature map.
• Stride of 2 (or more): You slide the filter, jumping $2$ pixels at a time, skipping the pixels in between.
• Why use a larger stride? It intentionally down-samples (shrinks) the output dimensions and drastically reduces the number of calculations the computer has to perform, speeding up the model.

3. Calculating Spatial Dimensions (The Math) To figure out exactly how big your output image will be, your notes provide a formula: Output Size = $\frac{\text{Input}-\text{Filter}}{\text{Stride}}+1$. Let's apply this to a $7\times 7$ image with a $3\times 3$ filter:

• Stride 1: $(7-3)/1+1=5$ (Valid $5\times 5$ output).
• Stride 2: $(7-3)/2+1=3$ (Valid $3\times 3$ output).
• Stride 3: $(7-3)/3+1=2.33$. What does 2.33 mean? It means the filter does not fit evenly across the image at this step size; it falls off the edge. In practice, programming frameworks will either throw an error or drop the remaining border pixels.

4. The Border Problem Think about a pixel in the exact center of your image. As the $3\times 3$ filter slides across, that center pixel gets included in $9$ different dot-product calculations. Now think about the pixel in the absolute top-left corner. It only gets included in one calculation (when the filter is in the very first position). Because of this, border pixels are treated "unfairly" and their information is largely ignored compared to center pixels.

5. The Solution: Zero Padding To fix both the shrinking problem and the border problem, we use Zero Padding. This means we artificially glue a border of dummy pixels (usually with a value of $0$) all the way around the outside of the original image.

• Fixing the borders: By adding a fake border, the original corner pixels are now mathematically treated like center pixels, allowing the filter to pass over them multiple times.
• Fixing the shrinking: If you add the correct amount of padding, you can force the output size to be exactly the same as the input size. This is called a "same convolution".
• The Padding Rule: To perfectly preserve the spatial size of an image when using a stride of 1, the rule of thumb is to use padding equal to $P=\frac{F-1}{2}$.
  • If Filter ($F$) = $3$, Padding = $1$ pixel border.
  • If Filter ($F$) = $5$, Padding = $2$ pixel border.
  • If Filter ($F$) = $7$, Padding = $3$ pixel border.

If you take a $7\times 7$ image, pad it by $1$ (making it $9\times 9$), and run a $3\times 3$ filter over it with stride $1$, the output will be exactly $7\times 7$. Padding does not negatively affect the output; it simply allows you to build much deeper networks without your data shrinking out of existence.

Convolution: translation-equivariance

• Process each window in the same way

• f(x) means a convolution
• T(x) means a translation (move the image)
• Again remember this is only true for translation
• The network so far is:
  • Translation equivariant
  • But not rotation equivariant
    • Note that in general filters are not rotated, they are fixed

Convolution = local connection + weight-sharing

Notes:

• Look at the right model, this is the fully connected layer we have talked about in 06 - Multi-Layer Perceptron-Networks
• How are convolutional layers related to this?
• Convolution is a special case of a fully connected layer
  • If your kernel size is 3, the output will only be connected to 3 inputs, you can see this is a locally connected layer (like in the middle model)
  • But the second thing about convolution is that we have shared parameters, each line of the same color has the same value.
• Note eventually these connection weights are trained from data, how can we guarantee that a matrix will have the same wiring ready for convolution?
  • In practice you do not need to worry about this but in implementation you will need to initialize parameters to the same value and in an update you compute the gradient of each and you do the average of each.
  • Not required to understand backward propagation in a convolutional layer in this class.

Convolution: linear transform

Notes:

• The left diagram is a one-dimensional convolution representation

• Think about it as an x vector (the input vector)

• Your output is a 4-dimensional y vector

• Values on the diagonal have to remain exactly the same

• Note that each output is connected to 3 inputs in this case

• That is equivalent to taking this w vector and get the product with the x vector

• You still have a fully connected layer with a W matrix but what is special about it is that some of the entries are fixed to 0 and some of them have the same value.

• Conceptually this is useful, but in practice this is not really what we implement, this is not the most efficient way of doing it.

• The idea here is inductive bias

  • The network is not smart enough, if you allow W to be a general matrix, it will not understand
  • You need to ask the network to learn by themselves

Receptive Field

Notes:

• Each unit will look at 3 units in the previous layer
• The network will be stacked with many layers
• each unit on a top layer will look at 3 units on the bottom layer
• So each unit on top will be able to look at a larger and larger areas of the input (covering a larger span)
• The idea is that:
  • If you have an image and you do some convolution and have some outputs
  • A unit is looking at some areas of the input
  • Our goal is to convert this area into a fixed length vector
  • We want x to be a one by one feature map
    • You could have a 100-dimensions by each one by one
    • You are basically capturing 100 different features of the input
  • This unit captures some kind of feature of the input image
• Translation Invariance:
  • If you have an object and move it, it will also move after a convolution is applied
  • If we keep reducing the size of the feature map with each layer, the final layer will be a very small feature map
  • This final layer is something called global pooling, we will compute the max over this final feature map
    • Now the location does not matter anymore

Checkpoint 3 (CNNs & Invariance)

To truly understand Convolutional Neural Networks (CNNs), we need to look at why they work so much better than traditional networks for images. This section covers the fundamental rules and "biases" we build into these models.

1. Translation Equivariance vs. Translation Invariance These two terms sound almost identical but mean very different things in Machine Learning (and professors love to test you on the difference!):

• Translation Equivariance (The Convolution property): Think of the math formula $f(T(x))=T(f(x))$. If $f$ is our convolution filter and $T$ is a translation (shifting an image to the right), this means: Convolving a shifted image gives you a shifted feature map. Because the filter slides uniformly across the entire image, if you move the cat in the image, the "cat detection" bright spot in the output simply moves by the exact same amount.
• Translation Invariance (The Goal property): "Invariance" means no change. If you shift the cat in the input image, the final classification output ("100% Cat") does not change at all. The network doesn't care where the cat is, just that it is there.

2. How CNNs achieve Invariance (Global Pooling) If convolutions only give us equivariance (the output map shifts), how do we achieve invariance? We use a Global Pooling layer at the very end of the network. Imagine your final 2D feature map detects "cat-ness." If the cat is in the top left, the high numbers are in the top left. If you apply a "Global Max Pooling" operation, you simply ask the computer to find the single highest number in that entire 2D grid and throw the rest of the grid away. Whether the high number was in the top-left or bottom-right, the final extracted number is the same. Location is discarded, and translation invariance is achieved!

3. Convolution is just a restricted Fully Connected (FC) Layer Your notes make a brilliant mathematical connection: a convolution is actually just a Fully Connected layer with two strict rules applied to it.

• Rule 1: Local Connections (Sparsity): In an FC layer, every output node connects to every single pixel in the input image. In a CNN, an output node only connects to a tiny local patch (like $3\times 3$ pixels). In matrix math, this is equivalent to taking the giant $W$ weight matrix of an FC layer and forcing most of the connections to be exactly $0$.
• Rule 2: Weight-Sharing: In an FC layer, every connection has its own unique weight. In a CNN, as the $3\times 3$ filter slides across the image, it uses the exact same 9 weights for every single patch. In matrix math, this means the non-zero diagonals in your $W$ matrix are forced to be identical numbers.
• Why this matters: In code, you don't literally build a giant sparse matrix with identical weights (that is inefficient). But conceptually, by restricting the network to local connections and shared weights, we drastically reduce the number of parameters the network has to learn.

4. Inductive Bias By applying the two rules above, we are giving the network an Inductive Bias. A neural network is essentially a blank slate. If you give an FC network an image, it has no idea that pixels close together form shapes. It has to learn that from scratch, which takes massive amounts of data. An Inductive Bias is our way of giving the network a hint. By forcing it to use small sliding windows with shared weights, we are embedding our human assumption that "local pixels form features, and features are useful no matter where they appear in the image."

5. The Receptive Field The "receptive field" is simply the area of the original input image that a specific hidden unit is "looking at."

• In Layer 1, a unit using a $3\times 3$ filter looks at exactly $3\times 3$ pixels of the input image.
• In Layer 2, a unit looks at a $3\times 3$ patch of Layer 1's outputs. But remember, each of those Layer 1 outputs is summarizing a $3\times 3$ patch of the original image! So that single unit in Layer 2 is indirectly looking at a $5\times 5$ area of the original image.
• As you stack more and more convolutional layers, the units at the top of the network develop a massive receptive field, eventually "seeing" the entire original image. This allows the network to combine small local features (edges) into massive global features (a whole dog) without needing giant filters!

Examples time:

Input volume: $\mathbf{3}\mathbf{2}\mathbf{\times }\mathbf{3}\mathbf{2}\mathbf{\times }\mathbf{3}$
$105\times 5$ filters with stride 1 , pad 2

• Note pad 2 means 4 in each dimension
• Remember these are fully connected layer (always care about this)

Output volume size: ?

• 32x32x10

  

  • Input volume: 32x32x3
  • 10 5x5 filters with stride 1, pad 2
  • Output volume size: (32 - 5 + 2 * 2) / 1 + 1 = 32 spatially, so 32x32x10

• There will be exam questions about this

Number of parameters in this layer?

• Look at #Convolution linear transform

• To generate one output slice you need a filter of size 5x5x3

• This has nothing to do with stride, pad, size of input or size of output

• This is only related to the size of filter and the number of input feature map

• To generate one output feature map you need 5x5x3 + 1 parameters

  • (+1 for bias)

• In this case you want to generate 10 slices so the number of parameters will be (5x5x3 + 1)x10

  

  • Input volue: 32x32x3
  • 10 5x5 filters with stride 1, pad 2
  • Number of parameters in this layer?
  • each filter has 5 * 5 * 3 + 1 = 76 params => 75 * 10 = 760
  • So what is an important factor for determining the number of parameters?
    • (5x5) size of filter -> larger filter = more parameters
    • (x3) input feature map -> every output is connected to every input feature map, this way you will have more filters
    • (x10) output slices -> more output slices = more parameters because each slice is completely independent

• To reduce the size of your parameters you can reduce any of the above

• What is the smallest filter you can use?

  • A 1x1 -> basically a number
    • But this will not have a interesting pattern
    • Although it is used in other applications
  • In reality if you want to extract an interesting pattern you need at least a 3x3 filter.

1x1 convolution layers make perfect sense

"it is just used to reduce the number of feature maps -> and therefore reduce the number of parameters"

• 56x56x64 -> 1x1 CONV with 32 filters -> 56x56x32
• (each filter has size 1x1x64, and performs a 64-dimensional dot product)

Notes:

• How are these filters useful?
• If you apply a 1x1 filter, the output size will not change even without padding
• Because you have 64 slices, you need to have 64 filters, each of these filters will be 1x1
• You just need to worry about generating one slice (each one is generated independently in the say manner):
  • You multiply each of the slices by the filter number
  • Each output slice will be a linear combination of the input slices
• This is useful just because:
  • If we want to apply a convolution, the size of the kernel will be small
  • Lets say we want to generate 3x3 filters for 32 slices
  • Number of parameters will be roughly 1x1x64 for 1 output slice
  • Then you apply the 3x3 filter directly on top of this
  • This way the total number of parameters compared to applying 3x3 directly ...
• 1x1 convolution is just used to change the number of feature maps

Pooling Layer

• makes the representations smaller and more manageable
• operates over each activation map independently:

• 224x224x64 (224x224 images) ->
  • downsampling -> 112x122 images ->
  • pool -> 112x112x64

MAX POOLING

Notes:

• If we use a stride of 2, the size of the output is reduced by a factor of 2
• So stride is used to reduce the output slice size
• This is useful because:
  1. 0 parameters
     • What if an object is slightly distorted
     • If you do max pooling that local distortion is actually ignored
     • There is no parameter here
  2. Slice by slice
     • Each slice is completely independent
     • You apply pooling to each of the slices completely independent
     • the output dimension is reduced by a factor of 2, but will not change the number of slices
• How to do backward propagation on convolutional networks is not required in this class

Checkpoint 4 (Examples, 1x1, Pooling layer)

To master Convolutional Neural Networks (CNNs) for your exam, you need to be able to calculate exactly how data changes as it flows through the network. This section breaks down the math, followed by two special techniques used to shrink the data.

1. The Output Size (Feature Map Size) Imagine you are given a $32\times 32\times 3$ color image (32 width, 32 height, 3 color channels). You want to apply 10 different $5\times 5$ filters to it, using a stride of 1 and a padding of 2. How big is the output?

• Spatial Size (Width/Height): We use the formula from the previous slides: $\frac{\text{Input}-\text{Filter}+2(\text{Padding})}{\text{Stride}}+1$. Plugging in the numbers: $\frac{32-5+2(2)}{1}+1=\frac{31}{1}+1=32$. So, the width and height remain $32\times 32$.
• Depth (Channels): Remember the golden rule: 1 filter = 1 output slice. Because you applied 10 independent filters, your output will be a 3D block with a depth of 10.
• Final Output Volume: $32\times 32\times 10$.

2. Calculating the Number of Parameters This is a guaranteed exam question! Your professor wants to know if you understand what a model is actually "learning."

• What does NOT matter: The size of the input image, the stride, and the padding have absolutely nothing to do with the number of parameters. They only dictate how many times the filter slides, not how big the filter itself is.
• What DOES matter: The size of the filter, the input depth, and the number of filters.
• The Math:
  1. Look at one single filter. Its spatial size is $5\times 5$, but remember its depth must match the input's depth (which is 3). So one filter has $5\times 5\times 3=75$ weights.
  2. Every filter gets 1 bias number added to it, making it 76 parameters per filter.
  3. Because you have 10 filters, you multiply: $76\times 10=760$ total parameters in this layer.

3. The Magic of 1x1 Convolutions A $1\times 1$ filter sounds completely useless at first—if it's only looking at 1 pixel at a time, how can it detect an edge or a pattern? The secret is that it looks through the depth. If you have an input that is $56\times 56\times 64$, a $1\times 1$ filter is actually a $1\times 1\times 64$ column. It does a 64-dimensional dot product, essentially acting as a mini fully connected layer across the channels for every single pixel.

• Why use them? They are a mathematically cheap way to reduce the number of channels (depth). If you apply 32 of these $1\times 1\times 64$ filters, your output becomes $56\times 56\times 32$. You just slashed the depth in half without losing spatial size. This acts as a "bottleneck" that drastically reduces the number of parameters needed for the next layer.

4. The Pooling Layer (Max Pooling) As you build a deep network, your representations need to become smaller and more manageable. While convolutions extract features, Pooling Layers simply shrink the spatial dimensions (width and height).

• How it works: A $2\times 2$ Max Pooling layer with a stride of 2 slides across the image. In every $2\times 2$ window, it looks at the 4 pixels, keeps the biggest number (the "max"), and throws the other 3 away. This cuts the image's height and width exactly in half.
• Slice-by-slice: Unlike convolutional filters that sum across all input channels, Pooling looks at each channel completely independently. If 64 slices go in, 64 smaller slices come out.
• 0 Parameters: Because the layer is just computing a mathematical "max," there are no weights or biases to learn. It requires 0 parameters.
• Distortion Tolerance: If an image is slightly shifted or distorted (e.g., an eye moves 1 pixel to the right), that pixel will likely still be the maximum value inside its $2\times 2$ window. Max pooling ignores that tiny shift, giving the network robustness (local translation invariance).

Fully Connected Layer

• Contains neurons that connect to the entire input volume, as in ordinary Neural Networks

How to perform BP in convolution, pooling layers?

Notes:

• ReLU is just not a linear function, you do not need to worry about it
  • ReLU(x) = max(0 x)
• This is useful because we can backward propagate with the max function

Optional for CSCE-421

Backpropagation in Convolutional Neural Networks

• Three operations of convolutions, max-pooling, and ReLU.
• The ReLU backpropagation is the same as any other network.
• Passes gradient to a previous layer only if the original input value was positive.
• The max-pooling passes the gradient flow through the largest cell in the input volume.
• Main complexity is in backpropagation through convolutions.

Backpropagating through Convolutions

• Traditional backpropagation is transposed matrix multiplication.
• Backpropagation through convolutions is transposed convolution (i.e., with an inverted filter).
• Derivative of loss with respect to each cell is backpropagated.
• Elementwise approach of computing which cell in input contributes to which cell in output.
• Multiplying with an inverted filter.
• Convert layer-wise derivative to weight-wise derivative and add over shared weights.

Backpropagation with an Inverted Filter (Single Channel)

• Multichannel case: We have 20 filters for 3 input channels (RGB) ⇒ We have $20\times 3=60$ spatial slices.
• Each of these 60 spatial slices will be inverted and grouped into 3 sets of filters with depth 20 (one each for RGB).
• Backpropagate with newly grouped filters.

Notes:

• Remember convolutions are just a special case of fully connected layers
• Can you convert back to a convolution?
  • Yes, backward propagation is still another convolution
  • But we need to use different filters (inverted in some way)
• Why is the filter inverted intuitively?
  • In backward propagation you are looking at the network in backward matter
  • In forward propagation your filter moves from left to right from top to bottom
  • The first time you will touch on a unit is when it came first
  • You are basically looking at the last number in the filter first

Mini-batch SGD

Loop:

1. Sample a batch of data
2. Forward prop it through the graph (network), get loss
3. Backprop to calculate the gradients
4. Update the parameters using the gradient

Activation Functions

tanh(x)

• Squashes numbers to range [-1,1]

• zero centered (nice)

• still kills gradients when saturated :(

  

ReLU (Rectified Linear Unit)

• Computes $\mathbf{f}(\mathbf{x})=max(\mathbf{0},\mathbf{x})$

• Does not saturate (in +region)

• Very computationally efficient

• Converges much faster than sigmoid/tanh in practice (e.g. 6x)

• Actually more biologically plausible than sigmoid

• Not zero-centered output

• An annoyance:

  • hint: what is the gradient when $x<0$ ?

  

Notes:

• Leaky ReLU is more complicated, basically you will have also a small negative slope as well

TLDR: in practice:

• Use ReLU. Be careful with your learning rates
• Try out Leaky ReLU / Maxout / ELU
• Try out tanh but don't expect much
• Don't use sigmoid

Checkpoint 5 (activation functions)

To understand this section, we first need to recall why we use activation functions. If you stack multiple layers of neurons together but only use linear math (like simple multiplication and addition), the entire network collapses mathematically into just one single linear layer. To allow the network to learn complex, curvy, non-linear shapes (like identifying a cat), we must pass the output of every neuron through a non-linear "activation function" before sending it to the next layer.

Here is the breakdown of the specific activation functions your professor discussed:

1. tanh(x) (Hyperbolic Tangent)

• What it does: It takes any real number and "squashes" it into a strict range between -1 and 1.
• The Good (Zero-Centered): Because it outputs both negative and positive numbers, the average output stays close to zero. This "zero-centered" property makes the math much friendlier for the next layer of neurons to process.
• The Bad (Saturated Gradients): Look at the graph of tanh(x). In the middle (around $x=0$), it is steep. But if you feed it a very large positive or negative number, the curve flattens out completely (it "saturates" at 1 or -1).
• Why is saturation bad? Remember that backpropagation uses the slope (gradient) of this curve to update the weights. If the curve is totally flat, the slope is 0. Multiplying your updates by 0 completely stops the learning process. This is famously known as the "vanishing gradient" problem.

2. ReLU (Rectified Linear Unit) Currently, ReLU is the undisputed king of deep learning activation functions.

• What it does: The formula is incredibly simple: $f(x)=max(0,x)$. If the input is negative, it outputs 0. If the input is positive, it leaves it exactly as it is.
• The Good:
  • No Positive Saturation: The right side of the graph is just a straight diagonal line. No matter how high the number gets, the slope is always exactly 1. Therefore, it never kills the gradient for positive numbers.
  • Computationally Efficient: The computer doesn't have to calculate complex exponential math (like it does for tanh or sigmoid). It just asks, "Is this number greater than 0?".
  • Fast Convergence: Because the gradient doesn't vanish on the positive side, networks learn (converge) much faster.
• The Bad:
  • Not Zero-Centered: It only outputs positive numbers (or zero), which can make weight updates slightly more erratic compared to tanh.
  • The Annoyance ("Dying ReLU"): Your notes ask: "What is the gradient when $x<0$?" If you look at the graph, the line is perfectly flat at 0. The slope is exactly 0. If a large weight update accidentally pushes a neuron so far into the negative side that it only ever receives negative inputs, it will only ever output 0. Because its gradient is now 0, it can never update its weights to recover. The neuron is effectively "dead" and will never learn again.

3. Leaky ReLU

• What it does: To fix the "dead neuron" problem of standard ReLU, Leaky ReLU introduces a tiny, slight slope on the negative side (for example, multiplying negative inputs by 0.01 instead of 0).
• Why it helps: Because the left side is no longer perfectly flat, the gradient is never exactly zero. Even if a neuron gets pushed into the negative region, it still has a tiny gradient that allows it to slowly learn its way back to life.

Data Preprocessing

Learning Rate in Gradient Descent

Notes:

• With smaller step size you will do more steps
• With larger step size you will do less steps

Normalization

• Same learning rate applied to all weights
• Large weights dominate updates
• Small weights oscillate (or diverge)

• Similar pace for all weights

Notes:

• You might need to use different linear rates for different w
• We need to do some data normalization to make use our data is roughly on the same scale

Data normalization in machine learning

Notes:

• How do we normalize training data?
  • Each row in the above example is x
  • You compute the mean and get the standard deviation for each of the features
  • You want each of this features here to be 0 and small standard deviation
  • In summary: Subtract the mean and divide (element-wise) by the standard deviation
• How do we normalize test data?
  • You still need a mean and standard deviation right?
  • Wouldn't it be the same as the training data?
  • Yes, you want your data to be subtracted and divided by the same numbers for training and test
  • Problem:
    • Predictions for each sample can depend on other samples
    • This means your prediction will change depending on who is in the same class
      • You can get different predictions for the same model
    • This is a very common mistake, and is a key idea oh how we do data normalization

TLDR: IN practice for Images: center only

• e.g. consider CIFAR-10 example with $[32,32,3]$ images
  • Subtract the mean image (e.g. AlexNet)
    • (mean image = [32,32,3] array)
  • Subtract per-channel mean (e.g. VGGNet)
    • (mean along each channel $=3$ numbers)

Not common to normalize variance, to do PCA or whitening

Normalization Modules

• We want to maintain variance for all layers
• normalize features in the network

Notes:

• You have a network of many layers and you need to insert some normalization layers between layers to make sure you are normalizing your data before inputing to the model layer itself.

Checkpoint 6 (preprocessing & normalization)

To understand this section, we need to look at how a neural network actually learns (Gradient Descent) and how the scale of our input data can either make this learning process smooth and easy, or chaotic and impossible.

1. The Learning Rate ($\eta$) When a network learns, it updates its weights using Gradient Descent. The formula in your notes is: $W_{new}=W_{old}-\eta \frac{\partial \epsilon }{\partial W}$

• What this means: $\frac{\partial \epsilon }{\partial W}$ is the gradient (the slope of the error). It tells the network which direction to adjust the weight to make the error smaller.
• The Learning Rate ($\eta$): This represents your "step size." If $\eta$ is very small, the network takes tiny, careful steps down the error mountain (meaning it will take a long time and many steps to finish training). If $\eta$ is very large, it takes massive jumps (meaning fewer steps, but it might accidentally jump completely over the valley and fail to find the minimum).

2. The Problem: Un-normalized Data Imagine predicting house prices using a neural network. Your first input feature is "Square Footage" (values around 2,000) and your second feature is "Number of Bedrooms" (values around 3). Because the math inside a neural network is just multiplication, the feature with the massive numbers (2,000) will cause massive gradients, while the feature with small numbers (3) will cause tiny gradients.

• The Trap: You only get to choose one learning rate ($\eta$) for the entire network. If you choose a small $\eta$ so the massive weights don't explode, the small weights will barely update at all. If you choose a large $\eta$ so the small weights can learn, the massive weights will violently oscillate or "diverge" (explode to infinity). The error landscape becomes a stretched-out ravine that is impossible to navigate efficiently.

3. The Cure: Data Normalization To fix this, we mathematically force all of our input features to be on the exact same scale. For every single feature (column) in your dataset, you do two things:

• Subtract the Mean: You calculate the average value of that feature across all samples, and subtract it from every data point. This "centers" the data at exactly 0.
• Divide by the Standard Deviation: The standard deviation measures how spread out the numbers are. By dividing by it, you force the data to have a "unit variance" (meaning the spread is scaled down to exactly 1).
• The Result: Now, "Square Footage" and "Bedrooms" both look like numbers between roughly -2.0 and +2.0. All weights can now update at a "similar pace," allowing gradient descent to work flawlessly.

4. The Golden Rule of Test Data Your notes highlight a very common, fatal mistake students make. When you normalize your Training Data, you calculate its specific Mean and Standard Deviation. When it is time to evaluate your Test Data, you must use the exact same Mean and Standard Deviation numbers you calculated from the Training Data.

• Why? If you recalculate a new mean and standard deviation using the Test Data, the exact same patient's test results would change depending on who else is in the test room! The network was trained to understand the specific scale of the training set. If you shift the scale by calculating a new mean, the network will be confused.

5. TL;DR for Images Images are a special case. Every single pixel in an image is already on the exact same scale: a brightness value between 0 and 255. Because they are naturally on the same scale, dividing by the standard deviation (variance) is unnecessary and mathematically wasteful. For images, we only center the data. We find the "average image" (or the average red, green, and blue values) across the training set, and just subtract that from every image.

6. Normalization Modules (Inside the Network) Data normalization fixes the inputs before they enter the network. But what happens inside? As data passes through layer 1, gets multiplied by weights, and passed through ReLU, the nice, neat scale gets ruined. By the time it reaches layer 10, the numbers might be massively distorted again. To fix this, we insert Normalization Modules (like Batch Normalization) between the hidden layers of the network. This ensures that the data is continuously re-normalized before it enters the next layer, keeping the variance perfectly maintained all the way through the deep network.

Batch Normalization

Batch Normalization

ONE OF THE MOST IMPORTANT TOPICS IN DEEP LEARNING

• Allows us to actually build networks of many layers
• There will be a question about batch normalization in the final exam

"you want zero-mean unit-variance activations? just make them so."
consider a batch of activations at some layer. To make each dimension zero-mean unit-variance, apply:

this is a vanilla differentiable function...

Notes:

• When we are training deep learning models, we do mini-batch SGD
  • You sample a batch of data
  • Use batch to train your network
  • calculate gradients
  • Update parameters
• You divide your data into mini-batches and you do forward propagation, backward propagation and anything needed for one layer before moving to the next batch
• The whole set of batches will be called epoch

"you want zero-mean unit-variance activations? just make them so."

1. compute the empirical mean and variance independently for each dimension.

   

   • Fully connected layer with D units
   • For each sample you will get an ... dimensional vector
   • Get the mean, get the standard deviation, subtract mean and divide standard deviation element-wise

2. Normalize

Notes:

• This is in training
• You normalize for each of the mini-batches

• BN = Batch Normalization
• FC = Fully Connected

Notes:

• Conceptually this normalization layer can be inserted anywhere, because each of the layers receives input, does some process and returns some output.
• You usually insert after fully connected or convolutional layers, and before nonlinearity

Batch Normalization (Math)

Normalize:

And then allow the network to squash the range if it wants to:

Note, the network can learn:

to recover the identity mapping.

Notes:

• A paper proposed doing something slightly different
• The normalization operation is purely computed of your data
• But with this change these two parameters become learnable as well
  • The key idea is that they try to introduce some learnable parameter in this layer
  • They try to do some data dependent adaptation (learning) and adjust normalization in that way since mean and standard deviation are not perfect.
• We can then use this to essentially un-do the normalization
  • Each of these of parameters are vectors of the same dimension of the mean and standard deviation vectors
• Remember that all of this is what happens in training, not in test time

Checkpoint 7 (Batch Normalization I)

To understand Batch Normalization (often called BatchNorm or BN), we need to remember the core problem with deep neural networks: as data passes through many layers, getting multiplied by weights and passed through activation functions, the scale of the numbers can go completely out of control.

We already solved this for the input data by normalizing it before it enters the network. But what about the data inside the hidden layers? By layer 50, the numbers could be massively shifted, causing gradients to vanish or explode. Batch Normalization is the ingenious solution to this: we simply force the data to be normalized inside the hidden layers too.

Here is the step-by-step breakdown of how it works and the math behind it.

1. The Context: Mini-Batches and Epochs When training deep learning models, we use Mini-batch Stochastic Gradient Descent (SGD).

• Instead of feeding the network one image at a time, or all 1,000,000 images at once, we feed it a small "chunk" (e.g., 32 or 128 images). This chunk is a mini-batch.
• The network does forward propagation, calculates the error, does backward propagation, and updates the weights for this specific mini-batch. Then it moves to the next chunk.
• Once the network has seen every single chunk in the dataset exactly once, that is called an Epoch.

2. The Core Math: Forcing Normalization Imagine you have a Fully Connected (FC) layer. For a mini-batch of 32 images, a specific hidden node will output 32 different numbers. Batch Normalization says: "Let's take those 32 numbers and normalize them so they have a mean of 0 and a variance of 1."

• The Mean ($\mathrm{E}[x]$): Calculate the average of those 32 numbers.
• The Variance ($\mathrm{Var}[x]$): Calculate how spread out those 32 numbers are.
• The Normalization (${\hat{x}}^{(k)}$): You take the original output $x$, subtract the mean, and divide by the standard deviation (which is just the square root of the variance).
• The Formula: ${\hat{x}}^{(k)}=\frac{x^{(k)}-\mathrm{E}[x^{(k)}]}{\sqrt{\mathrm{Var}[x^{(k)}]}}$ (Note: $k$ just represents a specific dimension or hidden unit).

3. Where does it go? Conceptually, Batch Normalization is treated as its own distinct layer. The standard order of operations in modern neural networks is:

1. Linear transformation (Fully Connected or Convolutional layer)
2. Batch Normalization layer
3. Non-linear Activation Function (like ReLU)

4. The Twist: Scale ($\gamma$) and Shift ($\beta$) If we rigorously force every single hidden layer to always have a mean of 0 and a variance of 1, we actually restrict what the network can learn. Sometimes, the network wants the data to be spread out or shifted to take full advantage of the activation function!

To fix this, the inventors of BatchNorm added a brilliant mathematical trick: Scale and Shift. After calculating the perfectly normalized data $\hat{x}$, we multiply it by a new parameter $\gamma$ (Gamma) and add a new parameter $\beta$ (Beta).

• The Final Formula: $y^{(k)}={\gamma }^{(k)}{\hat{x}}^{(k)}+{\beta }^{(k)}$

Why is this so important? $\gamma$ and $\beta$ are learnable parameters, exactly like the weights in a normal layer. The network uses gradient descent to learn the best possible scale and shift for the data.

• If the network decides that normalization is actually hurting its performance, it can easily learn to completely undo the normalization!
• To recover the original data (the identity mapping), the network just learns to set $\gamma =\sqrt{\mathrm{Var}[x]}$ (the standard deviation) and $\beta =\mathrm{E}[x]$ (the mean). The math cancels out perfectly.
• Therefore, adding Batch Normalization strictly expands the model's capabilities without ever risking a loss of representation power.

(Note: Everything we just discussed happens during Training. Test time behaves differently, which you will see in your next slides!)

Batch Normalization (Algorithm)

• Improves gradient flow through the network
• Allows higher learning rates
• Reduces the strong dependence on initialization
• Acts as a form of regularization in a funny way, and slightly reduces the need for dropout, maybe

Note: at test time BatchNorm layer functions differently:

• The mean/std are not computed based on the batch. Instead, a single fixed empirical mean of activations during training is used.
• (e.g. can be estimated during training with running averages)

Notes:

• During training you will need to somehow keep the learned parameters for batch normalization and once in test time you will use these

• Essentially you are using your training in your test

• Basically you need to do some rounding average in training of the mean and standard deviation and use that in our test.

  • "Maintain a global mean and global standard deviation in training"

• Pytorch already handles this for you

• Consequence of using mini-batch

  • Suppose you are training a network with many different layers
  • Lets say you do not use batch normalization, in training you only need to give one sample at a time, for each of the 10 samples, you won't update until you finish each of the 10 samples
  • If we add mini-batch between layers, can we still do that?
    • No because for each sample, we will need all other samples in order to do normalization
    • Basically you need to give this 10-sample at once to the network
  • The consequence is that:
    • In between each of these layers you need to keep somehow in memory the output of each layer
    • For each layer you will have some impact in memory
    • If you use very large mini-batch size, the activations for each layer will become larger -> takes more memory
    • So your mini-batch size will be limited by your memory
  • In theory larger mini-batch size is better but you are limited by your GPU memory
    • Usually not more than 10 GB
    • High end GPUs use 18 GB
    • So it is a very important point since we do not have huge memory sizes
    • But larger mini batches -> larger data set -> more accurate
  • QUESTION ON EXAM ABOUT LIMITATION OF MINI-BATCH

...

Batch Normalization: Test Time

Input: $x:N\times D$

Learnable params:

Intermediates:

Output: $y:N\times D$

Batch Normalization for ConvNets

Batch Normalization for fully-connected networks

Batch Normalization for convolutional networks (Spatial Batchnorm, BatchNorm2D)

Notes:

• That x vector is essentially why we need to understand mini-batch limitation, it gets bigger with your mini-batch size (N)

Layer Normalization

Batch Normalization for fully-connected networks

• Mean and standard deviation is d-dimention
• Here your normalization is across D

Layer Normalization for fully-connected networks Same behavior at train and test! Can be used in recurrent networks

• Your mean and standard deviation have dimension of the batch size
• Here your normalization is across N

Notes:

• Example: sentences
• If you have 10 sentences in the same mini-batch, and each sentence can have different length? how do you deal with that? You need to normalize!
  • You can use a sentence length threshold like 30 words and add or remove data from smaller or larger sentences
• Fun fact: graphics use layer normalization

Checkpoint 8 (Batch Normalization II)

To truly master normalization for your exam, you need to understand that training a model and testing a model are two completely different environments.

1. Batch Normalization at Test Time (The Rules Change) During training, Batch Normalization calculates the mean ($\mu$) and standard deviation ($\sigma$) across a mini-batch of data (like 32 images) to normalize the hidden layers.

• The Problem: What happens after the model is fully trained and deployed to the real world? A user might upload just one single image for the network to classify. You cannot calculate the variance of a single image!
• The Solution: During the training phase, the network quietly keeps track of a "running average" (or global average) of the means and variances it sees across all the mini-batches.
• When it is time for testing (inference), the network stops looking at batch statistics. Instead, it plugs in that frozen global running average for ${\mu }_j$ and ${\sigma }_j^2$. It also plugs in the final, learned scale and shift parameters (${\gamma }_j$ and ${\beta }_j$) to generate the output.

2. The Limitation of Mini-Batches (Exam Warning!) Your professor explicitly noted this will be on the exam. Why does Batch Normalization limit our mini-batch size?

• Imagine a deep network without Batch Normalization. You could theoretically feed 10 images through the network one by one, calculate their individual errors, and then average the gradients to update the weights. You only ever need to hold one image's worth of hidden-layer activations in the computer's memory at a time.
• Batch Normalization breaks this. To calculate the mean of 10 images at Layer 5, the computer must hold the Layer 5 activations for all 10 images in its memory at the exact same time.
• The size of this massive block of data is dictated by $N$ (the batch size). If $N$ is too large, the memory footprint of these simultaneous activations will exceed the physical VRAM limits of the GPU (which are typically around 10 to 18 GB). Therefore, while larger batches lead to better statistical estimates and accuracy, GPU memory strictly limits how large your mini-batch ($N$) can be.

3. Batch Normalization for Convolutional Networks (Spatial BatchNorm) When we apply Batch Normalization to a Fully Connected network, we have $N$ samples and $D$ neurons. We calculate exactly $D$ means (one for each neuron, averaged across the $N$ samples).

• In a Convolutional Network, our data is a 4D tensor: $N\times C\times H\times W$ (Batch Size $\times$ Channels $\times$ Height $\times$ Width).
• Instead of normalizing every single spatial pixel independently, we normalize per channel. For example, we find the average value of the "Red" channel by averaging across all $N$ images, and across all $H\times W$ pixels in that channel.
• Therefore, the calculated mean ($\mu$), variance ($a$), scale ($\gamma$), and shift ($\beta$) are all tiny tensors of size $1\times C\times 1\times 1$. We just have one learnable scale and shift per feature channel!

4. Layer Normalization (The Alternative to Batch Norm) Batch Normalization averages across the batch ($N$). What if we flip the math by 90 degrees and average across the features ($D$) instead? This is called Layer Normalization.

• How it works: If you have 32 samples and 100 features, LayerNorm looks at Sample #1, calculates the mean and variance of its 100 features, and normalizes Sample #1 against itself. The mean and standard deviation are vectors of size $N\times 1$.
• Why is this brilliant? Because Sample #1 is normalized completely independently of Sample #2, LayerNorm behaves exactly the same at Train time and Test time!. We do not need to keep track of running global averages.
• When do we use it? LayerNorm is essential for Recurrent Neural Networks (RNNs) and text. If you have a batch of 10 sentences, they all have completely different word lengths. Batch Normalization struggles to average the "15th word" across a batch if half the sentences only have 5 words. Because LayerNorm only normalizes within the sentence itself, it handles varying lengths perfectly!

Early Stopping

• Stop training the model when accuracy on the validation set decreases
• Or train for a long time, but always keep track of the model snapshot that worked best on val

Notes:

• If you train a Logistic regression you do not want to stop
• Once you move to networks, there is a hidden layer
• You cannot use the normal gradient because it will give you a local minimum
• You still need to have a validation data set somehow to tell you decide when to stop
  • This is because training accuracy will increase and increase but will eventually exaggerate
  • You need to reach that point of exaggeration where you need to stop, and you can now use this point with a validation data set

Regularization: Dropout

In each forward pass, randomly set some neurons to zero Probability of dropping is a hyperparameter; 0.5 is common

Notes:

• This is a very useful technique
• This only works on fully connected layers, if you have a network, there must be at least one fully connected layer and this will apply only to that layer
• Drop out:
  • It does something different in training and test
  • In training time we basically remove units, removing means removing input and output connection of a unit.
    • During forward and backward propagation you remove nodes
  • This is special for each sample
  • If your dropout probability is 0.2, you have 0.2 probability of removing each of these nodes

How can this possibly be a good idea?

Forces the network to have a redundant representation; Prevents co-adaptation of features

Notes:

• Forces you to create some kind of redundancy in your features so that if some of them are remove you can still recognize an object
• The idea is that you want to get rid of some redundancy of the representations but you still want to make accurate predictions

Another Interpretation

• Dropout is training a large ensemble of models (that share parameters).
  • Building multiple different models around a single set of data
  • Here comes the idea of a Random Forest
• Each binary mask is one model

• How many possible ways are there to drop out an entire layer if each unit has two possibilities (drop out or not drop out)
• An FC layer with 4096 units has $2^{4096}\sim {10}^{1233}$ possible masks!
  • This is a huge number! -> so many different models!
  • Only $\sim {10}^{82}$ atoms in the universe...

Notes:

• Each time you see a different dropout pattern, you are looking at a different network with the same training scheme
• You train multiple models with the same training set, training each of them costs a lot but this is a way to deal with that
• This is the most well known way to simulate ensemble in deep learning models
• Example:
  • Random Forest:
    • You need some kind of randomization that can come from data, you then build many different models
    • In deep learning this is not used because it is not practical to train different models
• remember that training a single neural network is very expensive
• For a network like the above each node has some possibility (drop out, or not)
• Each mask counts a different network, it is random and specific to each sample
  • During each mini-batch this mask might not been the same also

Dropout: Test time

Dropout makes our output random!

Want to "average out" the randomness at test-time

But this integral seems hard ...

Want to approximate the integral

Consider a single neuron.
At test time we have: $E[a]=w_{1}x+w_{2}y$
During training we have:

At test time, multiply by (1- dropout) probability

• This number is related to the dropout proabbility

Notes:

• In training we have some randomness but in test we have no randomness why>
  • In prediction everything needs to be deterministic (no randomness)
  • In training time each sample uses a different mask and you need to use every possible mask, each possible mask gives you a network and you will use each of these to make predictions.
• In prediction time you do not dropout any nodes, you use the full network to make predictions and then you multiply by the dropout probability

Regularization: A common pattern

Training: Add some kind of randomness

• Example: Batch Normalization
• Training: Normalize using stats from random minibatches

Testing: Average out randomness (sometimes approximate)

• Testing: Use fixed stats to normalize

Regularization: Data Augmentation

Notes:

• Another way to produce equivarianve is to do data augmentation
• We want the network to be invariant to image translation and rotation
• Operations such as flip, rotate, or slightly translate the image
• If we randomly rotate an image by 90 degrees, and tell the network this is a cat, then give it a 180 degrees rotated image, and keep telling the network this is a cat, and so on
  • We are doing this by giving examples and trying to teach the network to know rotations are also cats
  • But the network might still make different predictions
• During training time we try that our model be invariant, but during prediction time, this is no longer the case why?

Random crops and scales

Training: sample random crops / scales

ResNet:

1. Pick random $L$ in range [256, 480]
2. Resize training image, short side $=\mathrm{L}$
3. Sample random $224\times 224$ patch

Testing: average a fixed set of crops

ResNet:

1. Resize image at 5 scales: $\{224,256,384,480,640\}$
2. For each size, use $10224\times 224$ crops: 4 corners + center,+ flips

Regularization: A common pattern (summary)

Training: Add random noise
Testing: Marginalize over the noise

Examples:
Dropout
Batch Normalization
Data Augmentation
DropConnect

Checkpoint 9 (regularization)

To understand these slides, we have to talk about the biggest enemy in Machine Learning: Overfitting. Deep Neural Networks have millions of parameters. If you train them for too long, they will stop learning general patterns (like "what does a cat look like?") and start perfectly memorizing the training data (like "this specific pixel being blue means it's the cat from picture #4").

To fight this, we use Regularization, which is essentially a set of techniques to deliberately handicap the network so it is forced to learn general, robust features rather than memorizing the data.

1. Early Stopping

• The Concept: As you train your network, the accuracy on your training data will go up and up until it hits 100%. However, if you check the network's performance on a separate Validation Dataset (data it has never seen before), you will notice that the accuracy eventually peaks and then starts to drop. This drop is the exact moment the network starts memorizing (overfitting).
• The Solution: Early stopping simply means keeping track of the model's accuracy on the validation set after every epoch. Instead of using the model at the very end of training, you stop (or just save a snapshot of the weights) at the exact iteration where the validation accuracy was highest.

2. Regularization: Dropout Dropout is one of the most famous and effective regularization techniques in deep learning.

• The Concept (Training Time): Imagine a company where every day, 50% of the employees randomly call in sick. The remaining employees cannot rely on the "experts" to do all the work; they have to learn how to do a little bit of everything to keep the company running.
• In a neural network, during every single forward pass of the training phase, we randomly turn off (drop out) a percentage of the neurons (typically 50%, or $p=0.5$) in the Fully Connected layers. We sever their input and output connections completely.
• Why is this good? It prevents "co-adaptation". If neurons always work together, they get lazy and rely on specific other neurons to make decisions. By dropping them randomly, the network is forced to create a redundant representation of the data. It has to learn multiple ways to identify the same object because it never knows which neurons will be available.

3. Another Interpretation (The Ensemble Method) There is a second, highly mathematical way to look at why Dropout works so well.

• In machine learning, an Ensemble (like a Random Forest) is when you train 100 completely different models and let them vote on the final answer. Ensembles are incredibly accurate, but training 100 different Deep Neural Networks is computationally impossible.
• Because Dropout applies a random "mask" (a pattern of 1s and 0s turning nodes on and off) for every single sample, every forward pass is technically training a completely different mini-network.
• If your layer has 4096 units, there are $2^{4096}$ possible network combinations. Dropout is a mathematical trick to train an astronomically large ensemble of models simultaneously, all sharing the same base weights!

4. Dropout at Test Time

• The Problem: When you deploy your model to the real world (Test Time), you do not want it to be random. You want it to be deterministic (give the same answer every time). So, at test time, we turn off Dropout and use every single neuron in the network.
• The Math Fix: But wait! During training, the next layer was only receiving signals from 50% of the neurons. At test time, it is receiving signals from 100% of the neurons. The incoming numbers will be twice as large, which will completely overwhelm the network!
• To fix this and keep the expected value ($E[a]$) the same, we must multiply the outputs by the probability of keeping the neuron (which is $1-\text{dropout probability}$). If we dropped 50% during training, we multiply all weights by 0.5 at test time to scale the signals back down to normal.

5. Regularization: A Common Pattern Your professor highlights a beautiful, unifying theme across all deep learning regularization techniques (like Batch Normalization, Dropout, and Data Augmentation):

• During Training: We deliberately inject randomness and noise (random mini-batches, random dropped nodes, random image rotations).
• During Testing: We remove the randomness and average it out (using global fixed statistics, scaling the dropout weights, or averaging fixed image crops).

6. Regularization: Data Augmentation Finally, how do we make a network invariant to rotations or translations (meaning it recognizes a cat even if it's upside down or in the corner)?

• Training: We artificially expand our dataset by taking our original images and applying random transformations to them. We randomly flip them, rotate them, or take random zoomed-in crops (like ResNet taking a random $224\times 224$ patch). We feed these to the network and say, "This is still a cat."
• Testing: To average out the randomness at test time, we don't just test the image once. We take a fixed set of crops (e.g., the 4 corners + the center + horizontal flips) to create 10 images. We pass all 10 through the network and average their predictions together to get a highly confident final answer.

Transfer Learning

"You need a lot of a data if you want to train/use CNNs"

Transfer Learning with CNNs

Notes:

• Lets say you only have 2000 images and you want to teach a network with this set
• The idea is that in many cases if you try a network like the first one, the filters you learn in the lower layers are very similar even for different data sets, since a lot of these objects will have edges, so these first filters will basically give you edge specification
• If you have a very small data set you can take that network and you refine that network depending on those images
  • You need to fix many of these layers but mostly you need to change some of the later layers

Checkpoint 10 (transfer learning)

To understand Transfer Learning, you have to remember one fundamental rule about Deep Learning: training a Convolutional Neural Network (CNN) from scratch requires a massive amount of data. If you only have a small dataset—for example, just 2,000 images of specific dog breeds—trying to train a deep network from scratch will usually result in severe overfitting.

Transfer Learning is the ultimate shortcut to solving this problem.

• The Idea: Instead of building and training a network from scratch, you take an existing, powerful CNN that someone else has already trained on a giant dataset (like millions of general images).
• Why it works: When a CNN learns to process images, the early layers closest to the input always learn very basic, universal features. They learn how to detect simple edges, lines, and textures. Because almost every object in the world is made up of these basic edges and shapes, these early filters are highly reusable across completely different datasets.
• How to do it: You take the pre-trained network and "freeze" the early layers, meaning you lock their weights so they cannot change. You only update and "refine" the weights of the later, fully connected layers using your small dataset of 2,000 images.

By doing this, you are effectively transferring the "vision" capabilities the network already learned and simply teaching its final decision-making layers to recognize your specific dog breeds!

Summary:

• The Problem: Training deep CNNs from scratch requires vast amounts of data.
• The Concept: Repurposing a model trained on a massive, general dataset to solve a new, related task that only has a small dataset (e.g., 2,000 images).
• Why it works: The lower (early) layers of a CNN learn fundamental, universal features like edges and basic shapes. These low-level feature extractors are highly similar across completely different image datasets.
• The Implementation:
  1. Take a pre-trained network.
  2. Fix (freeze) the weights of the early layers to preserve the universal edge/shape detectors.
  3. Refine (retrain) only the later, task-specific layers using your new, small dataset.

CNN Architectures

Review: LeNet-5

This network will be similar to what will be on the final

Notes:

• Network:
  • You a convolutional layer
  • Pooling layer
  • convolutional layer
  • pooling layer
  • fully connected layer
• This is what you will need to implement in the next homework
• This is the idea from 1990s, at that time there was no GPU training, so this is kind of the best you could do with a CPU
• It is such a simple network and therefore not very effective
• LUSH is a software package that is open source and you can use it to implement these kind of networks
  • The L stand for LISP
  • It stands for List Processing, it is a very different language
  • Coding is very counter intuitive

Case Study: AlexNet

Details/Retrospectives:

• first use of ReLU
• used Norm layers (not common anymore)
• heavy data augmentation
• dropout 0.5
• batch size 128
• SGD Momentum 0.9
• Learning rate $1\mathrm{e}-2$, reduced by 10 manually when val accuracy plateaus
• L2 weight decay 5e-4
• 7 CNN ensemble: 18.2% -> 15.4%

Notes:

• The only thing that is new in this paper is the use of ReLU
• He used C++ and implemented convolutional networks on GPU for the first time
• There is very messy code
• This is now a much bigger network
  • You have a convolutional layer
  • Max Pool layer
  • Normalization (not batch normalization)
  • and so on
• The key about this paper is that they were able to write this code in a GPU so they were able to train a larger model

ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners

Notes:

• All of these are not deep learning
• This was essentially the first demonstration that deep learning can outperform other models!
• Before this no one thought deep learning will work
  • This tells us, sometimes if you try to develop something (even if you have a great idea) it is the implementation that will challenge you

Case Study: VGGNet

Small filters, Deeper networks

8 layers (AlexNet)
-> 16-19 layers (VGG16Net)

Only $3\times 3$ CONV stride 1, pad 1 and $2\times 2$ MAX POOL stride 2

11.7% top 5 error in ILSVRC'13 (ZFNet)
-> 7.3% top 5 error in ILSVRC'14

Notes:

• The only difference in the number is how many layers
• In these networks they only use 3x3 filetrs on top of each other
• There is no linear function between them
• They do this one after the other multiple time because:
  • If you want to achieve a larger receptive field, just stick one more convolutional layer
  • What you have shown is that it is better in terms of the function
    • They are nested, each layer is nested on top of each other
    • In total the function that can be represented here is larger
  • You can use a smaller number of parameters to represent a much complex function
• You do not need to understand this network you only need to know the idea that this is better than applying one layer of convolution with layer filter
• The only thing you need to understand here is that in this networks we only use 3x3 convolutions directly stacked on top of each other (there is no linear function in between)

Q : Why use smaller filters? ( $3\times 3$ conv)

Stack of three $3\times 3$ conv (stride 1) layers has same effective receptive field as one $7\times 7$ conv layer [7x7]

But deeper, more non-linearities

And fewer parameters: 3 * ( $3^2{\mathrm{C}}^2$ ) vs. $7^2{\mathrm{C}}^2$ for C channels per layer

Notes:

• Think about the number of parameters
• Lets say you have an input feature map
• Lets say you have that feature map 'c' there and you want to generate a 'c' feature map there.
• Lets do this 7x7 conv layer, how many parameters would be there? (lets do not count to bias also)
  • This will be 7 * 7 x c * c
  • This is 49c^2
• For a 3x3 kernel:
  • 3 * 3 * c * c
  • This is 9c^2
  • If you apply this 3 times you get:
    • 3 * 9 * c^2
    • This is 27x^2
• That is why we still use this (this idea is still common)

Example:

• The blue part tells you the number of parameters of each layer
• What can you see from this number?
• You can see memory will keep reducing, this is because as you go from layer to layer, the feature map size can be reduced over the network
  • Note that as the memory reduces, the number of parameters increases!
• What determine the number of parameters here is the number of feature maps
• When you reach the end it becomes a 4096 dimensional vector
• That is also the reason we use only one layer of fully connected layer?

Case Study: GoogLeNet

Apply parallel filter operations on the input from previous layer:

• Multiple receptive field sizes for convolution ( $1\times 1,3\times 3$, $5\times 5$ )
• Pooling operation ( $3\times 3$ )

Concatenate all filter outputs together depth-wise

Q: What is the problem with this? [Hint: Computational complexity]

Notes:

• This network itself is no longer useful but there is one important idea about it.
• Normally what we would do is to to apply some convolution to generate output
• What they want to do is to sue filters of different size to each layer (btw this idea is not longer use but it was they initial motivation)
• You cannot generate filters of different size (implementation wise is so difficult to handle)
• Before this, in each layer in any case the size of the feature map will be the same
• This will introduce so many parameters!
  • Because you will generate large number of feature maps
  • The number of parameters becomes too large

• What they tried to do is to use a 1x1 conv as a "bottleneck"

• This is a still useful technique!

• They want to reduce the number of feature maps

• They will use this 1x1 convolution mainly to reduce the number of feature maps

• Then the number of parameters will be reduced because for the layer between a 1x1 and a 3x3 convolution, it will be involve less feature maps

• This will also introduce some parameters right?

  • Yes but the key thing is that the feature size is very small, then that will reduce the size of the feature map after the 3x3 convolution

• If you calculate the number of parameters, the total number of parameters is smaller than if you applied the outputs from the previous layer directly to the 3x3 or 5x5 convolution.

• I will first use 1x1 convolutions to reduce the number of feature maps, then we apply convolutions

• That is why this is called bottle neck, you receive a large number of feature maps from the previous layer and you do 1x1 convolution to reduce this number of feature maps, and then you apply convolutions to increment the number of feature maps delivered to the next layer

Stack Inception modules with dimension reduction on top of each other

Checkpoint 11 (case studies)

Review: LeNet-5 This is a foundational 1990s architecture consisting of a simple, alternating sequence of convolutional and pooling layers that ends with a fully connected layer.

Case Study: AlexNet This architecture proved that training much larger models is possible by utilizing GPUs, and it was the first major network to successfully introduce the ReLU activation function.

ImageNet The ImageNet visual recognition challenge was the historical turning point that proved to the world that deep learning could vastly outperform traditional machine learning models.

Case Study: VGGNet This model demonstrated that stacking multiple small $3\times 3$ convolutional layers on top of each other gives the exact same "receptive field" as using a single large filter (like $7\times 7$), but it is far more computationally efficient and uses fewer parameters.

Case Study: GoogLeNet This architecture pioneered the "bottleneck" concept, using $1\times 1$ convolutions to drastically shrink the number of feature maps (and thus reduce parameters) before applying more expensive parallel filter operations.

Case Study: ResNet

What happens when we continue stacking deeper layers on a "plain" convolutional neural network?

56-layer model performs worse on both training and test error
-> The deeper model performs worse, but it's not caused by overfitting!

Notes:

• Note the 20-layer network is a subset of the 56-layer network
• In theory the 56-layer network has larger capacity than the 20-layer, just because the 56-layer network has a larger training area
• Same thing for test error

Hypothesis: the problem is an optimization problem, deeper models are harder to optimize

The deeper model should be able to perform at least as well as the shallower model.

A solution by construction is copying the learned layers from the shallower model and setting additional layers to identity mapping.

Solution: Use network layers to fit a residual mapping instead of directly trying to fit a desired underlying mapping

• Use layers to fit residual $F(x)=H(x)-x$ instead of $\mathrm{H}(\mathrm{x})$ directly

Notes:

• The idea is that sometimes doing nothing is actually better!
• You can see in this network the only thing we did is to copy the input feature maps to the output
• On one pass the input feature map will go through the convolutions and on the second pass it will also be passed directly to the output to be summed together.
• What does this mean?
  • Hard requirement:
    • You have two paths that have to produce exactly the same number of feature maps of the same size
    • This whole thing in general does not change the number and size of feature maps
    • Takeaway: same number and size of feature maps in input will be the same as the number and size of feature maps on the output
      • "Does not change the number of output feature maps"
• This is the straightforward case, there are more complex cases.

Full ResNet architecture:

• Stack residual blocks
• Every residual block has two 3x3 conv layers
• Periodically, double # of filters and downsample spatially using stride 2 (/2 in each dimension)
• Additional conv layer at the beginning
• No FC layers at the end (only FC 1000 to output classes)

Notes:

• Note the 7x7 conv at the start is not important here
• Note each two 3x3 blocks we have a resonate connection
  • Two layers of 3x3 convolution
• Note the notation 3x3 conv, 512, /2
  • Means the number of feature maps got divided by 2?
• After the final global pooling layer, the size of the feature maps become 1x1
• After this pooling layer, you will produce the vector x (a feture map of size 1x1) -> x will be the input for logistic regression.
• FC 1000 is our only fully connected layer
  • In each network we need to have at least one layer of fully connected layers
• Then we do softmax
• You can see for these layers (3x3 blocks), because we do padding, they do not change the number of feature maps
• The only tricky thing is the layers of different colors
  • Inside each of the same colors, ...
    • We basically need to do something between the different colors to make sure each color produce the same number of feature maps
  • The maroon one produces 64
  • The purple one has 128 as output but the size of the feature map is half the size while the number of feature maps is doubled
  • How to increase the number of feature maps? just add more filters
• Exam: Some operation needs to be done to reach that /2. How is it happening that the number of feature maps is doubled and the size of the feature maps is reduced by a factor of 2?
  • We basically use stride 2 1x1 conv(+BN), double the number of output feature maps
  • Note that there needs to be a batch normalization after it.
• Note after each convolutional layer there is batch normalization and there is relu after it:
  • The whole set of operations in the above block is:
    • Conv + BN + ReLU + conv + BN + addition + ReLU.
• What does it mean to do a 1x1 convolution with stride 2?
  • Kernel size is 1x1
  • Stride is 2
  • Basically you are skipping every other pixel
• The major difference is how many of these blocks you will use, there are networks of hundreds of layers:
  • Total depths 34, 50, 101, or 152 layers for ImageNet

"Bottleneck"

For deeper networks (ResNet-50+), use “bottleneck” layer to improve efficiency (similar to GoogLeNet)

Notes:

• Number of feature maps is reduced to 64 by using a 1x1 convolution, then you apply a 3x3 convolution while keeping the same number of feature maps.
• This entire path does not change the number of feature maps because you again apply the 1x1 convolution after the 3x3 output
• By doing this whole block, you will have a smaller number for parameters than by just applying a 3x3 convolution.
• This is called bottleneck because we go from smaller -> larger -> smaller number of feature maps.

Identity Mappings in Residual Learning

• Residual Learning: $h=$identity mapping

Notes:

• There is one more thing that is important

Identity Mappings in Deep Residual Networks

Improving ResNets...

• Improved ResNet block design from creators of ResNet
• Creates a more direct path for propagating information throughout network (moves activation to residual mapping pathway)
• Gives better performance

There is one more thing that is important:

• Sequence of operations (one block):
  • Conv + BN + ReLU + conv + BN + addition + ReLU.
• If you only follow this skip connection what ... is this block?
  • You still need to pad with ReLU every time.

Notes:

• If you look at a whole network there are some layers stacked on top of each other
  • You have many blocks on top of each other
• When you put a skip connection we need to start somewhere and end somewhere
• The discussion is where we start this skip connection and where we end it?
• Where exactly do we consider a certain sequence of layers a single block and end bound our skip connection based on that block?
• In this example if you follow the skip connections you will have ReLU at the end of each block
  • What if we shift the boundaries of the skip connection one layer upward (red line)
  • If you do that, the left image is what you would have
  • Now there is no ReLU in between each of the block, ReLU is now inside.
• There are no guidelines on why this is better than having ReLU at then end of a block but is an ongoing discussion and this is more used in practice
• This is just our view, in reality the network is a bunch of blocks on to of each other

Things to takeaway:

1. What is the skip operation (ResNet)
2. Bottleneck
3. Identity mapping

Checkpoint 12 (ResNet)

To understand ResNet (Residual Networks), we need to look at the biggest mystery that haunted deep learning before 2015: why do deeper networks perform worse?

1. Case Study: ResNet Intuitively, a 56-layer network should be much smarter than a 20-layer network. At the very least, it could just copy the 20 layers and do nothing for the remaining 36 layers. However, researchers found that 56-layer "plain" networks performed substantially worse on both training and test data. This was not caused by overfitting, but by an optimization problem: it is incredibly difficult for Gradient Descent to update weights effectively across 56 layers without the signal vanishing.

The solution was the Residual Block. Instead of forcing a layer to learn a completely new, complex mapping ($H(x)$), the network forces the layer to learn a "residual" or a difference ($F(x)=H(x)-x$).

• The Skip Connection: We physically take the input data $x$, route it around the convolutional layers, and add it back to the output at the very end.
• Why it works: If those convolutional layers are useless, the network simply learns to set their weights to $0$. Because the input $x$ is being added via the skip connection, $0+x=x$. The network easily learns an "identity mapping" (doing nothing), allowing the signal to flow perfectly through the deep network without degrading.

2. "Bottleneck" In extremely deep networks (like ResNet-50 or ResNet-152), applying $3\times 3$ convolutions across hundreds of feature maps becomes far too computationally expensive. To fix this, ResNet uses a "Bottleneck" design (borrowed from GoogLeNet). Instead of one large convolution, the block uses three steps:

1. A $1\times 1$ convolution to compress/reduce the number of feature maps (the bottleneck).
2. A standard $3\times 3$ convolution on this smaller, manageable amount of data.
3. Another $1\times 1$ convolution to expand the feature maps back to their original depth. This drastically reduces the number of parameters while maintaining the exact same network capability.

3. Identity Mappings in Residual Learning For the skip connection ($x$) and the main convolutional path ($F(x)$) to be added together ($F(x)+x$), there is a strict mathematical requirement: they must have the exact same spatial size (width/height) and depth (number of feature maps). If the main path uses a stride of 2 (which shrinks the image) or increases the number of filters, the matrix addition will crash because the dimensions no longer match.

4. Identity Mappings in Deep Residual Networks This refers to an ongoing debate and improvement by the creators of ResNet regarding where to put the activation functions (ReLU) and Batch Normalization (BN) inside the block.

• Original ResNet: The block ends with an addition, and then ReLU is applied after the addition.
• Improved (Pre-activation) ResNet: Researchers found that moving BN and ReLU inside the residual pathway (before the convolutions) creates a much cleaner, more direct skip connection path. By removing the ReLU at the very end, the skip connection becomes a pure identity mapping, which improves performance and gradient flow.

Question 1: What if shortcut mapping $h\ne \text{ identity }$?

(This is not in exam/not in homework)

What this paper propose is to make that skip connection along that identity.

Turns out that there is one person who did something similar but added more operations on every block (making it more complex), and it ended up not working.

• constant scaling: basically multiplying by certain number, making the error to scale too!
• The idea of gate is quite popular and you can get in in many ways:
  • Imagine 1 feature map as input, 1 feature map as output with just an applied convolution in between.
  • If you have some object in feature map A (obviously you want to recognize it), it is more important, we need to choose some feature map to capture the important features of the object.
  • The idea of gate is that you will have a side pass that will do some operations (most likely a convolution), it produces a feature map and applies sigmoid element-wise to every location in the new feature map on the side pass, so you get another map where everything is a number between 0-1. Then on the main path you do a regular convolution, and at the end you multiply element-wise the 0-1 feature map with the main path new feature map.
  • This is essentially doing a mask! It is very nice but in reality, now this wouldn't work, just because a mask is not smart enough why?
    • How can you learn the parameter before the side pass to capture the correct features of the object to recognize?
    • There is not enough information for you to know this.
    • Mask segmentation is a much difficult task in between.
    • You are not told what are the locations of the object to recognize in the original feature map, this is a difficult task and most times you will make mistakes on the locations you are interested on.
    • This is exactly what was done in that 1990 paper -> if you add a gate it won't work
    • But there are a tons of different gate.
• This is a debate, some people used this long time before what we used now.

Identity Mappings in Residual Learning