Keeping the Right Signals: L1 and L2 Regularization
Understand how L1 and L2 regularization penalize neural network weights to reduce overfitting and improve predictions on new songs.
Imagine building a music app that guesses how much you will like a song. You give songs ratings from zero to ten. The app gets information about each song, such as its speed, bass strength, and amount of singing. Its job is to use that information to predict your rating for a song you have not heard yet. To learn, the app studies songs you have already rated. These are its training examples. Each example contains the song information and your actual rating. Imagine the app predicts eight for a song you rated six. Its prediction is two points too high. Learning means changing the app so that its future predictions tend to get closer to your ratings. But there is a sneaky problem. Suppose most songs you rated highly happened to have titles beginning with the letter B. The app might start treating that letter as an important clue. That could help it match your old ratings without helping it understand your taste. Matching examples is not always the same thing as learning a useful pattern. When a model learns details of its training examples that do not carry over well to new examples, we call that overfitting. A model is the adjustable system making the predictions. Our goal is not just a model that remembers your old playlist. We want one that handles new songs well. That ability is called generalization. One kind of model is a neural network. Picture several rows of small processing units. Information about a song enters the first row, travels through later rows, and eventually becomes a predicted rating. Each row is called a layer. A deep neural network has multiple layers between its incoming information and its answer. Deep describes those layers, not human-like understanding. A processing unit receives numbers, adjusts their influence, combines them, and passes a result onward. It also uses a rule that can change how it responds to that combined number. Those rules help the network learn relationships that simple addition alone could not capture. For example, strong bass might matter differently in a slow song than in a fast one. The adjustable influence on a connection is called a weight. Imagine one unit receives a bass measurement of two. A weight of three makes that connection contribute six, because two times three is six. A weight of one makes it contribute two. A weight of zero makes that connection contribute nothing, no matter what bass measurement arrives. Weights can also be negative. With the same bass measurement of two, a weight of negative three contributes negative six. That contribution pulls the unit's combined number downward instead of upward. A weight's size means its distance from zero, regardless of direction. Three and negative three have the same size, even though their effects point in opposite directions. In a deep network, one connection does not usually translate directly into one clear preference, such as loving bass. Its effect depends on the other connections and processing rules around it. Many connections work together. Still, adjusting their weights changes the prediction. Training repeatedly makes those adjustments, using the mistakes on the training songs as guidance. To guide those adjustments, we need a number that measures prediction mistakes. That number is called a loss. A smaller loss means better predictions on the examples being measured. There are different ways to calculate it. For our ratings, we will square each prediction error, which means multiplying that error by itself, and then average the results. Suppose two songs have actual ratings of six and seven. The app predicts eight and six. For the first song, prediction minus actual rating is two. For the second, it is negative one. Squaring gives four and one, both positive. Add them to get five, then divide by two songs. The average loss is two point five. Squaring prevents positive and negative errors from canceling each other. It also makes a large error count more strongly than a small one. During training, a method called an optimizer uses information about how weight changes affect the loss. It repeatedly nudges the weights toward values that reduce that score. Think of many small adjustments rather than one perfect guess. If training cares only about mistakes on old songs, the network may build an elaborate collection of unreliable clues. We can change the goal: make good predictions, but also pay a cost for certain weight choices. Adding this extra pressure is called regularization. It does not tell the network which song clues are true. It encourages solutions that are less dependent on costly weights. The extra cost is called a penalty. During training, the optimizer now tries to reduce prediction loss plus a weighted penalty. The penalty is not a punishment for a wrong rating. It measures something about the network's weights. This creates a trade-off: a weight must help predictions enough to justify the extra cost it adds. We control the strength of that trade-off with a number usually called lambda. Lambda multiplies the penalty before we add it to prediction loss. If lambda is zero, this penalty has no effect. A small positive lambda adds gentle pressure. A larger lambda makes weight size more expensive. Lambda is a setting we choose, not a song rating. Before comparing weights, we should make input measurements reasonably comparable. Song speed might be measured in hundreds of beats per minute, while bass strength might run from zero to one. Those different measuring scales can require very different weights. Rescaling the inputs helps prevent the penalty from favoring a clue merely because of the units used to measure it. The two methods we are studying differ in how they calculate the penalty. L1, spoken as L one, adds the distances of weights from zero. L2, spoken as L two, adds their squares. We will examine a tiny group of weights so the arithmetic stays clear. In a real network, the same idea can apply across many layers and thousands or millions of connections. For L1, we first take each weight's absolute value. Absolute value means its distance from zero, so negative three becomes three, and positive three stays three. Then we add those distances. Suppose two weights are three and negative one. Their L1 penalty is three plus one, which equals four. Both directions cost something; opposite signs do not cancel. Now suppose that network's prediction loss is two, and lambda is one half. The penalty contribution is one half times four, which equals two. Add that to the prediction loss of two, and the total training score is four. The optimizer tries to reduce this combined score, not just the prediction mistakes. That is how weight size enters the training decision. L1 charges the same extra amount for each extra step away from zero. Increasing a weight's size from one to two adds one to the unscaled penalty. Increasing it from four to five also adds one. So shrinking either weight by one saves the same penalty amount. The prediction loss still matters: shrinking a useful weight could make predictions worse. This steady pressure can push weakly useful weights all the way to zero. Picture a connection that barely helps the rating predictions. Keeping it active costs something, while removing it hurts very little. Zero can therefore become the best trade-off. Certain optimizers can produce exact zeros; ordinary small-step training may instead leave some weights merely very close to zero. A network with many zero weights is called sparse. Imagine some paths carrying the song information have gone silent, leaving fewer active connections. L1 often encourages this result. But one silent connection does not necessarily erase an entire song clue. In a deep network, bass information might still travel along other paths. Removing whole inputs requires more than just noticing a few zero weights. L2 calculates its penalty by squaring each weight and adding the results. Using our weights of three and negative one, three squared is nine. Negative one times negative one is one. The L2 penalty is therefore ten. Squaring makes both contributions positive, while making the larger weight contribute much more strongly to the cost. Keep the prediction loss at two and lambda at one half. Half of the L2 penalty of ten is five. Adding the prediction loss gives a total training score of seven. The L1 score was four, but that does not mean L1 is automatically better. Their penalties have different scales, so we usually choose a suitable lambda separately for each method. Notice how L2 changes the cost of another step. A weight of one has a squared cost of one. A weight of two costs four, an increase of three. A weight of three costs nine, an increase of five. Steps farther from zero cost more. This gives L2 stronger pressure against large weights than against small ones. Looking only at the penalty's influence, L2 tends to pull a large weight toward zero more strongly than a small weight. As the weight gets smaller, that pull weakens. This usually leads to smaller weights rather than many exact zeros. Prediction loss can push in the other direction, so a useful weight may settle at a nonzero value where those pressures balance. Here is a comparison that shows the difference clearly. Imagine two weights of four and zero, or two weights of two and two. Both pairs have an L1 penalty of four. But their L2 penalties are sixteen and eight. If both pairs gave equally good predictions, L2 would prefer two and two. It can favor spreading influence instead of concentrating it in one large weight. Return to the accidental clue about song titles beginning with B. Suppose using that clue slightly improves predictions on the training songs. Without a penalty, even that small improvement is worth chasing. With a penalty, the connection must earn its cost. The network may decide that the tiny training improvement is not worth keeping a strong dependence on the title letter. This is why regularization can improve predictions on new songs even while making predictions on old songs slightly worse. We accept a little less perfect matching in exchange for less reliance on fragile details. But smaller weights are not proof of better understanding. Useful patterns can require strong connections, and accidental clues can still survive. Regularization changes the odds, not the rules of reality. Too much pressure causes a different problem. If lambda is very large, the network may shrink useful connections so much that it misses your real preferences. It might give nearly the same rating to every song. Failing to learn enough of the useful pattern is called underfitting. We want neither memorizing the playlist nor ignoring what makes its songs different. To choose the strength, we keep some rated songs out of training. These form a validation set: examples used to compare choices such as different lambda values. We train each version on the training songs and check its prediction loss on the validation songs. For this comparison, we examine prediction mistakes without adding the weight penalty, because new-song accuracy is what we care about. After making those choices, a separate test set gives a final check on songs not used for training or choosing settings. Keeping that final set separate matters. If we repeatedly change the model to improve its score on those same songs, they start guiding our choices too. Then they no longer provide such a fresh picture of performance on unfamiliar music. L1 is useful when we want pressure toward fewer active connections. L2 is useful when we want to discourage especially large weights while keeping many connections available. Neither method always wins. We can even combine their penalties to encourage both effects. The choice depends on the task, the network, and what actually works on songs held aside for validation. In deep networks, these penalties usually cover selected weight groups across multiple layers, not necessarily every adjustable number. A network can also have adjustable offsets called biases, which shift a unit's combined number without multiplying an incoming signal. Biases are often left out of these penalties. What matters is checking which parts of the network the regularization actually affects. Hold on to the central picture. Training asks the network to predict your ratings well. Regularization adds a price for the weights it uses to do that. L1 prices distance from zero and can silence connections. L2 prices squared distance and especially discourages large weights. The aim is not a quieter network for its own sake. It is a network whose rhythm still works when a new song starts.