Linear Regression : Fit, Error, Tradeoff
Understand how linear regression fits a line, how mean squared error measures prediction errors, and how bias and variance shape model performance.
Imagine you have four students who studied for different numbers of hours. One studied for one hour and scored forty, another studied for two hours and scored fifty, another studied for three hours and scored sixty, and another studied for four hours and scored seventy. You can already see a pattern: more study tends to mean a higher score. But the real question is, how would you draw one line that captures that pattern? You could draw many possible lines through this cloud of points. One might pass above most of them, another below most of them, and another might run roughly through the middle. We want a rule that chooses the line that represents the relationship best. The key point is that we are not trying to connect every point perfectly. We are trying to capture the overall trend. Suppose our line predicts a score of forty-eight for two hours of study, sixty-one for three hours, and seventy-two for four hours. The predictions are close, but not identical to the observed scores. That difference is important. A machine learning model is useful precisely because it can make predictions for new inputs, not because it memorizes the old points. Now we can name the idea: linear regression. In the simplest case, it assumes the prediction changes at a constant rate as the input changes. We write the prediction as a line: y hat equals m times x plus b. The number m controls the slope, and b controls where the line crosses the vertical axis. Why not simply choose any reasonable slope and intercept? Because two different lines can both look reasonable, while one may make much larger prediction mistakes. So we need a way to judge the line using all the observed points. That brings us to the next question: how should we measure how wrong a line is? Pause here. Imagine two candidate lines. One is usually close but sometimes misses by a little more. The other is always moderately wrong. Which line should win? The answer depends on how we combine all of those individual mistakes into one score. We have our line, and now we need one number that tells us how well it fits. For each data point, compare the actual score with the predicted score. If the actual score is seventy and the line predicts seventy-two, the error is two points in magnitude. But notice a problem: some errors are positive and some are negative, so simply adding them could make them cancel each other. We can fix that by squaring every error. A mistake of two becomes four, while a mistake of five becomes twenty-five. The key point is that squaring makes larger mistakes count much more heavily. This is often useful because a model that is wildly wrong on a few cases may be much less desirable than one that is consistently close. Now imagine four squared errors: four, one, nine, and sixteen. Add them and you get thirty. But that number depends on how many data points we had. Ten points and one thousand points would not be directly comparable. So we divide by the number of observations. We have now arrived at mean squared error. Mean squared error is simply the average of the squared prediction errors. In spoken form: M S E equals the sum of squared errors divided by the number of observations. It gives us one score for the whole line. Smaller means the predictions are, on average, closer to the observed values. Let us compare two models. Model A has errors of one, two, and three. Model B has errors of zero, zero, and six. Model A looks consistently reasonable, while Model B has two perfect predictions and one terrible one. Squaring reveals the difference. Model A has squared errors one, four, and nine. Model B has zero, zero, and thirty-six. Which model has lower mean squared error? You can calculate it: Model A totals fourteen, so its mean squared error is fourteen divided by three. Model B totals thirty-six, so its mean squared error is thirty-six divided by three. Model A wins. But now another question appears: can we keep making a model more flexible until its error becomes almost zero? We just used mean squared error to reward accurate predictions. Now imagine three models trying to predict the same data. The first is a simple straight line. The second is a smooth curve. The third bends sharply to pass almost exactly through every training point. On the training data, the third model can look spectacular. But that is where the trap begins. Suppose we collect a second set of points that the model has never seen. The rigid straight line may miss an important pattern and make systematic errors. The highly flexible model may also fail, because it learned the accidental quirks of the original sample. The middle model may generalize best. The key point is that lower training error does not automatically mean better performance on new data. The simple model has high bias. Its assumptions are too restrictive, so it misses real structure in the data. The very flexible model has high variance. It reacts strongly to the particular training sample, including noise and unusual points. A useful mental picture is a target: bias is repeatedly missing in the same direction, while variance is scattering widely from one training sample to another. This gives us the bias variance tradeoff. As model complexity increases, bias often decreases because the model can capture more patterns. But variance often increases because the model becomes more sensitive to the details of the training data. We are balancing two kinds of error rather than trying to make either one disappear completely. Now connect all three ideas. Linear regression gives us a model that predicts with a line. Mean squared error gives us a way to quantify how close those predictions are to observed values. The bias variance tradeoff reminds us that the best model is not necessarily the one that fits the training data most tightly. Pause and answer this: if a model performs badly on both training and test data, is it more likely to have high bias or high variance? High bias. If training error is tiny but test error is much larger, which one is more likely? High variance. The moral is simple: in machine learning, you are not just fitting the data you have. You are trying to learn a pattern that survives data you have not seen yet.