Regression

A flat-rental platform in Bengaluru wants to show owners a suggested monthly rent the moment they list, and to flag listings priced far above similar flats. Each past listing has its area (sq ft), bedrooms, bathrooms, building age, floor, distance to the nearest metro station, locality, furnishing, who listed it (owner, agent or builder), parking, the date it was listed, and the rent it actually went for. The target is that rent, in thousands of rupees a month.

This is a regression problem — the answer is a number, not a category — and the platform needs more than a number: owners ask *why* the suggestion is ₹42,000 and not ₹50,000. A linear model answers that directly, one feature at a time, which is why it is the first model to try and the one every fancier model has to beat.

The model predicts rent_hat = w1*area + w2*bathrooms + ... + wd*xd + b: a weight for every feature plus an intercept b. Fitting chooses the weights that make the mean squared error smallest — the average of the squared vertical gaps between each listing's real rent and the model's prediction.

It is fast, gives the same answer every time, and every weight can be read in plain words. With one feature the model is a line through the points; with d features it is a flat surface (a hyperplane) in d + 1 dimensions, and fitting is finding the surface closest to all the points at once.

Each weight is the change in the prediction when that one feature goes up by one unit and every other feature stays the same. In the rent model, area_sqft has weight 0.018: each extra square foot adds ₹18 a month. building_age has −0.305: each year older takes off ₹305. Furnished has 6.85: a fully furnished flat rents for ₹6,850 more than an unfurnished one like it.

This is what makes linear regression explainable to a customer: the suggested rent can be broken into '₹18 × 1,100 sq ft, minus ₹305 × 12 years…' The sign says the direction and the size says how much, per unit.

Standardise every feature first (subtract its mean, divide by its standard deviation), then refit. Now each weight is 'the change in rent for a one-standard-deviation change in this feature', every feature is on the same footing, and the size of the weight (its absolute value) ranks importance.

On raw features the ranking is an accident of units. In the rent model, raw weights put area_sqft last of 12 — yet a typical difference in area (one standard deviation, 346 sq ft) moves rent by ₹6,300, more than almost anything else. After scaling, area ranks second, behind locality.

A way to find the lowest point of a function by walking downhill. Take f(x) = x².

1. 1Step 1: Start anywhere, say x = 3.
2. 2Step 2: Measure the slope there: f'(3) = 6, positive, so the function rises to the right.
3. 3Step 3: Step the other way: x_new = x − η·f'(x), where η (eta) is the learning rate. With η = 0.1, x goes 3 → 2.4 → 1.92 → … towards the minimum at 0.

Fitting linear regression means finding the weights where the loss is lowest, and the loss is a bowl-shaped function of the weights. Gradient descent is how that bowl is searched — and the same idea trains logistic regression, neural networks and gradient-boosted trees.

Both measure how wrong the predictions are. MAE, the mean absolute error, averages |y − y_hat|. MSE averages (y − y_hat)². For fitting with gradient descent, MSE wins: its slope is 2·(y_hat − y), which is large when a prediction is far off and shrinks to zero as it gets close. MAE's slope is only ever +1 or −1, and at exactly zero error it has no slope at all (a sharp corner).

A gradient that shrinks as you approach the answer lets gradient descent slow down and settle. MAE's constant ±1 keeps taking the same size step and jitters around the minimum. MSE is also smooth everywhere, so the calculus works and, for linear regression, there is even an exact solution.

For one listing the loss is L = (y − (w1x1 + w2x2 + b))². Differentiate with the chain rule: ∂L/∂b = −2(y − y_hat) and ∂L/∂w1 = −2(y − y_hat)·x1. The pattern is always −2 × error × the feature that weight multiplies. Over m listings, average it: ∂L/∂wj = (1/m) Σ −2(y − y_hat)·xj.

It turns an abstract idea into two lines of arithmetic. A weight moves a lot when its feature is large on the rows the model gets most wrong, and not at all when the error is zero. That is also why the gradient for w10 is −2(y − y_hat)·x10, not ·w10.

Every weight is updated together: wj = wj − α·∂L/∂wj and b = b − α·∂L/∂b, where α is the learning rate — how big a step to take. Too small (a walk) and it takes forever; right (a jog) and it settles quickly; too large (a run) and every step overshoots the bottom, so the loss grows instead of shrinking.

It is the one setting that most often decides whether training works. On the standardised rent data, α = 0.1 is within a whisker of the best possible MSE (15.2) after 25 steps, α = 0.01 is still at 42.8 after 100, and α = 0.5 blows up — the loss triples every step.

Four small functions: predict returns X·W + b; score returns R²; update_weights computes the gradients for all the weights at once and takes one step; fit starts the weights at zero and repeats the update for a fixed number of iterations.

Writing it once shows there is no magic in LinearRegression: on the rent data this class reaches R² = 0.899, the same as scikit-learn. It also shows why the matrix shapes work: X is (m rows, d features), the error is (m,), so X.T @ error gives one number per weight, (d,).

When features have very different ranges — area in the hundreds or thousands, bathrooms between 1 and 4 — the loss surface is a long, narrow valley. A learning rate small enough not to blow up along the steep direction is far too small along the gentle one, so gradient descent zig-zags and crawls. Scaled features make the valley round, and each step points more directly at the bottom.

On the rent data the difference is stark. Unscaled, a learning rate of 0.000001 already blows up, and 0.0000001 leaves the MSE stuck near 93 after 300 steps; scaled, a learning rate of 0.1 reaches 15.25 — essentially the best possible, 15.24. Same data, same model; only the scaling changed.

For squared error the optimum has an exact solution, the normal equation w = (XᵀX)⁻¹Xᵀy, which scikit-learn computes via a stable least-squares routine rather than an explicit inverse. SGDRegressor solves the same problem iteratively.

The closed form is exact and needs no learning rate, but its cost grows with the cube of the feature count and it needs the whole matrix in memory. Knowing where the crossover lies tells you when to stop reaching for LinearRegression.

R² compares the model with the dumbest possible one — always predicting the average rent: R² = 1 − SS_res / SS_tot, where SS_res is the sum of squared errors of the model and SS_tot that of the average. 1 means perfect, 0 means no better than the average, and on new data it can even be negative.

Add any feature to the training data — even a column of random numbers — and training R² can only go up or stay the same, because the fit can always set that feature's weight to zero and lose nothing. By chance the noise lines up a little with the rent, so R² creeps up. It rewards piling on features whether they help or not.

Adjusted R² = 1 − (1 − R²)·(m − 1)/(m − d − 1), with m rows and d features. Adding a feature raises d, which makes the fraction bigger and pulls the score down; the feature has to raise R² by enough to pay for itself. It is at most 1 and can go below 0 (when the model is worse than the average).

It is a quick honest check when deciding whether a feature is worth keeping. With 50 listings and 3 features, R² = 0.850 gives adjusted R² = 0.840. Adding a fourth, useless feature nudges R² to 0.852 — but adjusted R² falls to 0.839. The extra feature did not earn its place.

statsmodels fits the same ordinary least squares (OLS) model as scikit-learn but reports far more: for every weight its standard error, a t-statistic, a p-value and a 95% confidence interval, plus R², adjusted R², a condition number and a Durbin–Watson statistic. Unlike scikit-learn it does not add an intercept on its own — sm.add_constant does that.

scikit-learn is built for predicting; statsmodels is built for explaining. When the platform needs to say 'a typical step up in area (one standard deviation, about 350 sq ft) adds between ₹6,090 and ₹6,370 a month, and we are confident it is not zero', that comes from the summary. The warnings at the bottom also catch problems — a huge condition number means multicollinearity.

Linear regression is the best linear model when five things hold:

1. 1Step 1: the relationship between the features and the target is linear;
2. 2Step 2: no feature is (nearly) a combination of the others;
3. 3Step 3: the errors are roughly normally distributed;
4. 4Step 4: their spread is the same everywhere (no heteroskedasticity);
5. 5Step 5: they are independent of each other (no autocorrelation). Each has its own check, in the cards that follow.

Prediction survives mild violations; explanation does not. A p-value or confidence interval computed when these fail is wrong by an unknown amount — which matters the moment someone quotes a weight in a business decision.

The model can only draw straight-line relationships: every extra square foot adds the same ₹18, whether the flat is 500 or 2,500 sq ft. If the real effect curves — rent rising steeply near the metro and flattening beyond 5 km, say — a straight line is wrong in a predictable way: too high in some ranges and too low in others.

It is the one assumption whose failure hurts the predictions themselves. You see it by plotting the residuals against a feature: a healthy plot is a shapeless band around zero; a U or an arch says the relationship bends.

Two features are collinear when one is a straight-line function of the other: f1 = a·f2 + b. Multicollinearity is the same with several: one feature can be written as a combination of others. In rent data it is everywhere: building_age is exactly 2026 − year_built, bedrooms is largely predicted by area_sqft, and a listing's super built-up area is its carpet area plus a near-fixed 25% share of walls, stairs and lobby.

The predictions survive, but the weights do not. Suppose rent = 1·x1 + 2·x2 + 3·x3 + 5 and x2 is always 1.5·x1. Then 4·x1 + 0·x2 + 3·x3 + 5 gives exactly the same predictions — so do endless other combinations — and the data cannot choose. The weights become arbitrary, and feature importance means nothing.

The variance inflation factor of a feature measures how well the *other* features predict it. Treat that feature as the target, fit a linear regression on the rest, take its R² (call it R²ⱼ), and compute VIF = 1 / (1 − R²ⱼ). If the others explain nothing, R²ⱼ = 0 and VIF = 1. If they explain it perfectly, R²ⱼ = 1 and VIF is infinite.

It gives a number to act on. Rule of thumb: above 10 very high — drop it; 5 to 10 high; below 5 fine. Then repeat: drop the feature with the highest VIF, refit, recompute, until every VIF is under 5 — or until adjusted R² starts to fall, which means you are now removing real information.

The residuals — real rent minus predicted rent — should look like a bell curve centred on zero. Check with a histogram, a Q-Q plot, or the Shapiro–Wilk test (a statistic W near 1 means close to normal). A long tail on one side, or a few residuals far from the rest, usually means outliers: rows the model gets wildly wrong.

Squared error lets outliers drag the fit towards them. In the rent data, 25 listings had an extra zero typed into the rent (₹4,00,000 instead of ₹40,000). Their residuals ran from 150 to 730, while 99.9% of the other rows sat within ±15. Flagging residuals more than 5 standard deviations out caught 18 rows in the training set — all of them typos — and W went from 0.14 to 0.995.

The spread of the errors should be the same everywhere — the model should be about as wrong on a ₹20,000 flat as on a ₹60,000 one. Heteroskedasticity is when it is not: plot the residuals against the predictions and the band widens into a funnel.

The weights stay reasonable, but the standard errors, p-values and confidence intervals in the statsmodels summary assume one spread for all rows, and become wrong. It also often signals that effects are proportional — a furnished flat adds a *percentage*, not a fixed amount — which modelling log(rent) captures.

Each row's error should be unrelated to the next row's. It fails when rows have an order — usually time — and something that changes over time is missing from the model. The Durbin–Watson statistic summarises it: about 2 means no autocorrelation; well below 2, errors next to each other tend to have the same sign.

Bengaluru rents rose through 2025, but the model has no listing-date feature. Sorted by listing month, its residuals drift from −₹2,200 in January to +₹1,700 in December: it underprices recent listings and overprices old ones. Adding the month as a feature removes the drift. Autocorrelated errors also make the model far too confident — its intervals assume more independent information than the data has.

Adding x**2, x**3 … as extra columns keeps the model linear in its parameters while letting it curve. The polynomial degree is a capacity dial: degree 1 is a straight line, degree 9 can pass through ten points exactly.

It is the cleanest demonstration of the capacity tradeoff in the whole syllabus. Nothing changes but one integer, and the model moves from too rigid to capture the shape to flexible enough to interpolate the noise.

Adds alpha * sum(w**2) to the squared-error cost. Large coefficients now carry a price, so the fit trades a little training error for smaller weights. Coefficients shrink smoothly toward zero as alpha rises but never reach it.

It is the standard remedy for collinearity and for the high-variance regime where the feature count approaches the sample count. It also keeps every feature in the model, which matters when the features are genuinely all weakly informative.

Adds alpha * sum(abs(w)) instead. The absolute-value penalty has a corner at zero, so the optimum lands exactly on zero for weak features: Lasso performs selection as well as shrinkage.

When you believe only a handful of features matter, Lasso returns a sparse model you can read and deploy cheaply. The set of features it keeps, as alpha varies, is also a genuinely useful ranking.