from Hacker News

Why do polynomials have a bad reputation for overfitting?

by andrew_v4 on 4/21/21, 9:37 AM with 2 comments

• by anothernewdude on 4/21/21, 9:44 AM

  Because the coefficients of the larger powers are more sensitive, and most attempts to fit those models don't account for that, penalising high weights for those values more. You could have a regularisation regime that accounts for that. It's also why extrapolation fails on them so often.

  x^2 grows fast, x^3 even faster. They outpace the other parts of the model. Any changes to those coefficients also means you'll need to change the coefficients of the smaller powers to adjust to the change.

  A small amount of error in the largest power means two things - one the error will be exaggerated, because the power has an oversized effect, two the values of the smaller powers will also be wrongly adjusted to account for that error, and will contribute that poor fit to the model as well.

• by Bostonian on 4/21/21, 10:26 AM

  Cubic splines, which are piecewise cubic polynomials, have been widely used by statisticians for nonparametric regression. "Natural" splines are constrained to be linear in the tails, which reduces the danger of extrapolation.