Correlation and R-Squared

What is R2? In the context of predictive models (usually linear regression), where y is the true outcome, and f is the model’s prediction, the definition that I see most often is: In words, R2 is a measure of how much of the variance in y is explained by the model, f. Under “general conditions”, as Wikipedia says, R2 is also the square of the correlation (correlation written as a “p” or “rho”) between the actual and predicted outcomes: I prefer the “squared correlation” definition, as it gets more directly at what is usually my primary concern: prediction. If R2 … Continue reading Correlation and R-Squared

Problems That May Occur in Time Series Multiple Regression

Multicollinearity. If one independent variable is excessively linearly correlated with another independent variable, then it will be impossible to determine their separate influences.  The problem is with the data and not with the regression model itself and will be signified three schemes: (1) a high R^2 with low values for the t statistics, (2)  high values for simple correlation coefficients between the independent variables, and (3)  regression coefficients that are sensitive to model specification when both variables are included.  A variable may even take the wrong sign.   Multicollinearity may not be a problem in forecasting but it will be … Continue reading Problems That May Occur in Time Series Multiple Regression

What is the difference between L1 and L2 regularization?

From Quora Justin Solomon has a great answer on the difference between L1 and L2 norms and the implications for regularization. ℓ1 vs ℓ2 for signal estimation: Here is what a signal that is sparse or approximately sparse i.e. that belongs to the ell-1 ball looks like. It becomes extremely unlikely that an ℓ2 penalty can recover a sparse signal since very few solutions of such a cost function are truly sparse. ℓ1 penalties on the other hand are great for recovering truly sparse signals, as they are computationally tractable but still capable of recovering the exact sparse solution. ℓ2 … Continue reading What is the difference between L1 and L2 regularization?

A Complete Tutorial on Ridge and Lasso Regression in Python

AARSHAY JAIN , JANUARY 28, 2016 / 39 Introduction When we talk about Regression, we often end up discussing Linear and Logistics Regression. But, that’s not the end. Do you know there are 7 types of Regressions ? Linear and logistic regression is just the most loved members from the family of regressions.  Last week, I saw a recorded talk at NYC Data Science Academy fromOwen Zhang, current Kaggle rank 3  and Chief Product Officer at DataRobot. He said, ‘if you are using regression without regularization, you have to be very special!’. I hope you get what a person of his stature referred to. … Continue reading A Complete Tutorial on Ridge and Lasso Regression in Python

Correlation between A, B and C

A strong correlation between A and B. Strong correlation between B and C. No correlation between A and C. Is this possible? Suppose that the correlation ρABρAB between AA and BB and the correlation ρBCρBC between BB andCC are both the same number ρρ. Then ρACρAC lies between 2ρ2−12ρ2−1 and 11, the light blue shaded region below. If they’re fully correlated, ρ=1ρ=1, then, of course, ρACρAC is also 1.  But as ρρ decreases from 1, the minimum value for ρACρAC decreases four times as fast.  If ρ=0.9ρ=0.9, then  ρACρAC could be almost as low as 0.60.6.  With ρ=0.7ρ=0.7, AA and … Continue reading Correlation between A, B and C