Comparing models with transformations

In the process of searching for a good model, a popular step is to try different transformations of the variables. This can become a bit tricky when we are transforming the response variable, Y.

Consider, for instance, two very simple models for predicting home sales. Let's assume that in both cases we use predictors such as the home's attributes, geographical location, market conditions, time of year, etc. The only difference is that the first model is linear:

(1) SalesPrice = bo + b1 X1 + ...

whereas the second model is exponential:

(2) SalesPrice = exp{c0 + c1 X1 + ...}

The exponential model can also be written as a linear model by taking a natural-log on both sides of the equation:

(2*) log(SalesPrice) = c0 + c1 X1 + ...

Now, let's compare models (1) and (2*). Let's assume that the goal is to achieve a good explanatory model of house prices for this population. Then, after fitting a regression model, we might look at measures such as the R-squared, the standard-error-of-estimate, or even the model residuals. HOWEVER, you will most likely find that model (2*) has a much lower error!

Why? This happens because we are comparing objects that are on two different scales. Model (1)yields errors in $ units (assuming that the original data are in $), whereas Model (2*) yields residuals in log($) units. A similar distortion will occur if we compare predictive accuracy using measures such as RMSE or MAPE. Standard software output will usually not warn you about this, especially if you created the transformed variable yourself.

So what to do? Compute the predictions of model (2*), then transform them back to the original units. In the above example, we'd take an exponent of the prediction to obtain a $-valued prediction. Then, compute residuals by comparing the re-scaled predictions to the actual y-values. These will be comprabale to a model with no transformation. You can even compare the re-scaled predictions with those from model (1) or any other model that has re-scaled predictions.

The unfortunate part is that you'll probably have to compute all the goodness-of-fit or predictive accuracy measures yourself, using the re-scaled residuals. But that's usually not too hard.