Model Evaluation | Logistic Regression Scoring | Vantage Analytics Library - Model Evaluation - Vantage Analytics Library

Prediction Success Table

The prediction success table is computed using only probabilities and not estimates based on a threshold value. Using an input table that contains known values for the dependent variable, the sum of the probability values π(x) and 1 – π(x), which correspond to the probability that the predicted value is 1 or 0 respectively, are calculated separately for rows with actual values of 1 and 0. This produces a report as shown in the following table:

An interesting and useful feature of this table is that it is independent of the threshold value that is used in scoring to determine which probabilities correspond to an estimate of 1 and 0, respectively. This is possible because the entries in the “Estimate Response” column are the sums of the probabilities π(x) that the outcome is 1, summed separately over the rows where the actual outcome is 1 and 0 and then totaled. Similarly, the entries in the “Estimate Non-Response” column are the sums of the probabilities 1 – π(x) that the outcome is 0.

Multi-Threshold Success Table

This table provides values similar to those in the prediction success table, but instead of summing probabilities, the estimated values based on a threshold value are summed instead. Rather than just one threshold, however, several thresholds ranging from a user-specified low to high value are displayed in user-specified increments. This allows the user to compare several success scenarios using different threshold values, to aid in the choice of an ideal threshold.

For example, consider the ideal threshold value is one that maximizes the number of correctly classified observations. However, subjective business considerations may be applied by looking at all of the success values. It may be that wrong predictions in one direction (for example, estimate 1 when the actual value is 0) is more tolerable than in the other direction (estimate 0 when the actual value is 1). One may, for example, mind less overlooking fraudulent behavior than wrongly accusing someone of fraud.

The following is an example of a logistic regression multi-threshold success table.

Cumulative Lift Table

The following is an example of a logistic regression cumulative lift table.