TD_SELECTION_CRITERIA computes a series of model selection metrics to assist a data scientist in selecting the best model.
Model selection is the process of choosing the best algorithm or model from a set of candidate models for a given dataset. It is done by comparing various model candidates on evaluation metrics calculated on an evaluation schema. Model selection is a critical step to determine the accuracy and effectiveness for predicting or classifying new data. The goal of model selection is to find a model that generalizes well on unseen data, rather than just fitting well to the training data.
The model selection metrics are:
- Akaike Information Criteria (AIC): Test how well your model fits the data set without over-fitting it. An AIC score is compared with the AIC score of a competing model. A model with a lower AIC score is expected to have balance between its ability to fit the data set and its ability to avoid over-fitting the data set.
- Schwarz Bayesian Information Criteria (SBIC): Quantify and select the least complex probability model among options. This approach ignores the prior probability and instead compares the efficiencies of different models at predicting outcomes. That efficiency is measured by an index of each model’s parameters using a likelihood function, and then applying a penalizing function for models with more parameters.
- Hannan Quinn Information Criteria (HQIC): Measure of the goodness-of-fit of a statistical model, and is often used as a criterion for model selection among a finite set of models. It is related to Akaike's information criterion. Like AIC, the HQIC has a penalty for the number of parameters in the model, but the HQIC penalty is larger than the AIC penalty.
- Maximum Likelihood Rule (MLR): Determine values for the model parameters. The parameter values are such that they maximize the likelihood that the process described by the model produced the data that was observed.
- Mean Squared Error (MSE): Measure the amount of error in statistical models. It assesses the average squared difference between the observed and predicted values. When a model has no error, the MSE equals zero. As model error increases, its MSE value increases. The MSE is also known as Mean Squared Deviation (MSD).