TD_OutlierFilterFit function calculates the lower percentile, upper percentile, count of rows, and median for the specified input table columns. The calculated values for each column help the TD_OutlierFilterTransform function detect outliers in the input table.
An outlier is a data point that significantly differs from other observations in a dataset. These observations may occur due to a variety of reasons such as measurement errors, natural variations, or even rare events that are not captured by the underlying model. In machine learning, outliers can have a significant impact on the performance of models, leading to inaccurate predictions and skewed results.
Outlier filtering is a technique used in machine learning to identify and remove outliers from the dataset before training a model. The process involves analyzing the distribution of data points and identifying the ones that deviate significantly from the expected behavior.
One way to identify outliers is by using percentile-based methods such as Tukey and Carling methods. The percentile-based methods rely on the interquartile range (IQR), which is the range between the 25th and 75th percentiles of the data. The Tukey method considers any data point outside the range of 1.5 times the IQR to be an outlier. The Carling method considers any data point outside the range of 3 times the IQR to be an outlier.
- Removing them: Remove the outliers from the dataset completely. However, this approach can result in loss of information, particularly if the outliers are genuine data points that carry important information.
- Replacing them: Replace the outliers with the mean or median of the dataset. This approach can work well if the outliers are due to measurement errors or other anomalies that are unlikely to occur again.
- Binning them: Group the outliers into a new category or bin, allowing them to be treated as a separate entity in the analysis.
- Transforming them: Transform the data using mathematical functions such as logarithm or square root, which can normalize the data and reduce the impact of outliers.