| | |
- AdaptiveHistogram(data, columns=None, bins=10, exclude_columns=None, spike_threshold=10, subdivision_method='means', subdivision_threshold=30, filter=None, gen_sql_only=False)
- DESCRIPTION:
Histogram analysis reveals the distribution of continuous numeric or date values in a
column. Histogram analysis is also referred to as binning because it counts the
occurrence of values in a series of numeric ranges called bins. Adaptive Histogram
analysis supplements Histogram analysis by offering options to further subdivide the
distribution. You can specify the frequency percentage above which a single value
should be treated as a Spike, and a similar percentage above which a bin should be
subdivided. The Adaptive Histogram analysis modifies the computed equal-sized bins to
include a separate bin for each spike value, and to further subdivide a bin with an
excessive number of values, returning counts and boundaries for each resulting bin.
The subdivision of a bin with too many values is performed by first dividing the bin
by the same originally requested number of bins, and then merging this with a subdivision
in the region around the mean value within the bin, the region being the mean +/- the
standard deviation of the values in the bin. Subdividing can optionally be performed
using quantiles, giving approximately equally distributed sub-bins. (The quantiles
option is not recommended for teradataml DataFrames with extremely large numbers of
rows, such as input DataFrame with billions of rows.)
Adaptive binning can reduce the number of function calls needed to understand the
distribution of the values assumed by a column. Without adaptive binning, spike values
and/or overpopulated bins can distort the bin counts, as they are not separated or
subdivided unless the Adaptive Histogram function is used. It should be noted that
adaptive binning does not offer many of the specialized options that the standard
Histogram analysis does.
PARAMETERS:
data:
Required Argument.
Specifies the input data to perform adaptive histogram analysis.
Types: teradataml DataFrame
columns:
Required Argument.
Specifies the name(s) of the column(s) to analyze. Occasionally, it can also
accept permitted strings to specify all columns, or all numeric columns,
or all numeric and date columns.
Permitted Values:
* Name(s) of the column(s) in "data".
* Pre-defined strings:
* 'all' - all columns
* 'allnumeric' - all numeric columns
* 'allnumericanddate' - all numeric and date columns
Types: str OR list of Strings (str)
bins:
Optional Argument.
Specifies the number of equal width bins to create.
If multiple columns are requested, multiple bin sizes may be specified, such as
bins = [5, 10]. If fewer sizes are specified than columns, leftover columns are
associated with the default size of 10 bins.
Default Value: 10
Types: int OR list of Integers (int)
exclude_columns:
Optional Argument.
Specifies the name(s) of the column(s) to exclude from the analysis, if a column
specifier such as 'all', 'allnumeric', 'allnumericanddate' is used in the "columns"
argument.
Types: str OR list of Strings (str)
spike_threshold:
Optional Argument.
Specifies a percentage of rows, expressed as an integer from 1 to 100, above which
an individual value of a column is identified as a separate bin. Values with this
or a larger percentage of rows are identified as a Spike.
Default Value: 10 (10% of the total number of rows)
Types: int
subdivision_method:
Optional Argument.
Specifies the option to subdivide the bins.
Permitted Values:
* 'means' - Subdivide bins with too many values using a range of +/- the
standard deviation around the mean value in the bin.
* 'quantiles' - Subdivide bins with too many values using quantiles, giving
approximately equally distributed bins.
Default Value: 'means'
Types: str
subdivision_threshold:
Optional Argument.
Specifies a percentage of rows, expressed as an integer from 0 to 100, above which
a bin is subdivided into sub-bins. Values with this or a larger percentage of rows
are subdivided into sub-bins.
Default Value: 30 (30% of the total number of rows)
Types: int
filter:
Optional Argument.
Specifies the clause to filter rows selected for analysis within Adaptive Histogram.
For example,
filter = "cust_id > 0"
Types: str
gen_sql_only:
Optional Argument.
Specifies whether to generate only SQL for the function.
When set to True, function SQL is generated, not executed, which can be accessed
using show_query() method, otherwise SQL is just executed but not returned.
Default Value: False
Types: bool
RETURNS:
An instance of AdaptiveHistogram.
Output teradataml DataFrames can be accessed using attribute references, such as
AdaptiveHistogramObj.<attribute_name>.
Output teradataml DataFrame attribute name is: result.
RAISES:
TeradataMlException, TypeError, ValueError
EXAMPLES:
# Notes:
# 1. To execute Vantage Analytic Library functions,
# a. import "valib" object from teradataml.
# b. set 'configure.val_install_location' to the database name where Vantage
# analytic library functions are installed.
# 2. Datasets used in these examples can be loaded using Vantage Analytic Library
# installer.
# Import valib object from teradataml to execute this function.
from teradataml import valib
# Set the 'configure.val_install_location' variable.
from teradataml import configure
configure.val_install_location = "SYSLIB"
# Create required teradataml DataFrame.
df = DataFrame("customer")
print(df)
# Example 1: Shows execution with 10 equal width bins created by default for the
# 'income' column.
# The default value of 10 is used for "spike_threshold", and 30 for
# "subdivision_threshold", with means as the default "subdivision_method".
obj = valib.AdaptiveHistogram(data=df, columns="income")
# Print the results.
print(obj.result)
# Example 2: Shows execution on two columns with different numbers of bins, 5 for
# 'income' and 3 for 'age'. Threshold values and other parameters assume
# default values.
obj = valib.AdaptiveHistogram(data=df,
columns=["income", "age"],
bins=[5,3])
# Print the results.
print(obj.result)
# Example 3: Shows execution analysis done on two columns 'income' and 'age' with 5 bins
# for 'income' and default for 'age'. The non-default values of 12 is used for
# "spike_threshold", and 24 for "subdivision_threshold", with 'quantiles' as
# the default "subdivision_method".
obj = valib.AdaptiveHistogram(data=df,
columns=["income", "age"],
bins=5,
spike_threshold=12,
subdivision_method="quantiles",
subdivision_threshold=24,
filter="cust_id > 0")
# Print the results.
print(obj.result)
# Example 4: Generate only SQL for the function, but do not execute the same.
obj = valib.AdaptiveHistogram(data=df,
columns=["income", "age"],
bins=[5,3],
gen_sql_only=True)
# Print the generated SQL.
print(obj.show_query("sql"))
# Print both generated SQL and stored procedure call.
print(obj.show_query("both"))
# Print the stored procedure call.
print(obj.show_query())
print(obj.show_query("sp"))
|