EXPLAIN Confidence Levels | Teradata Vantage - EXPLAIN Confidence Levels

EXPLAIN Confidence Levels | Teradata Vantage - EXPLAIN Confidence Levels - Teradata Vantage

Teradata® VantageCloud Lake

Deployment

VantageCloud

Edition

Lake

Product

Teradata Vantage

Published

January 2023

Language

English (United States)

Last Update

2024-04-03

dita:mapPath

phg1621910019905.ditamap

dita:ditavalPath

pny1626732985837.ditaval

dita:id

phg1621910019905

The Optimizer estimates relation and join cardinalities with a level of confidence in the accuracy of the estimate. Optimizer confidence levels express a qualitative level of confidence that a given cardinality estimate is accurate given knowledge about the available statistics for the tables being analyzed in the process of optimizing a query.

Confidence levels are a factor the Optimizer uses to determine which available strategy is best at each step of creating a query plan. The lower the confidence level, the more conservative the strategy used, particularly for join planning, because the errors in a query plan are cumulative (and in the case of join planning, errors are multiplicative). Therefore, the Optimizer chooses to pursue a less aggressive query plan, particularly with join planning, whenever the confidence in the accuracy of the data used to plan a query step is not high, or if complete and accurate statistics are not available.

A cardinality estimate based on stale statistics can be inaccurate, causing the Optimizer to generate a less optimal query plan. Partial statistics collected from a dynamic single-AMP sample may produce a better plan than complete but stale statistics, depending on how poorly the stale statistics reflect the demographics of the current set of values making up the population of a column set or index.

Optimizer Confidence Levels

An EXPLAIN can report any or all of the following confidence levels for a cardinality estimate:

No confidence
Low confidence
High confidence
Index Join confidence
Vantage reports Index Join confidence only for join operations.

These confidence levels are based heavily on the presence or absence of statistics for the column and index sets specified as predicates in the SQL request being reported. The only exception to this is the case where the query conditions are so complex that statistics cannot be used. In such cases, an EXPLAIN reports no confidence.

Even when a join operation has no confidence, the Optimizer uses any statistics available for the condition to enhance the likelihood of producing a better query plan. A join operation where there is no confidence on one relation, but low, index join, or high on the other relation, has an overall confidence level of no confidence, even though there are statistics on the join columns of one relation. This is because the Optimizer assumes a confidence level equal to the lower confidence level assigned to one relation in the join.

Understand that no confidence by itself does not indicate a bad plan. Even with steps with no confidence, the Optimizer can typically generate a reasonable plan, especially with appropriate statistics. If a step references only base tables, consider collecting statistics on these tables.

Similarly, a high confidence level is not a guarantee of an accurate cardinality estimate. For example, suppose the Optimizer locates a query predicate value in one of the statistical histograms for a column or index set. Confidence is assumed to be High. But suppose the available statistics are stale. The Optimizer, by pursuing the assumptions that accrue to a High confidence level, can then produce a bad plan as a result.

The following sections are meant to provide only an overview of confidence levels, and are not comprehensive.

Confidence levels are assigned to the cardinality estimates for only two types of operations:

Single-table retrievals
Joins

The confidence levels the Optimizer reports and the confidence levels recorded in the Level attribute of the QCD StatsRec table are not related in any way.

Confidence Levels for Single-Table Retrieval Operations

The following table lists the meaning of each confidence level in context and reasons why each confidence level is assigned for single-table retrieval operations:

Confidence Level	Meaning	Reason
No	Vantage has neither Low nor High confidence in the cardinality and distinct value estimates for the relation. The Optimizer pursues conservative strategies to optimize the relevant steps.	Any of the following situations exists: No statistics have been collected on the column or index sets specified in the predicate. The predicate contains nondeterministic expressions for which base table statistics cannot be collected. For example, you cannot collect base table statistics for either of the following expressions: udf_random(col1) (UDF declared as nondeterministic) random(col1) You can collect statistics on a single-table join index defined using a complex expression (see Using Join Index Statistics to Estimate Single-Table Expression Cardinalities), and the optimizer can use those statistics to make single-table cardinality estimates. If you have collected such statistics, then a cardinality estimate can be made with high confidence. For an aggregate estimation, no statistics have been collected on the grouping columns.
Low	Vantage is moderately certain that the estimated cardinality and distinct value estimates for the relation are accurate. The Optimizer pursues more aggressive strategies to optimize the relevant steps.	One of the following states exists for the relation: There are conditions in the query on an index set for which no statistics have been collected. Cardinality estimates can be made based on sampling the index set. There are conditions in the query on an index or column set with collected statistics that are ANDed with conditions on unindexed columns. There are conditions in the query on an index or column set with collected statistics that are ORed with other conditions. For an aggregate estimation, there are statistics on single columns of the grouping column set or statistics on some, but not all, of the grouping columns. The confidence for single-AMP dynamic AMP statistical samples is Low. EXPLAIN reports express no confidence in estimates where no statistics have been collected, but Vantage samples dynamically from at least one AMP when the query runs.
High	Vantage has adequate confidence that the estimated cardinality and distinct value estimates for the relation are accurate. The Optimizer pursues more aggressive strategies to optimize the relevant steps.	Retrieval from a single relation with no predicates: There are conditions in the query on the primary index and statistics have been collected on the primary index. There are conditions in the query on the primary index, but no statistics have been collected on the primary index. The confidence is High under any of the following situations: 5 or more AMPs are sampled dynamically. Rows per value are sampled using a dynamic AMP sample. No skew is detected. Retrieval from a single relation with predicates: Statistics have been collected on the predicate columns or indexes and there is no skew. There are multiple equality predicates on which covering multitable statistics have been collected. Single-table join index statistics have been collected on a complex expression specified in a predicate written against the underlying base table of the single-table join index. Retrieval from a single temporal relation with predicates and a unique join index that can be used as an access path: Statistics have been collected on the join index that match the query predicate columns. For an aggregate estimation, the confidence is high under any of the following situations: The grouping columns are constants. The grouping columns have equality predicates. The grouping columns are all covered by a set of single multicolumn statistics. Statistics have been collected on the single grouping column.
Index Join	Not applicable	Applies only to joins.

For a retrieval operation from a spool, the confidence level is the same as the confidence level for the step that generated the spool.

Confidence Levels for Join Operations

For join operations, the Optimizer must know approximately how many rows result from each step in each join operation required to perform an SQL request. The Optimizer uses this information to select an optimal plan for joining the relations. Using factors such as join method and join geography, the join plan determines the best order for joining the relations. Because you can join up to 128 tables and single-table views per join clause, minimizing join cardinality estimation errors is critical for generating an optimal query plan.

Errors in join processing are cumulative, so minimize the possibilities for errors to occur in join planning by keeping fresh statistics on all your indexes and non-index join columns.

Join cardinality and rows per value estimates typically have a lower confidence level than is seen for a single table retrieval under analogous conditions, for reasons including the following:

Join cardinality estimates only rate High confidence when there is only a single row in both the left and right relation in the join.
The confidence level for a join operation never exceeds that of its input relations and assumes the confidence for the relation having the lower confidence.

The following table lists the meaning of each confidence level in context and reasons why each confidence level is assigned for join operations. All relational joins are binary operations. Joins on multiple relations are chained together such that the result of an earlier join operation is spooled and then joined to the next relation in the sequence the Optimizer determines for its join plan.

Confidence Level	Meaning	Reason
No	Vantage has neither Low nor High nor Index Join confidence in the estimated join cardinality.	One or both of the relations in the join does not have statistics on the join columns.
Low	Vantage is moderately certain that the estimated join cardinality is accurate.	Statistics have been collected on the join columns of both the left and right relations. One relation in the join has Low confidence and the other has any of the following confidence levels: Low High Index Join
High	Vantage has adequate confidence that the estimated join cardinality is accurate.	One relation in the join has High confidence and the other has either of the following confidence levels: High Index Join
Index Join	Vantage has adequate confidence that the estimated join cardinality is accurate, because of a uniqueness constraint on the join columns.	There is a unique index on the join columns. There is a foreign key relationship between the two relations in the join. Because of the way Vantage implements PRIMARY KEY and UNIQUE INDEX constraints, these are essentially two ways of saying the same thing.

Effect of Dynamic AMP Sampling on Reported Confidence Levels

The analysis of skew is based on the distribution of rows from each of the AMPs, and is a contributing factor to the confidence level expressed by the Optimizer.

Skew analysis computes the expected number of rows per AMP, and if that number is less than 5 percent of the expected number of rows per AMP, the AMP is moved to the skewed AMP list. If the total number of AMPs in the skewed list is less than or equal to 5 percent of the total number of AMPs sampled, the confidence level is set to low, otherwise to high.

When statistics are sampled from only one dynamically selected AMP, the confidence level is set to low.