The following tables compare teradatamlspk.sql.functions with pyspark.sql.functions
Array Privileges and Configuration
To run array functions, you must have either the UDTTYPE or UDTMETHOD privilege on the SYSUDTLIB database. Use the following syntax to grant permission.
GRANT UDTMETHOD ON SYSUDTLIB TO <USER> GRANT SELECT ON UDTInfo TO <USER>
Configuration options for arrays enable users to specify element types and array length to work within Teradata's 64KB array size limit. This is necessary because Teradata uses structured UDT-based arrays with predefined options, while PySpark employs dynamic allocation without size constraints.
The default_array_size configuration option sets the array size irrespective of type, defaulted to 100.
Usage Example:
>>> from teradatamlspk import TeradataSession
>>> session = TeradataSession.builder.getOrCreate(...)
# Set the configuration option to store only 10 elements across the session.
>>> session.conf.set("default_array_size", 10)
Configuration Options
| Array Type | Configuration Option | Default Value |
|---|---|---|
| array_decimal | decimal_precision | 39 |
| decimal_scale | 19 | |
| array_number | number_precision | 38 |
| number_scale | 19 | |
| array_interval_year_to_month | interval_precision | 4 |
| array_interval_day_to_second | interval_frac_precision | 6 |
| interval_precision | 4 | |
| array_char | char_length | 2000 |
| array_varchar | char_length | 2000 |
| array_byte | byte_length | 64000 |
| array_varbyte | byte_length | 64000 |
| Function Name | Supported | Notes |
|---|---|---|
| col | ||
| column | ||
| asc | ||
| asc_nulls_last | ||
| desc | ||
| desc_nulls_first | ||
| desc_nulls_last | ||
| avg | ||
| any_value | ||
| row_number | ||
| count | ||
| rank | ||
| cume_dist | ||
| dense_rank | ||
| percent_rank | ||
| max | ||
| mean | ||
| min | ||
| sum | ||
| std | ||
| stddev_pop | ||
| stddev_samp | ||
| stddev | ||
| var_pop | ||
| var_samp | ||
| variance | ||
| lag | ||
| lead | ||
| count_distinct | ||
| countDistinct | ||
| corr | ||
| covar_pop | ||
| covar_samp | ||
| first | ||
| first_value | ||
| last | ||
| last_value | ||
| regr_avgx | ||
| regr_avgy | ||
| regr_count | ||
| regr_intercept | ||
| regr_r2 | ||
| regr_slope | ||
| regr_sxx | ||
| regr_sxy | ||
| regr_syy | ||
| sum_distinct | ||
| sumDistinct | ||
| lit | ||
| broadcast | ||
| coalesce | ||
| isnan | ||
| isnull | ||
| monotically_increasing_id | ||
| nanvl | ||
| rand | ||
| randn | ||
| spark_partition_id | ||
| when | ||
| bitwise_not | ||
| bitwiseNOT | ||
| expr | ||
| greatest | ||
| least | ||
| sqrt | ||
| abs | ||
| acos | ||
| asin | ||
| asinh | ||
| atan | ||
| atanh | ||
| atan2 | ||
| bin | ||
| cbrt | ||
| ceil | ||
| ceiling | ||
| input_file_name | ||
| named_struct | ||
| conv | ||
| cos | ||
| cosh | ||
| cot | ||
| csc | ||
| e | ||
| exp | ||
| expm1 | ||
| factorial | ||
| floor | ||
| hex | ||
| unhex | ||
| hypot | ||
| ln | ||
| log | ||
| log10 | ||
| log1p | ||
| log2 | ||
| negate | ||
| negative | ||
| pi | ||
| pmod | ||
| positive | ||
| pow | ||
| power | ||
| rint | ||
| round | ||
| bround | ||
| shiftleft | ||
| shiftright | ||
| shiftrightunsigned | ||
| sign | ||
| signum | ||
| sin | ||
| sinh | ||
| tan | ||
| tanh | ||
| toDegrees | ||
| try_add | ||
| try_avg | ||
| try_divide | ||
| try_multiply | ||
| try_subtract | ||
| try_sum | ||
| try_to_number | ||
| degrees | ||
| toRadians | ||
| radians | ||
| width_bucket | ||
| add_months | ||
| convert_timezone | ||
| curdate | ||
| current_date | ||
| current_timestamp | ||
| current_timezone | ||
| date_add | ||
| date_diff | ||
| date_format | ||
| date_from_unix_date | ||
| date_trunc | ||
| dateadd | ||
| datediff | ||
| day | ||
| date_part | ||
| datepart | ||
| dayofmonth | ||
| dayofweek | ||
| dayofyear | ||
| extract | ||
| second | ||
| weekofyear | ||
| year | ||
| quarter | ||
| month | ||
| last_day | ||
| localtimestamp | ||
| make_dt_interval | ||
| make_interval | ||
| make_timestamp | time zone values in column support only teradataml timezone strings. teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration. |
|
| make_timestamp_ltz | time zone values in column support only teradataml timezone strings. teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration. |
|
| make_timestamp_ntz | ||
| make_ym_interval | ||
| minute | ||
| months_between | ||
| next_day | ||
| hour | ||
| make_date | ||
| now | ||
| from_unixtime | ||
| unix_timestamp | ||
| to_unix_timestamp | ||
| to_timestamp | time zone values in column support only teradataml timezone strings. teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration. |
|
| to_timestamp_ltz | time zone values in column support only teradataml timezone strings. teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration. |
|
| to_timestamp_ntz | time zone values in column support only teradataml timezone strings. teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration. |
|
| to_date | ||
| trunc | ||
| weekday | ||
| unix_date | ||
| concat | teradatamlspk doesn't support nested functions for array-type columns. | |
| from_utc_timestamp | timezone format specifiers support only Vantage timezone format strings. teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration. |
|
| to_utc_timestamp | timezone format specifiers support only Vantage timezone format strings. teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration. |
|
| window | ||
| session_window | ||
| timestamp_micros | time zone values in column support only teradataml timezone strings. teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration. |
|
| timestamp_millis | time zone values in column support only teradataml timezone strings. teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration. |
|
| timestamp_seconds | time zone values in column support only teradataml timezone strings. teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration. |
|
| try_to_timestamp | time zone values in column support only teradataml timezone strings. teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration. |
|
| unix_micros | time zone values in column support only teradataml timezone strings. teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration. |
|
| unix_millis | time zone values in column support only teradataml timezone strings. teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration. |
|
| unix_seconds | time zone values in column support only teradataml timezone strings. teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration. |
|
| window_time | ||
| array |
|
|
| array_contains | teradatamlspk doesn't support nested functions. | |
| arrays_overlap | teradatamlspk doesn't support nested functions. | |
| array_join | teradatamlspk doesn't support nested functions. | |
| create_map | ||
| slice | teradatamlspk doesn't support nested functions. | |
| array_position |
|
|
| element_at | Map Type columns are not supported. | |
| array_append | teradatamlspk doesn't support nested functions. | |
| array_size | ||
| array_sort |
|
|
| array_insert |
|
|
| array_remove |
|
|
| array_prepend | teradatamlspk doesn't support nested functions. | |
| array_distinct | teradatamlspk doesn't support nested functions. | |
| array_intersect | teradatamlspk doesn't support nested functions. | |
| array_union | teradatamlspk doesn't support nested functions. | |
| array_except | teradatamlspk doesn't support nested functions. | |
| array_compact | teradatamlspk doesn't support nested functions. | |
| array_agg |
|
|
| transform | ||
| exists | ||
| forall | ||
| filter | ||
| aggregate | ||
| zip_with | ||
| transform_keys | ||
| tranform_values | ||
| map_filter | ||
| map_from_arrays | ||
| map_zip_with | ||
| explode |
|
|
| explode_outer |
|
|
| pos_explode |
|
|
| posexplode_outer |
|
|
| inline | ||
| inline_outer | ||
| get | Map Type columns are not supported. | |
| get_json_object | ||
| json_tuple | ||
| from_json | ||
| schema_of_json | ||
| to_json | ||
| json_array_length | ||
| json_object_keys | ||
| size | ||
| cardinality | ||
| struct | ||
| sort_array | teradatamlspk doesn't support nested functions. | |
| array_max |
|
|
| array_min |
|
|
| shuffle | teradatamlspk doesn't support nested functions. | |
| reverse | teradatamlspk doesn't support nested functions for array-type columns. | |
| flatten | ||
| sequence | teradatamlspk doesn't support nested functions. | |
| array_repeat | teradatamlspk doesn't support nested functions. | |
| map_contains_key | ||
| map_keys | ||
| map_values | ||
| map_entries | ||
| map_from_entries | ||
| arrays_zip | ||
| map_concat | ||
| from_csv | ||
| schema_of_csv | ||
| str_to_map | ||
| to_csv | ||
| try_element_at | Map Type columns are not supported. | |
| years | ||
| months | ||
| days | ||
| hours | ||
| bucket | ||
| ascii | ||
| base64 | ||
| btrim | ||
| char | ||
| character_length | ||
| char_length | ||
| concat_ws | ||
| contains | ||
| decode | ||
| elt | ||
| encode | ||
| endswith | ||
| find_in_set | ||
| format_number | ||
| format_string | ||
| ilike | ||
| initcap | ||
| instr | ||
| lcase | ||
| length | ||
| like | ||
| lower | ||
| left | ||
| levenshtein | ||
| locate | ||
| lpad | ||
| ltrim | ||
| mask | ||
| octet_length | ||
| parse_url | ||
| position | ||
| printf | ||
| rlike | Regex pattern varies from pyspark to teradatamlspk. As teradata uses posix for regex match. | |
| regexp | ||
| regexp_like | ||
| regexp_count | ||
| regexp_extract | ||
| regexp_extract_all | ||
| regrexp_replace | ||
| regexp_substr | ||
| regexp_instr | ||
| replace | ||
| right | ||
| ucase | ||
| unbase64 | ||
| rpad | ||
| repeat | ||
| rtrim | ||
| soundex | ||
| split | ||
| split_part | ||
| startswith | ||
| substr | ||
| substring | ||
| substring_index | ||
| overlay | ||
| sentences | ||
| to_binary | ||
| to_char | Format specifiers for PySpark is different from Teradata if format specifiers lies in a Column. Make sure to update the format specifiers in the corresponding column accordingly. |
|
| to_number | Format specifiers for PySpark is different from Teradata if format specifiers lies in a Column. Make sure to update the format specifiers in the corresponding column accordingly. | |
| to_varchar | Format specifiers for PySpark is different from Teradata if format specifiers lies in a Column. Make sure to update the format specifiers in the corresponding column accordingly. | |
| translate | ||
| trim | ||
| upper | ||
| url_decode | ||
| url_encode | ||
| bit_count | ||
| bit_get | ||
| getbit | ||
| call_function | ||
| call_udf | Columns passed to the UDF should be present in the corresponding DataFrame. | |
| pandas_udf | ||
| udf | Columns passed to the UDF should be present in the corresponding DataFrame. | |
| udtf | ||
| unwrap_udt | ||
| aes_decrypt | ||
| bitmap_bit_position | ||
| bitmap_bucket_number | ||
| bitmap_construct_agg | ||
| bitmap_count | ||
| bitmap_or_agg | ||
| current_catalog | ||
| current_database | ||
| current_schema | ||
| current_user | ||
| input_file_block_length | ||
| input_file_block_start | ||
| md5 | ||
| sha | ||
| sha1 | ||
| sha2 | ||
| crc32 | ||
| hash | ||
| xxhash64 | ||
| assert_true | ||
| raise_error | ||
| reflect | ||
| hll_sketch_estimate | ||
| hll_union | ||
| java_method | ||
| stack | ||
| try_aes_decrypt | ||
| typeof | ||
| user | ||
| version | Returns Teradata database version. | |
| equalnull | ||
| ifnull | ||
| isnotnull | ||
| nullif | ||
| nvl2 | ||
| nvl | ||
| xpath | ||
| xpath_boolean | ||
| xpath_double | ||
| xpath_float | ||
| xpath_int | ||
| xpath_long | ||
| xpath_number | ||
| xpath_short | ||
| xpath_string |