PySpark API Supportability Matrix | Functions | pyspark2teradataml - Functions - Teradata Package for Python

Teradata® pyspark2teradataml User Guide

Deployment
VantageCloud
VantageCore
Edition
Enterprise
IntelliFlex
VMware
Product
Teradata Package for Python
Release Number
20.00
Published
December 2024
ft:locale
en-US
ft:lastEdition
2024-12-18
dita:mapPath
oeg1710443196055.ditamap
dita:ditavalPath
ayr1485454803741.ditaval
dita:id
oeg1710443196055
Product Category
Teradata Vantage
Function Name Supported Notes
col  
column  
asc  
asc_nulls_last  
desc  
desc_nulls_first  
desc_nulls_last  
avg  
any_value  
row_number  
count  
rank  
cume_dist  
dense_rank  
percent_rank  
max  
mean  
min  
sum  
std  
stddev_pop  
stddev_samp  
stddev  
var_pop  
var_samp  
variance  
lag  
lead  
count_distinct  
countDistinct  
corr  
covar_pop  
covar_samp  
first  
first_value  
last  
last_value  
regr_avgx  
regr_avgy  
regr_count  
regr_intercept  
regr_r2  
regr_slope  
regr_sxx  
regr_sxy  
regr_syy  
sum_distinct  
sumDistinct  
lit  
broadcast  
coalesce  
isnan  
isnull  
monotically_increasing_id  
nanvl  
rand  
randn  
spark_partition_id  
when  
bitwise_not  
bitwiseNOT  
expr  
greatest  
least  
sqrt  
abs  
acos  
asin  
asinh  
atan  
atanh teradatamlspk raises an error when the value in a column is 1.
atan2  
bin  
cbrt Function is only valid for positive integer values.
ceil  
ceiling  
input_file_name  
named_struct  
conv  
cos  
cosh  
cot teradatamlspk raises an error when the value in a column is 0.
csc teradatamlspk raises an error when the value in a column is 0.
e  
exp  
expm1  
factorial  
floor  
hex If the value of a column is a floating number, cast the column to Integer type and run the function.
unhex  
hypot  
ln Function raises an error when value in a column is 0.
log  
log10 Function raises an error when value in a column is 0.
logip Function raises an error when value in a column is 0.
log2  
negate  
negative  
pi  
pmod  
positive  
pow  
power  
rint  
round  
bround  
shiftleft Function supports only Integer type values.
shiftright Function supports only Integer type values.
shiftrightunsigned  
sign Function raises error when value in a column is 0.
signum Function raises error when value in a column is 0.
sin  
sinh  
tan  
tanh  
toDegrees  
try_add  
try_avg  
try_divide  
try_multiply  
try_subtract  
try_sum  
try_to_number  
degrees  
toRadians  
radians  
width_bucket  
add_months  
convert_timezone  
curdate  
current_date  
current_timestamp  
current_timezone  
date_add  
date_diff  
date_format  
date_from_unix_date  
date_trunc  
dateadd  
datediff  
day  
date_part  
datepart  
dayofmonth  
dayofweek  
dayofyear  
extract  
second  
weekofyear  
year  
quarter  
month  
last_day  
localtimestamp  
make_dt_interval  
make_interval  
make_timestamp time zone values in column supports only teradataml timezone strings.

teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration.

make_timestamp_ltz time zone values in column supports only teradataml timezone strings.

teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration.

make_timestamp_ntz  
make_ym_interval  
minute  
months_between  
next_day  
hour  
make_date  
now  
from_unixtime  
unix_timestamp  
to_unix_timestamp  
to_timestamp time zone values in column supports only teradataml timezone strings.

teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration.

to_timestamp_ltz time zone values in column supports only teradataml timezone strings.

teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration.

to_timestamp_ntz time zone values in column supports only teradataml timezone strings.

teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration.

to_date  
trunc  
weekday  
unix_date  
concat  
from_utc_timestamp time zone values in column supports only teradataml timezone strings.

teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration.

to_utc_timestamp time zone values in column supports only teradataml timezone strings.

teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration.

window  
session_window  
timestamp_micros time zone values in column supports only teradataml timezone strings.

teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration.

timestamp_millis time zone values in column supports only teradataml timezone strings.

teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration.

timestamp_seconds time zone values in column supports only teradataml timezone strings.

teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration.

try_to_timestamp time zone values in column supports only teradataml timezone strings.

teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration.

unix_micros time zone values in column supports only teradataml timezone strings.

teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration.

unix_millis time zone values in column supports only teradataml timezone strings.

teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration.

unix_seconds time zone values in column supports only teradataml timezone strings.

teradatamlspk timezone strings don't consider Daylight Saving Time(DST). Teradata recommends using Vantage timezone strings for DST consideration.

window_time  
array  
array_contains  
arrays_overlap  
array_join  
create_map  
slice  
array_position  
element_at  
array_append  
array_size  
array_sort  
array_insert  
array_remove  
array_prepend  
array_distinct  
array_intersect  
array_union  
array_except  
array_compact  
transform  
exists  
forall  
filter  
aggregate  
zip_with  
transform_keys  
tranform_values  
map_filter  
map_from_arrays  
map_zip_with  
explode  
explode_outer  
posexplode  
posexplode_outer  
inline  
inline_outer  
get  
get_json_object  
json_tuple  
from_json  
schema_of_json  
to_json  
json_array_length  
json_object_keys  
size  
cardinality  
struct  
sort_array  
array_max  
array_min  
shuffle  
reverse DataFrameColumn.reverse is supported on string type columns and not supported on array type columns.
flatten  
sequence  
array_repeat  
map_contains_key  
map_keys  
map_values  
map_entries  
map_from_entries  
arrays_zip  
map_concat  
from_csv  
schema_of_csv  
str_to_map  
to_csv  
try_element_at  
years  
months  
days  
hours  
bucket  
ascii  
base64  
btrim  
char  
character_length  
char_length  
concat_ws  
contains  
decode  
elt  
encode  
endswith  
find_in_set  
format_number  
format_string  
ilike
  • Accepts only string values for argument ‘pattern'.

    Argument escapeChar is not supported.

initcap  
instr  
lcase  
length  
like
  • Accepts only string values for argument ‘pattern'.

    Argument escapeChar is not supported.

lower  
left  
levenshtein  
locate  
lpad  
ltrim  
mask  
octet_length  
parse_url Does not support key parameter.
position  
printf  
rlike  
regexp  
regexp_like  
regexp_count  
regexp_extract  
regexp_extract_all  
regrexp_replace  
regexp_substr  
regexp_instr  
replace  
right  
ucase  
unbase64  
rpad  
repeat  
rtrim  
soundex  
split  
split_part  
startswith  
substr
  • Argument ‘pos' and len' accepts only Integer type columns.
  • Argument ‘pos' and len' does not accept ColumnExpressions.
substring  
substring_index  
overlay  
sentences  
to_binary  
to_char Argument 'pattern' does not accept Column.

Format specifiers for PySpark is different from Teradata if format specifiers lies in a Column. Make sure to update the format specifiers in the corresponding column accordingly

to_number Argument 'pattern' does not accept Column.
to_varchar Argument 'pattern' does not accept Column.

Format specifiers for PySpark is different from Teradata if format specifiers lies in a Column. Make sure to update the format specifiers in the corresponding column accordingly

translate  
trim  
upper  
url_decode  
url_encode  
bit_count  
bit_get  
getbit  
call_function  
call_udf  
pandas_udf  
udf Lambda functions are not supported.
udtf  
unwrap_udt  
aes_decrypt  
bitmap_bit_position  
bitmap_bucket_number  
bitmap_construct_agg  
bitmap_count  
bitmap_or_agg  
current_catalog  
current_database  
current_schema  
current_user  
input_file_block_length  
input_file_block_start  
md5  
sha  
sha1  
sha2  
crc32  
hash  
xxhash64  
assert_true  
raise_error  
reflect  
hll_sketch_estimate  
hll_union  
java_method  
stack  
try_aes_decrypt  
typeof  
user  
version Returns database version.
equalnull  
ifnull  
isnotnull  
nullif  
nvl2  
nvl  
xpath  
xpath_boolean  
xpath_double  
xpath_float  
xpath_int  
xpath_long  
xpath_number  
xpath_short  
xpath_string