PySpark API Supportability Matrix | Spark SQL Input | pyspark2teradataml - Input - Teradata Package for Python

Teradata® pyspark2teradataml User Guide

Deployment
VantageCloud
VantageCore
Edition
Enterprise
IntelliFlex
VMware
Product
Teradata Package for Python
Release Number
20.00
Published
December 2024
ft:locale
en-US
ft:lastEdition
2024-12-18
dita:mapPath
oeg1710443196055.ditamap
dita:ditavalPath
ayr1485454803741.ditaval
dita:id
oeg1710443196055
Product Category
Teradata Vantage
  • Except for CSV, DataFrameReader will read the files from cloud storage only. Reading the files from local is not supported for files other than CSVs. DataFrameReader uses ReadNos from teradataml to load the data to DataFrame.

    All the arguments for teradataml ReadNos can be passed as options.

  • DataFrameReader.csv can read the file either from local file system or from cloud storage. If the specified CSV exists in local file system, DataFrameReader uses teradataml read_csv to load the data in to a DataFrame. Else, DataFrameReader.csv considers the file is in a cloud storage and uses teradataml ReadNos.
  • DataFrameWriter can store data in any cloud storage with correct id and password.
Once uploaded, there are no options to delete, overwrite, or append. You must update data at a new path.
Function Name Supported Notes
DataFramerReader.csv  
DataFramerReader.format  
DataFramerReader.json  
DataFramerReader.load  
DataFramerReader.option option are same as teradataml ReadNOS function
DataFramerReader.options options are same as teradataml ReadNOS function
DataFramerReader.parquet  
DataFramerReader.schema  
DataFramerReader.table  
DataFramerWriter.csv Doesn't support mode parameter.
DataFramerWriter.parquet Doesn't support mode parameter.
DataFramerWriter.json   Only available for local platform.
DataFramerWriter.orc   Only available for local platform.
DataFramerWriter.save  
DataFramerWriter.format  
DataFramerWriter.options options is same as teradataml WriteNOS function.
DataFramerWriter.option option is same as teradataml WriteNOS function.
DataFramerWriter.insertInto  
DataFramerWriter.saveAsTable  
DataFramerWriterV2.append  
DataFramerWriterV2.create  
DataFramerWriterV2.replace  
DataFramerWriterV2.createOrReplace  
DataFrameWriterV2.partitionedBy 'bucket' function is not supported.

'primary_index' is required to be used in "options".

DataFrameWriterV2.option  
DataFrameWriterV2.options