A Historical Perspective of Python and R in Teradata | VantageCloud Lake - A Historical Perspective of Python and R in Teradata - Teradata VantageCloud Lake

Lake - Analyze Your Data with ClearScape Analytics™
Deployment
VantageCloud
Edition
Lake
Product
Teradata VantageCloud Lake
Release Number
Published
February 2025
ft:locale
en-US
ft:lastEdition
2026-02-20
dita:mapPath
tcl1683670667798.ditamap
dita:ditavalPath
pny1626732985837.ditaval
dita:id
tcl1683670667798

Teradata customers have been always interested in using external languages such as Python and R for in-database analytics. By design, these languages run in-memory. This characteristic can make it challenging for users to process large volumes of data or build applications that can scale.

Since 2010, Teradata has been progressively expanding its capabilities in using Python and R in the following areas, while also enabling users to take advantage of the Vantage architecture parallelism in conjunction with Python and R scripting:
  • Enable in-database analytics with external languages.
  • Execute user scripts natively in the database nodes, and scale performance by running them in parallel across all processing units.
  • Simplify user interaction with the database by providing Python and R client wrappers for database analytic functions and operations.
The progression in landmarks for Python and R in Teradata is illustrated in the following diagram.


These progressive advances brought significant convenience for Teradata Vantage Python and R users over the past decade. Continuing integration across the latest tools has been also accomplished through the following areas:
  • The advent of Python and R wrapper functions for calls to the SCRIPT table operator via the Teradata client packages.
  • A sandbox environment for users to test their scripts prior to running them in the database.
  • The introduction by teradataml of the map_row() and map_partition() methods of teradataml Python DataFrames. The latter objects call the SCRIPT table operator in the background in a way that is transparent to the users, thus simplifying Python scripting in the database.
The Teradata In-nodes Packages for Python and R resolved long-standing customer pain points, such as the pre-existing lack of supported language distributions for in-database usage with the SCRIPT table operator. Still, this approach is sometimes challenged by nuances, such as the following:
  • There could be limited amounts of memory available for the Teradata In-nodes Packages for Python and R in the Database Engine 20.

    Namely, the in-nodes interpreters are allowed to access only non-file segment (non-FSG) cache memory. This is the amount of system memory left over after the database needs have been accounted for. The FSG cache memory typically takes about 85% to 90% of each system node memory, hence database users are often left with relatively moderate amounts of memory for the scaled execution of scripts in the database. This set-up could lead to insufficient resources for use cases where it might be necessary to load large data sets in memory and process them in the scripts.

  • In some cases, users might need language libraries outside the list of titles available in the current Teradata in-nodes distribution.

    Library requests are typically satisfied within the time frame of the following release. However, older operating systems in the Database Engine 20 nodes might occasionally prevent installation of some libraries due to lack of suitable dependency versions. In contrast, it would be convenient to simply fetch desired libraries from a preferred repository and install them on the spot.

  • Python users also commonly make use of Python virtual environments, yet the limited in-database resources make it practically unfeasible to operate with virtual environments in Database Engine 20 nodes.
  • Limited resources for languages in the database also render very difficult to execute multiple instances of the SCRIPT table operator concurrently. In fact, in Vantage Enterprise it is suggested that only concurrency 1 is used with the SCRIPT table operator. To this end, it would be desirable to allow for higher levels of concurrency.
  • On the topic of security, more elaborate operational recommendations have been identified over time to further secure execution of the SCRIPT and ExecR table operators on the Database Engine 20 nodes.

In the VantageCloud Lake product, the SCRIPT and ExecR table operators are retired to make way for an entirely new approach. This approach introduces the Open Analytics Framework, which we will overview in the next section.