spark.config Structure - Aster Analytics

Teradata AsterĀ® Spark Connector User Guide

Product
Aster Analytics
Release Number
7.00.00.01
Published
May 2017
Language
English (United States)
Last Update
2018-04-13
dita:mapPath
dbt1482959363906.ditamap
dita:ditavalPath
Generic_no_ie_no_tempfilter.ditaval
dita:id
dbt1482959363906
lifecycle
previous
Product Category
Software

The configuration file, spark.config, is in JSON format and has this structure:

  • Default identifier (key: default)

    Specifies the identifier of the default Hadoop/Spark cluster, which must be defined in the configuration file. This field cannot be empty.

  • Hadoop/Spark cluster identifier (key: Spark-cluster-identifier)

    Defines the identifier of a Hadoop/Spark cluster. The configuration file can have multiple Hadoop/Spark cluster identifiers, each of which has these components:

    • Spark parameters (key: spark-params)

      Specifies the default parameter values to use when submitting, monitoring, and canceling Spark jobs.

    • Spark properties (key: spark-properties)

      Specifies the properties to pass to Spark.

    • RunOnSpark parameters (key: run-on-spark)

      Specifies the default RunOnSpark parameters.

    Any of the preceding fields can be empty, in which case, the default function parameter values apply. Function parameter values specified in a query override any values specified by the preceding fields.

  • IP addresses of vworkers (key: workers-ip-addresses)

    Specifies the IP addresses of the vworkers, either as a list or as the string with which all vworker IP addresses start. If you use a list, put a comma (',') between adjacent addresses.