The configuration file, spark.config, is in JSON format and has this structure:
- Default identifier (key: default)
Specifies the identifier of the default Hadoop/Spark cluster, which must be defined in the configuration file. This field cannot be empty.
- Hadoop/Spark cluster identifier (key: Spark-cluster-identifier)
Defines the identifier of a Hadoop/Spark cluster. The configuration file can have multiple Hadoop/Spark cluster identifiers, each of which has these components:
- Spark parameters (key: spark-params)
Specifies the default parameter values to use when submitting, monitoring, and canceling Spark jobs.
- Spark properties (key: spark-properties)
Specifies the properties to pass to Spark.
- RunOnSpark parameters (key: run-on-spark)
Specifies the default RunOnSpark parameters.
Any of the preceding fields can be empty, in which case, the default function parameter values apply. Function parameter values specified in a query override any values specified by the preceding fields.
- Spark parameters (key: spark-params)
- IP addresses of vworkers (key: workers-ip-addresses)
Specifies the IP addresses of the vworkers, either as a list or as the string with which all vworker IP addresses start. If you use a list, put a comma (',') between adjacent addresses.