Designating a Spark SQL connector property as overridable in the Teradata Fabric portlet allows you to override configured Spark SQL connector properties when running queries during an individual processing session.
For the Spark SQL initiator session, setSessionOverride() is used at the Foreign Server level to override configured Spark SQL connector properties. The session override applies to all non-native tables that belong to the Foreign Server.
- Under Fabric Configuration, select Fabrics.
- Select the fabric.
- Perform one of the following:
Option Description To override a Spark SQL connector property in the Connectors tab - Select the Connectors tab.
- Select
next to the connector that has a property to override. - Select Edit.
- Select
next to Connector Software. - Find the Spark SQL connector property to override and select the Overridable check box.
- Select OK.
- Select Save..
To override a Spark SQL connector property in the Links tab - Select the Links tab.
- Select next to the link that has a property to override.
- Select Edit.
- Select next to Initiating Connector or Target Connector.
- Find the Spark SQL link property to override and select the Overridable check box.
- Select OK.
- Select Save..
- [Optional] To override the Spark SQL connector property during a Spark SQL session for Foreign Server s1, use the syntax in the following example where the numExecutors and hadoopProperties connector properties are overridden.s1.setSessionOverride("linkBufferSize=100000, numExecutors=4, hadoopProperties=name1:value1 | name2:value2")When overriding hadoopProperties, you must use the pipe (|) symbol to split Hadoop properties and a colon (:) to split name value pairs, as shown in these examples.
- [Optional] To clear Spark SQL connector property overrides without closing a Spark SQL session for Foreign Server S1, use the syntax in the following example:s1.setSessionOverride("")