General
When setting parameters for Spark SQL target connectors in NVP link pairings, make sure the setting of the Conf File Paths property has the correct pathname. QueryGrid heavily depends on this setting when processing data transfers. See Spark SQL Connector and Link Properties.
Kerberos
You can set up QueryGrid to use Kerberos authentication with a Spark SQL target connector. There are two forms of authentication available with Kerberos:
- Username/Password
- The Spark SQL target connector authenticates the username and password against Kerberos before sending the query to the data source.
- Kerberos Keytab
- Spark SQL can be configured to enable Kerberos Keytab authentication.
- Select Kerberos for the Authentication Mechanism property.
- Verify the QueryGrid user has permission to run kinit. See Verifying Permission to Run kinit (Spark SQL).
Trusted Kerberos
The Spark SQL connector supports the authentication mechanism value Trusted Kerberos, which allows impersonation of the QueryGrid end-users with a single Kerberos service account. If the Kerberos service account has Hadoop-level privileges to impersonate a requested user, then QueryGrid queries run in the session as that requested user.
This feature is supported only with Kerberized CDP clusters using Spark with Execution Mechanism as Spark Application and QueryGrid 3.08 and later. The Kerberos setup is the same as the Kerberos authentication setup, but with additional settings.
The impersonated user is passed to the target system using the spark-submit command’s --proxy-user parameter. The impersonated user must be in Allowed OS users to run queries on the target system. There are two ways to define the impersonated user:
| Options | Description |
|---|---|
| User on the initiating system (Default) | The Teradata system user is passed to the target system as the impersonated user with the username in all lowercase. |
| User Mapping | User mapping can define the impersonated user with a case-sensitve username on the target system for any user on the initiating system. A typical use-case is when the impersonated username is not all lowercase. |
When configuring user impersonation on Hadoop cluster, the service user must be able to impersonate the requested user. This requires configuring the properties hadoop.proxyuser.kerberos_service_user.groups and hadoop.proxyuser.kerberos_service_user.hosts in the core-site.xml file for the Kerberos service account. QueryGrid queries are triggered by spark-submit from the driver node; therefore, enter the IP addresses of the driver nodes in the hadoop.proxyuser.kerberos_service_user.hosts property.
- The hadoop.proxyuser.krb service.groups should list the group where impersonated user belongs to. This allows the krb service` to impersonate the users belong to that group.
- krb service must belong to the primary group of impersonated user to access the temp files under the HDFS directory /tdqg-spark that QueryGrid creates while running the queries.
- Avoid using wild cards (*) as the value for the hadoop.proxyuser.krb service.hosts and hadoop.proxyuser.krb service.groups properties. Apply the intended restrictions on the user groups that can be impersonated by different hosts.