2.10 - Creating an HDFS Directory for the Spark SQL Connector - Teradata QueryGrid

Teradata® QueryGrid™ Installation and User Guide

prodname
Teradata QueryGrid
vrm_release
2.10
created_date
September 2019
category
Administration
Configuration
Installation
User Guide
featnum
B035-5991-099K
Before using the Spark SQL Connector (initiator or target), the Hadoop administrator must create the hdfs:///tdqg-spark/ directory. This directory serves the following purpose:
  • It stores a dummy text file created by the Spark SQL connector when used for the first time and is required for the Spark SQL connector to work.
  • It stores the cache files for user-defined foreign server objects that are used by the Spark SQL initiator.
  • It stores temporary files when running the target connector using the Spark Application Execution Mechanism.
All users accessing the Spark SQL connector (initiator or target) must have WRITE access permission in the directory.
  1. Log on to any Hadoop node.
  2. Create the directory using command: hdfs dfs -mkdir /tdqg-spark/
  3. Enter the permissions as in the example below: hdfs dfs -chmod 777 /tdqg-spark/
    The permission 777 is an example, actual permissions are determined by the Hadoop administrator if the requirements to create the directory are met.