Before using the Spark SQL Connector (initiator or target), the Hadoop administrator must create the hdfs:///tdqg-spark/ directory. This directory serves the following purpose:
- It stores a dummy text file created by the Spark SQL connector when used for the first time and is required for the Spark SQL connector to work.
- It stores the cache files for user-defined foreign server objects that are used by the Spark SQL initiator.
- It stores temporary files when running the target connector using the Spark Application Execution Mechanism.
- Log on to any Hadoop node.
- Create the directory using command: hdfs dfs -mkdir /tdqg-spark/
Enter the permissions as in the example below:
hdfs dfs -chmod 777 /tdqg-spark/
The permission 777 is an example, actual permissions are determined by the Hadoop administrator if the requirements to create the directory are met.