Before using the Spark SQL Connector (initiator or target), the Hadoop administrator must create the hdfs:///tdqg-spark/ directory. This directory serves the following purpose:
- It stores a dummy text file created by the Spark SQL connector when used for the first time and is required for the Spark SQL connector to work.
- It stores the cache files for user-defined foreign server objects that are used by the Spark SQL initiator.
- Log in to any Hadoop node.
- Create the directory using command: hdfs dfs -mkdir /tdqg-spark/
-
Enter the permissions as in the example below:
hdfs dfs -chmod 777 /tdqg-spark/
The permission 777 is an example, actual permissions are determined by the Hadoop administrator as long as the requirements are met to create the directory.