The configureAsterSpark script generated the installation/configuration script AsterSpark_hadoop_node_host_name_hadoop.sh, which creates the HDFS directories and jar files that Spark jobs submitted by sparkJobSubmitter need.
The AsterSpark_hadoop_node_host_name_hadoop.sh is idempotent: rerunning it does not duplicate previous work. If an item to create or copy already exists, the script reports this fact and continues.
-
Copy the Hadoop installation/configuration script to the Hadoop Spark name node:
scp AsterSpark_hadoop_node_host_name_hadoop.sh sparkJobSubmitter@hadoop_node_host_name:/tmp
-
On the Hadoop Spark name node, run the script as the user sparkJobSubmitter:
./AsterSpark_hadoop_node_host_name_hadoop.sh
The script creates the HDFS directories, copies jar files to them, and displays:++ hadoop fs -test -e /user/runonspark/runonspark ...
The script creates two HDFS subdirectories and copies the spark-assembly*.jar file from the queen node to the appropriate aster-spark-extension-spark*.jar file.
- Proceed to Executing the Teradata Aster Spark Queen Installation/Configuration Script.