Executing the Teradata Aster Spark Hadoop Installation/Configuration Script - Aster Analytics

Teradata AsterĀ® Spark Connector User Guide

Product
Aster Analytics
Release Number
7.00.00.01
Published
May 2017
Language
English (United States)
Last Update
2018-04-13
dita:mapPath
dbt1482959363906.ditamap
dita:ditavalPath
Generic_no_ie_no_tempfilter.ditaval
dita:id
dbt1482959363906
lifecycle
previous
Product Category
Software

The configureAsterSpark script generated the installation/configuration script AsterSpark_hadoop_node_host_name_hadoop.sh, which creates the HDFS directories and jar files that Spark jobs submitted by sparkJobSubmitter need.

The AsterSpark_hadoop_node_host_name_hadoop.sh is idempotent: rerunning it does not duplicate previous work. If an item to create or copy already exists, the script reports this fact and continues.
  1. Copy the Hadoop installation/configuration script to the Hadoop Spark name node:
    scp AsterSpark_hadoop_node_host_name_hadoop.sh sparkJobSubmitter@hadoop_node_host_name:/tmp
  2. On the Hadoop Spark name node, run the script as the user sparkJobSubmitter:
    ./AsterSpark_hadoop_node_host_name_hadoop.sh
    The script creates the HDFS directories, copies jar files to them, and displays:
    ++ hadoop fs -test -e /user/runonspark/runonspark
    ...

    The script creates two HDFS subdirectories and copies the spark-assembly*.jar file from the queen node to the appropriate aster-spark-extension-spark*.jar file.

  3. Proceed to Executing the Teradata Aster Spark Queen Installation/Configuration Script.