Problem: RunOnSpark sanity check queries have the following error in the application master logs (accessible from the Ambari Resource Manager UI):
Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster
Solution: Before running the AsterToSpark installation script, do the following on the Hadoop name node:
- Find the Hadoop version with this command:
hdp-select status hadoop-client
- Add the file /usr/hdp/hadoop_version/etc/spark/conf/java-opts (if it is not there).
- To the file /usr/hdp/hadoop_version/etc/spark/conf/java-opts, add:
Dhdp.version=hadoop_version
- To the file /usr/hdp/2.4.*/etc/spark/conf/spark-defaults.conf, add:
spark.driver.extraJavaOptions -Dhdp.version=hadoop_version spark.yarn.am.extraJavaOptions -Dhdp.version=hadoop_version