RunOnSpark non-ssh sanity check queries fail on HDP with AD6.20 - Aster Analytics

Problem: RunOnSpark sanity check queries have the following error in the application master logs (accessible from the Ambari Resource Manager UI):

Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster

Solution: Before running the AsterToSpark installation script, do the following on the Hadoop name node:

1. Find the Hadoop version with this command:

   hdp-select status hadoop-client

2. Add the file /usr/hdp/hadoop_version/etc/spark/conf/java-opts (if it is not there).
3. To the file /usr/hdp/hadoop_version/etc/spark/conf/java-opts, add:

   Dhdp.version=hadoop_version

4. To the file /usr/hdp/2.4.*/etc/spark/conf/spark-defaults.conf, add:

   spark.driver.extraJavaOptions -Dhdp.version=hadoop_version
   spark.yarn.am.extraJavaOptions -Dhdp.version=hadoop_version