If you installed Hadoop with a custom library path or stored any Hadoop .jar files outside of the default Hadoop library, you must manually configure the Spark connector with the custom path for any files that do not reside in the default Hadoop library path.
This only affects the target Spark connector configuration and any files outside the default Hadoop path.
- Enable the Hadoop Library Path property as described in Spark SQL Connector and Link Properties.
- Enter the custom library path in the Hadoop Library Path NVP using comma-separated values for all .jar files not in the default Hadoop library, for example:
/usr/hdp/2.6.0.3-8/hive-hcatalog/share/hcatalog/,/usr/hdp/2.6.0.3-8/hadoop-hdfs/,/usr/hdp/2.6.0.3-8/hadoop-mapreduce