Problem: The RunOnSpark query continues to run after the YARN job ends with a failure message, or despite that the YARN job never started.
Solution: Use the STATUS_INTERVAL_SEC argument to check for the status of the YARN job at regular intervals.You can also use the TIMEOUT_SEC argument to forcefully cancel the RunOnSpark query after it has run for more than a specified number of seconds.