Amazon-EMR [SOLVED]: Yarn resource manager fails the application even though Spark application executes successfully

Amazon-EMR [SOLVED]: Yarn resource manager fails the application even though Spark application executes successfully

Home Forums Amazon Web Services Amazon EMR Amazon-EMR [SOLVED]: Yarn resource manager fails the application even though Spark application executes successfully

Viewing 2 posts - 1 through 2 (of 2 total)
  • Author
    Posts
  • #233157

    Cloudy Point
    Keymaster

    QuestionQuestion

    I am running simple hello world python script with AWS EMR + Spark + Yarn.

    Looking at the logs, even though the Spark application succeeds, overall job is marked as failed by Yarn resource manager.

    • Logs for the spark application shows success. “Hello world” is printed in stdout also. (See pastebin for application logs)
    • Logs for the node manager show no issue or error. (See pastebin for node manager logs)
    • Logs for the resource manager on master host show that resource manager marks the application as FAILED even though application completion seems to be successful. There is no apparent reason in the log for the failure! (See pastebin for resource manager logs)

    I checked all logs and cannot really figure out the root cause. What could be the issue? How can I debug further?

    #233158

    Cloudy Point
    Keymaster

    Accepted AnswerAnswer

    Your logs have the following statement:-

    ERROR ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors. Failing the application.
    

    This arises typically if you are setting .master() in the SparkSession builder.

    Source: https://stackoverflow.com/questions/47538689/yarn-resource-manager-fails-the-application-even-though-spark-application-execut
    Author: Sanchay
    Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Viewing 2 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.