Amazon-Data-Pipeline [SOLVED]: Load props file in EMR Spark Application

Amazon-Data-Pipeline [SOLVED]: Load props file in EMR Spark Application

Home Forums Amazon Web Services Amazon Data Pipeline Amazon-Data-Pipeline [SOLVED]: Load props file in EMR Spark Application

Viewing 2 posts - 1 through 2 (of 2 total)
  • Author
    Posts
  • #233161

    Cloudy Point
    Keymaster

    QuestionQuestion

    I am trying to load custom properties in my spark application using :-

    command-runner.jar,spark-submit,--deploy-mode,cluster,--properties-file,s3://spark-config-test/myprops.conf,--num-executors,5,--executor-cores,2,--class,com.amazon.Main,#{input.directoryPath}/SWALiveOrderModelSpark-1.0-super.jar
    

    However, I am getting the following exception:-

    Exception in thread “main” java.lang.IllegalArgumentException: Invalid
    properties file ‘s3://spark-config-test/myprops.conf”. at
    org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
    at
    org.apache.spark.launcher.AbstractCommandBuilder.loadPropertiesFile(AbstractCommandBuilder.java:284)
    at
    org.apache.spark.launcher.AbstractCommandBuilder.getEffectiveConfig(AbstractCommandBuilder.java:264)
    at
    org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:233)
    at org

    Is this the correct way to load file from S3?

    #233162

    Cloudy Point
    Keymaster

    Accepted AnswerAnswer

    You can’t load a properties file directly from S3. Instead you will need to download the properties file to your master node somewhere, then submit the spark job referencing the local path on that node. You can do the download by using command-runner.jar to run the aws cli utility.

    Source: https://stackoverflow.com/questions/47535424/load-props-file-in-emr-spark-application
    Author: RyanW
    Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Viewing 2 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.