yep,u can submit a app to spark ensemble by spark-submit command ,e.g.
spark-submit --master spark://gzsw-02:7077 --class org.apache.spark.examples.JavaWordCount --verbose --deploy-mode client ~/spark/spark-1.4.1-bin-hadoop2.4/lib/spark-examples-1.4.1-hadoop2.4.0.jar spark/spark-1.4.1-bin-hadoop2.4/RELEASE,spark/spark-1.4.1-bin-hadoop2.4/README.md
but also,u can spawn a command to run a app via spark-shell,
spark-shell --jars lib/spark-examples-1.4.1-hadoop2.4.0-my.jar --master spark://gzsw-02:7077 --executor-memory 600m --total-executor-cores 16
note:this param total-executor-cores is a must,else u will get a java.lang.OutOfMemoryError: Java heap space exception after this log
15/11/25 12:08:17 INFO SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:56889/user/Executor#-1124709965]) with ID 0
after than ,i check the master ui,i found that the app's cores shown as the max integer value(2147483647).so u can add this property to set used cores right.
second,if u want to limit the executors to run ,u can specify this property to chieve that
set in spark-defaults.conf
spark.deploy.spreadOut=false
(but with spark-class command ,this will be a side affect:it maybe cause the same effect with "java heap space" exceptions like above,so comment it is better in this case)
then ,u can import the entry class of your app like this
import class.to.your.appentry
followed by invoking the entry method
val arr = ...//params for running this app JavaWordCount.main(arr)
but in this demo,the app will issue a new sparkcontext which differs with the default one initiated by spark-shell,so launcher will fail to complain :
WARN SparkContext: Multiple running SparkContexts detected in the same JVM!
so u can add a property below in the spark-defaults.conf to ignore it
spark.driver.allowMultipleContexts=true
after all,u will the app is launched same as command spark-submit.
ref: