用spark + java混合实现spark项目,我想肯定有我一样坚持要用eclipse + maven来配置开发环境,而不愿意换Intelij + sbt的同学吧。
照着文章中的步骤配置,完全ok。好文共享,原文挺好懂的,就不翻译了~
============================== 我是正文分隔线 ==============================
By Adrian Null
Maven is a build/project management tool. It favours “convention over configuration”; it can greatly simplify builds for “standard” projects and a Maven user can usually understand the structure of another Maven project just by looking at its pom.xml
(Project Object Model). Maven is a plugin-based architecture, making it easy to add new libraries and modules to existing projects. For example, adding a new dependency usually involves only 5 extra lines in the pom.xml
. These “artifacts” are downloaded from repositories such as The Central Repository.
You can also check out the official example project which uses the same Scala plugin we will show here.
If you’re familiar with Maven, you can go ahead with the Scala Maven Plugin.
We’ll be using the Scala Maven Plugin (GitHub repo, website) (formerly known as the maven-scala-plugin; renamed to honour the new naming policy where only Maven core plugins are prefixed with “maven”), by far the dominant plugin for Scala projects. Note: the plugin includes Scala from the Central Repository so there’s no need to install it yourself if you’re compiling with Maven.
On Debian and Debian-derivatives, Maven is usually available via apt-get
. Just do (sudo) apt-get install maven
and you’re good to go.
OSX prior to 10.9 (Mavericks) comes with Maven 3 built in. If you don’t have it, you can get it with the package managers MacPorts, Homebrew, or Fink. The Scala Maven Plugin requires Maven 3.0+
You can download Maven from its Apache homepage. After extracting it (tar -zxvf apache-maven-X.X.X-bin.tar.gz
, or use something like 7-zip) to your directory of choice (on Linux and OSX, Unix-like systems, I like to put them in /opt/
. On Windows I would probably put this inC:/
), you need to add Maven to your environment Path variable:
/usr/bin
, which is already on your Path
ln -s /usr/bin/mvn /opt/apache-maven-X.X.X/bin/mvn
bin
folder directly to your path, using your shell configuration file (e.g. ~/.bash_profile
)
export PATH=$PATH:/opt/apache-maven-X.X.X/bin
to .bash_profile
(or whatever profile for the shell you use)echo "export PATH=$PATH:/opt/apache-maven-X.X.X/bin" >> ~/.bash_profile
mvn
shell script in an existing path location
$HOME/bin
in your path$HOME/bin
(mv apache-maven-X.X.X "$HOME/bin/"
)mvn
in $HOME/bin
"$HOME/bin/apache-maven-X.X.X/bin/mvn" $@
to it, and chmod u+x mvn
to make it executable$HOME/bin
is usually added to the user’s path by default, and if not, it’s a useful thing to do/have anyways. The shell script simply invokes the Maven location (which is at some other location) and passes on the argumentsC:\apache-maven-X.X.X
). Use backslashes to be safe, and do not include a trailing slash%MAVEN3_HOME%\bin
;%MAVEN3_BIN%
to itThe easiest way to create new projects is using an “archetype”. An archetype is a general skeleton structure, or template for a project. Think back to “convention over configuration”; in our case, the Scala Maven Plugin provides an archetype for scala projects.
You run the archetype plugin like this:
mvn archetype:generate
If this is your first time, you’ll notice that Maven is downloading many jar files. Maven resolves dependencies and downloads them as needed (and only once). Right now, Maven is downloading its core plugins.
Afterwards, it should give you a list of archetypes (in the hundreds). The Scala Maven Plugin was 339 on my list: “net.alchim31.maven:scala-archetype-simple (The maven-scala-plugin is used for compiling/testing/running/documenting scala code in maven.)”. You can type “scala” (or something else) to filter the results. As of 2015 January 27, there you can choose version 3.1.4 or 3.1.5 of this plugin; you should choose the latest
Choose net.alchim31.maven:scala-archetype-simple version:
1: 1.4
2: 1.5
Next, Maven will ask you for a groupId, artifactId, and package. You can read the guide to naming conventions, but in short:
The groupId and artifactId together should serve as a globally unique identifier for your project
When it’s done, you should see a new folder named with the artifactId. cd
into it and run:
mvn package
You’ll see Maven downloading dependencies including the Scala library (as mentioned above),JUnit, ScalaTest, and Specs2 (the latter three are test frameworks; the archetype includes an example “Hello world” program, and tests with each of the frameworks).
In your project root, you’ll see a pom.xml
, src
folder, and target
folder (target folder only appears after building). Note: this archetype also includes a .gitignore
Inside the src
folder you’ll see main
and test
; main
includes your application code, and test
includes your test suites. Inside each of those you’ll find a scala
folder, followed by your package structure (actually, test/scala
includes a sample package, but you should replace this with your own package and tests). If you want to mix Scala and Java source code, simply add a java
folder inside main
or test
.
target
includes generated/built files, such as .class
and .jar
files. You can read aboutpom.xml
at the Maven page.
Example structure:
Again, you can read more about the Scala Maven Plugin at its website.
By default the jar created by the Scala Maven Plugin doesn’t include a Main-Class
attribute in the manifest. I had to add the Maven Assembly Plugin to my pom.xml
in order to specify custom attributes in the manifest. You can check the latest version of this plugin at the project summary or at The Central Repository
...>
X.X.X
...
...
...
...
...
...
org.apache.maven.plugins
maven-assembly-plugin
2.4
jar-with-dependencies
com.your-package.MainClass
package
single
After adding this, mvn package
will also create [artifactId]-[version]-jar-with-dependencies.jar
under target
. Note: this will also copy the Scala library into your Jar. This is normal. Be careful that your dependencies use the same version of Scala, or you will quickly end up with a massive Jar.
mvn dependency:copy-dependencies
: copy all libraries and dependencies to thetarget/dependency
foldermvn clean
mvn package
: compile, run tests, and create jarThere are instructions at the Scala Maven Plugin FAQs, but I thought I’d expand a bit. The maven-eclipse-plugin is a core plugin (all plugins prefixed with “maven” are, and are available by default) to generate Eclipse configuration files. However, this plugin does not know about our new Scala source files. We’ll be using the build-helper-maven-plugin to add new source folders:
...
org.codehaus.mojo</groupId>
build-helper-maven-pluginartifactId>
add-source</id>
generate-sourcesphase>
add-source</goal>
goals>
src/main/scala</source>
sources>
</configuration>
execution>
add-test-source</id>
generate-sourcesphase>
add-test-source</goal>
goals>
src/test/scala</source>
sources>
</configuration>
execution>
</executions>
plugin>
...
After adding that to your pom.xml
:
mvn eclipse:eclipse
- this generates the Eclipse project files (which are already ignored by our archetype’s .gitignore
)mvn -Declipse.workspace="path/to/your/eclipse/workspace" eclipse:configure-workspace
- this adds an M2_REPO
classpath variable to Eclipsemvn package
to ensure you have all the dependencies in your local Maven repo Unfortunately, the integration isn’t perfect. Firstly, open up the generated .classpath
file (it will be hidden by default as it’s a dotfile, but it should be in your project root directory; where you ran mvn eclipse:eclipse
). You should see something like this near the top.
kind="src" path="src/test/scala" output="target/test-classes" including="**/*.java"/>
kind="src" path="src/main/scala" including="**/*.java"/>
Change the *.java
to *.scala
(or duplicate the lines and change them to *.scala
if you also have Java sources).
Secondly, open the .project
eclipse file (again, in the same folder). Change
and
to look like this. Now Eclipse knows to use the Scala editor and it won’t think that everything is a syntax error.
org.scala-ide.sdt.core.scalabuilder
org.scala-ide.sdt.core.scalanature
org.eclipse.jdt.core.javanature
Finally, in Eclipse, under “File” choose “Import…” and find your project.
m2eclipse-scala is a work in progress, and their website/repository may have updated information. It aims to ease integration between m2eclipse and Scala IDE for Eclipse.
Under “Help -> Install New Software”, enter “http://alchim31.free.fr/m2e-scala/update-site” and hit enter. You should see “Maven Integration for Eclipse -> Maven Integration for Scala IDE”.
Afer it installs, go to “New -> Project -> Other” and select “Maven Project”. Search fo “scala-archetype” choose the one with the group “net.alchim31.maven”. The wizard will more or less guide you through what was done with mvn archetype:generate
above, and you should end up with a new Scala project!
To import an existing project, simply go to “File -> Import -> Existing Maven Project” and find the directory containing your project.
The first thing I do is look for “Maven” in the project page. For example, Google’s [Guava] page includes Maven Central links. As you can see in the previous link, The Central Repository conveniently includes the snippet you have to add to your pom.xml
on the left sidebar.
If you can’t find Maven information at the project page, try a Google search for “[project name] maven”. Sometimes, you still won’t find anything. For scopt (Scala command line option parser), I couldn’t find the latest version from Google. However, manually searching The Central Repository did
Afterwards, running
mvn package
Will download any new dependencies before packaging
I’m not going to explain all of Maven in this tutorial (though I hope to add more in the future, because I do feel that the resources are a bit scattered), so here are some useful articles: - Maven Lifecycle (explanation of goals like clean, package, install)
原文链接: http://docs.scala-lang.org/tutorials/scala-with-maven.html