【Hadoop】macbookpro m1/m2 arm 编译hadoop-3.3.1

在大数据数仓实践过程中，hadoop技术栈充当了非常重要的环节，很多大数据平台，中台都是基于hive+hadoop+spark来搭建的，所以我们对于hadoop的编译，构建，安装是需要掌握的非常清楚。

而我们大数据的研发人员比较常用mac电脑来开发，mac 最新的 m1/m2 arm版本的系统在编译hive/hadoop的过程中会遇到protobuf编译失败的问题。

下面就是在编译过程中，遇到的问题，希望能给大家带来清晰的指导

在arm上编译protoc，出现异常

[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.xolstice.maven.plugins:protobuf-maven-plugin:0.5.1:compile (default-cli) on project hadoop-hdfs: Missing:
[ERROR] ----------
[ERROR] 1) com.google.protobuf:protoc:exe:osx-aarch_64:3.7.1
[ERROR] 
[ERROR]   Try downloading the file manually from the project website.
[ERROR] 
[ERROR]   Then, install it using the command: 
[ERROR]       mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.7.1 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file
[ERROR] 
[ERROR]   Alternatively, if you host your own repository you can deploy the file there: 
[ERROR]       mvn deploy:deploy-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.7.1 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR] 
[ERROR]   Path to dependency: 
[ERROR]   	1) org.apache.hadoop:hadoop-hdfs:jar:3.3.1
[ERROR]   	2) com.google.protobuf:protoc:exe:osx-aarch_64:3.7.1
[ERROR] 
[ERROR] ----------
[ERROR] 1 required artifact is missing.
[ERROR] 
[ERROR] for artifact: 
[ERROR]   org.apache.hadoop:hadoop-hdfs:jar:3.3.1
[ERROR] 
[ERROR] from the specified remote repositories:
[ERROR]   aliyunmaven (https://maven.aliyun.com/repository/spring-plugin, releases=true, snapshots=true),
[ERROR]   alimaven (http://maven.aliyun.com/nexus/content/groups/public/, releases=true, snapshots=false)
[ERROR] 

于是去官方的issue上，找到了相关的问题，目前没有兼容m1芯片的版本，需要手动指定x86_64的版本

<plugin>
  <groupId>org.xolstice.maven.pluginsgroupId>
  <artifactId>protobuf-maven-pluginartifactId>
  <version>${protobuf-maven-plugin.version}version>
  <extensions>trueextensions>
  <configuration>
    <protocArtifact>
      com.google.protobuf:protoc:${hadoop.protobuf.version}:exe:${os.detected.classifier}
    protocArtifact>
    <attachProtoSources>falseattachProtoSources>
  configuration>
  <executions>
    <execution>
      <id>src-compile-protocid>
      <phase>generate-sourcesphase>
      <goals>
        <goal>compilegoal>
      goals>
      <configuration>
        <includeDependenciesInDescriptorSet>falseincludeDependenciesInDescriptorSet>
        <protoSourceRoot>${basedir}/src/main/protoprotoSourceRoot>
        <outputDirectory>${project.build.directory}/generated-sources/javaoutputDirectory>
        <clearOutputDirectory>falseclearOutputDirectory>
        <skip>trueskip>
      configuration>
    execution>
    <execution>
      <id>src-test-compile-protocid>
      <phase>generate-test-sourcesphase>
      <goals>
        <goal>test-compilegoal>
      goals>
      <configuration>
        <protoTestSourceRoot>${basedir}/src/test/protoprotoTestSourceRoot>
        <outputDirectory>${project.build.directory}/generated-test-sources/javaoutputDirectory>
        <clearOutputDirectory>falseclearOutputDirectory>
        <skip>trueskip>
      configuration>
    execution>
  executions>
plugin>

修改插件的 ${os.detected.classifier} 值 为 osx-x86_64

<plugin>
  <groupId>org.xolstice.maven.pluginsgroupId>
  <artifactId>protobuf-maven-pluginartifactId>
  <version>${protobuf-maven-plugin.version}version>
  <extensions>trueextensions>
  <configuration>
    <protocArtifact>
      com.google.protobuf:protoc:${hadoop.protobuf.version}:exe:osx-x86_64
    protocArtifact>
    <attachProtoSources>falseattachProtoSources>
  configuration>
  <executions>
    <execution>
      <id>src-compile-protocid>
      <phase>generate-sourcesphase>
      <goals>
        <goal>compilegoal>
      goals>
      <configuration>
        <includeDependenciesInDescriptorSet>falseincludeDependenciesInDescriptorSet>
        <protoSourceRoot>${basedir}/src/main/protoprotoSourceRoot>
        <outputDirectory>${project.build.directory}/generated-sources/javaoutputDirectory>
        <clearOutputDirectory>falseclearOutputDirectory>
        <skip>trueskip>
      configuration>
    execution>
    <execution>
      <id>src-test-compile-protocid>
      <phase>generate-test-sourcesphase>
      <goals>
        <goal>test-compilegoal>
      goals>
      <configuration>
        <protoTestSourceRoot>${basedir}/src/test/protoprotoTestSourceRoot>
        <outputDirectory>${project.build.directory}/generated-test-sources/javaoutputDirectory>
        <clearOutputDirectory>falseclearOutputDirectory>
        <skip>trueskip>
      configuration>
    execution>
  executions>
plugin>

希望对正在查看文章的您有所帮助，记得关注、评论、收藏，谢谢您