springboot集成spark并使用spark-sql的示例详解

首先添加相关依赖：

  4.0.0
  
    org.springframework.boot
    spring-boot-starter-parent
    1.5.6.RELEASE
    
  
  com.cord
  spark-example
  1.0-SNAPSHOT
  spark-example
  
  http://www.example.com
  
    UTF-8
    UTF-8
    1.8
    2.10.3
    1.8
    1.8
  
  
    
      org.springframework.boot
      spring-boot-starter
      1.5.6.RELEASE
        
            
                org.springframework.boot
                spring-boot-starter-logging
            
        
    
    
      org.apache.spark
      spark-core_2.10
      1.6.1
      provided
        
            
                org.slf4j
                slf4j-log4j12
            
            
                log4j
                log4j
            
        
    
    
      org.apache.spark
      spark-sql_2.10
      1.6.1
      provided
    
    
      org.apache.spark
      spark-hive_2.10
      1.6.1
      provided
    
    
      org.scala-lang
      scala-library
      ${scala.version}
      provided
    
    
    
      mysql
      mysql-connector-java
      5.1.22
    
  
    
        
            
                org.apache.maven.plugins
                maven-shade-plugin
                
                    
                        org.springframework.boot
                        spring-boot-maven-plugin
                        1.5.6.RELEASE
                    
                
                
                    false
                    false
                    
                        
                            *:*
                            
                                META-INF/*.SF
                                META-INF/*.DSA
                                META-INF/*.RSA
                            
                        
                    
                    
                        
                            META-INF/spring.handlers
                        
                        
                            META-INF/spring.factories
                        
                        
                            META-INF/spring.schemas
                        
                        
                        
                            com.cord.StartApplication
                        
                    
                
                
                    
                        package
                        
                            shade
                        
                    
                
            
        
    

需要注意的是依赖中排除掉的日志模块，以及特殊的打包方式

定义配置类:

SparkContextBean.class

@Configuration
public class SparkContextBean {
    private String appName = "sparkExp";
    private String master = "local";
    @Bean
    @ConditionalOnMissingBean(SparkConf.class)
    public SparkConf sparkConf() throws Exception {
        SparkConf conf = new SparkConf().setAppName(appName).setMaster(master);
        return conf;
    }
    @Bean
    @ConditionalOnMissingBean
    public JavaSparkContext javaSparkContext() throws Exception {
        return new JavaSparkContext(sparkConf());
    }
    @Bean
    @ConditionalOnMissingBean
    public HiveContext hiveContext() throws Exception {
        return new HiveContext(javaSparkContext());
    }
    ......
}

启动类：

StartApplication.class

@SpringBootApplication
public class StartApplication implements CommandLineRunner {
    @Autowired
    private HiveContext hc;
    public static void main(String[] args) {
        SpringApplication.run(StartApplication.class, args);
    }
    @Override
    public void run(String... args) throws Exception {
        DataFrame df = hc.sql("select count(1) from LCS_DB.STAFF_INFO");
        List result = df.javaRDD().map((Function) row -> {
            return row.getLong(0);
        }).collect();
        result.stream().forEach(System.out::println);
}

执行方式：

spark-submit \
    --class com.cord.StartApplication  \
    --executor-memory 4G \
    --num-executors 8 \
    --master yarn-client \
/data/cord/spark-example-1.0-SNAPSHOT.jar

参考链接：

https://stackoverflow.com/questions/45189701/submitting-spring-boot-application-jar-to-spark-submit

https://my.oschina.net/woter/blog/1843755

到此这篇关于spring-boot集成spark并使用spark-sql的文章就介绍到这了,更多相关spring-boot集成spark内容请搜索脚本之家以前的文章或继续浏览下面的相关文章希望大家以后多多支持脚本之家！