elasticsearch 5.2.2 学习笔记之源码阅读1-构建源码与核心模块/类

elasticsearch 5.2.2 学习笔记之源码阅读1-构建源码与核心模块/类

概述

• 参考
  • https://github.com/elastic/elasticsearch/tree/v5.2.2
  • https://www.jianshu.com/p/a22492d40fd1 (----可按照此操作,只需注意gradle版本即可)
  • https://www.cnblogs.com/zziawanblog/p/6493212.html (----版本较低(有部分参考意义))

编译源码,可能踩的坑

• 编译的时候一定要去https://github.com/elastic/elasticsearch/tree/v5.2.2 查看对应版本的gradle
  • eg:5.2.2对应的是You’ll need to have version 2.13 of Gradle installed,安装2.13即可编译成功,高版本的gradle会有各种问题

核心服务类

模块设计

• 使用guice进行依赖管理,分模块进行管理依赖
• Module
  • AbstractModule
    • NodeModule
    • ActionModule(elasticsearch的每一个功能都对应三个个Action,eg:Indexaction
      和TransportIndexAction,RestIndexAction)
    • RepositoriesModule
      • for Snapshot/Restore
    • GatewayModule
    • IndicesModule
      • shared by indices on each node
    • ClusterModule
      • affect the entire cluster
  • ScriptModule
  • SearchModule
    • search time like queries, aggregations, and suggesters

查询设计

• QueryBuilder
  • AbstractQueryBuilder
    • QueryStringQueryBuilder
  • MultiTermQueryBuilder
  • SpanQueryBuilder

核心接口与类

• 使用AbstractComponent和AbstractLifecycleComponent管理核心组件的配置与生命周期
• Releasable(Closeable)
  • LifecycleComponent(start/stop/addLifecycleListener)
    • HttpServerTransport
      • Netty4HttpServerTransport
    • Discovery
    • Repository
    • Transport
  • AbstractComponent
    • PluginsService
    • NetworkService
    • JvmService
    • FaultDetection
      • 确保节点存活
      • MasterFaultDetection/NodesFaultDetection
    • NodeJoinController
      • ElectMasterService#electMaster(clusterStateVersion降序排列后master优于other nodes后nodeId)
    • TransportAction
    • TaskManager
      • track of running tasks on the nodes
    • ElasticsearchClient/AbstractClient
      • Client/ClusterAdminClient/IndicesAdminClient
      • TransportClient
      • NodeClient
        • actions on the local node
    • AbstractLifecycleComponent
      • ClusterService
      • SearchService
      • HttpServer
        • serve http requests, backed by rest handlers
        • 委托给RestController#dispatchRequest处理
      • RoutingService
      • GatewayService
      • MonitorService
      • TribeService
      • NodeConnectionsService
        • for connecting to nodes once they are added to the cluster state
      • CircuitBreakerService
      • ZenDiscovery
      • HttpServerTransport/Netty4HttpServerTransport
      • TransportService
        • 利用Transport进行通信
      • TcpTransport(Transport)
        • 用于节点/集群间相互通信
        • Netty4Transport(委托netty创建server/client)
        • 先写19字节TcpHeader头/channel.writeAndFlush
  • IndexComponent
    • AbstractIndexComponent
      • IndexService
      • IndexStore
      • IndexAnalyzers
      • IndexQueryCache
      • IndexCache
      • SimilarityService
      • MapperService
      • AbstractIndexAnalyzerProvider
      • AbstractCharFilterFactory
      • AbstractTokenFilterFactory
      • AbstractTokenizerFactory

Elasticsearch启动过程分析

• org.elasticsearch.bootstrap.Elasticsearch#main
  • 委托给org.elasticsearch.bootstrap.Bootstrap#init进行初始化
    • 进行必要的环境、版本等检查,并进行初始化,创建Node
      • 创建NodeEnvironment环境信息
      • 创建PluginsService/ClusterService/等,加载各种模块SearchModule/ActionModule等
    • 启动node(org.elasticsearch.node.Node#start)
    • 启动过程可以参看日志输出去追踪启动过程,这也是个不错的办法

"D:\Program Files\Java\jdk1.8.0_65\bin\java" -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:49951,suspend=y,server=n -Dlog4j2.disable.jmx=true -Des.path.home=E:\code\github\elasticsearch\core -javaagent:C:\Users\Administrator\.IntelliJIdea2017.2\system\groovyHotSwap\gragent.jar -Dfile.encoding=UTF-8 -classpath "D:\Program Files\Java\jdk1.8.0_65\jre\lib\charsets.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\deploy.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\ext\access-bridge-64.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\ext\cldrdata.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\ext\dnsns.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\ext\jaccess.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\ext\jfxrt.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\ext\localedata.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\ext\nashorn.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\ext\sunec.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\ext\sunjce_provider.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\ext\sunmscapi.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\ext\sunpkcs11.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\ext\zipfs.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\javaws.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\jce.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\jfr.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\jfxswt.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\jsse.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\management-agent.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\plugin.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\resources.jar;D:\Program Files\Java\jdk1.8.0_65\jre\lib\rt.jar;E:\code\github\elasticsearch\core\build-idea\classes\main;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\6.4.1\2a18924b9e0ed86b318902cb475a0b9ca4d7be5b\lucene-core-6.4.1.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\6.4.1\c6f0f593503080204e9d33189cdc59320f55db37\lucene-analyzers-common-6.4.1.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\6.4.1\b0ab8aca2b0025b8733411778b6d27afe1c451f5\lucene-backward-codecs-6.4.1.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\6.4.1\8297adfa469abd571079ee75a1645fc5124fff5b\lucene-grouping-6.4.1.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\6.4.1\95c0f76fc7893240483b25a5c420bed77a10d05d\lucene-highlighter-6.4.1.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\6.4.1\6d2f1ff94dba19b9bc6f15930b7104b890cab1ce\lucene-join-6.4.1.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\6.4.1\825a946902f03a38257851733da908949d69f3da\lucene-memory-6.4.1.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\6.4.1\4d147a6f0fcfc54630260d1bb3deecfc0d0d10f7\lucene-misc-6.4.1.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\6.4.1\6de41d984c16185a244b52c4d069b00f5b2b120f\lucene-queries-6.4.1.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\6.4.1\1fc5795a072770a2c47dce11a3c85a80f3437af6\lucene-queryparser-6.4.1.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\6.4.1\d4a49664668c58aa23aba74717f3d74a61378d7c\lucene-sandbox-6.4.1.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\6.4.1\d6ceb47fdea913a8f468a240a6ea307368094463\lucene-spatial-6.4.1.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\6.4.1\5b764e5f95f610f79622ca0e957bfc23f094e4c7\lucene-spatial-extras-6.4.1.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\6.4.1\2c58459e671040046305dbee698fce1e6a1de71d\lucene-spatial3d-6.4.1.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\6.4.1\47014b400916eea259645a9e3118558ef6e95441\lucene-suggest-6.4.1.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securesm\1.1\1e423447d020041534be94c0f31a49fbdc1f2950\securesm-1.1.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.5\5f01da7306363fad2028b916f3eab926262de928\joda-time-2.9.5.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.15\3b132bea69e8ee099f416044970997bde80f4ea6\snakeyaml-1.15.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.6\2ef7b1cc34de149600f5e75bc2d5bf40de894e60\jackson-core-2.8.6.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.6\71590ad45cee21249774e2f93e5eca66e446cef3\jackson-dataformat-smile-2.8.6.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.6\8bd44d50f9a6cdff9c7578ea39d524eb519e35ab\jackson-dataformat-yaml-2.8.6.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.6\b88721371cfa2d7242bb5e52fe70861aa061c050\jackson-dataformat-cbor-2.8.6.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.0\84ccf145ac2215e6bfa63baa3101c0af41017cfc\t-digest-3.0.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.6\7495feb7f71ee124bd2a7e7d83590e296d71d80e\HdrHistogram-2.1.6.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.6\21b15310bddcfd8c72611c180f20cf23279809a3\spatial4j-0.6.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\com.vividsolutions\jts\1.13\3ccfb9b60f04d71add996a666ceb8902904fd805\jts-1.13.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.7\8de00e382a817981b737be84cb8def687d392963\log4j-api-2.7.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.7\a3f2b4e64c61a7fc1ed8f1e5ba371933404ed98a\log4j-core-2.7.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.7\39f4e6c2d68d4ef8fd4b0883d165682dedd5be52\log4j-1.2-api-2.7.jar;C:\Users\Administrator\.gradle\caches\modules-2\files-2.1\net.java.dev.jna\jna\4.2.2\5012450aee579c3118ff09461d5ce210e0cdc2a9\jna-4.2.2.jar;D:\Program Files\JetBrains\IntelliJ IDEA 2017.2.3\lib\idea_rt.jar" org.elasticsearch.bootstrap.Elasticsearch
Connected to the target VM, address: '127.0.0.1:49951', transport: 'socket'
[2018-09-26T09:53:22,034][INFO ][o.e.n.Node               ] [] initializing ...
[2018-09-26T09:53:25,932][INFO ][o.e.e.NodeEnvironment    ] [KGDjKo7] using [1] data paths, mounts [[(E:)]], net usable_space [143.6gb], net total_space [233gb], spins? [unknown], types [NTFS]
[2018-09-26T09:53:25,933][INFO ][o.e.e.NodeEnvironment    ] [KGDjKo7] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-09-26T09:53:25,934][INFO ][o.e.n.Node               ] node name [KGDjKo7] derived from node ID [KGDjKo7fQG-xJnVpmi4qQw]; set [node.name] to override
[2018-09-26T09:53:25,936][INFO ][o.e.n.Node               ] version[5.2.2-SNAPSHOT], pid[5696], build[Unknown/Unknown], OS[Windows 7/6.1/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_65/25.65-b01]
[2018-09-26T09:53:25,936][WARN ][o.e.n.Node               ] version [5.2.2-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
[2018-09-26T09:53:35,404][INFO ][o.e.p.PluginsService     ] [KGDjKo7] loaded module [aggs-matrix-stats]
[2018-09-26T09:53:35,405][INFO ][o.e.p.PluginsService     ] [KGDjKo7] loaded module [ingest-common]
[2018-09-26T09:53:35,405][INFO ][o.e.p.PluginsService     ] [KGDjKo7] loaded module [lang-expression]
[2018-09-26T09:53:35,405][INFO ][o.e.p.PluginsService     ] [KGDjKo7] loaded module [lang-groovy]
[2018-09-26T09:53:35,405][INFO ][o.e.p.PluginsService     ] [KGDjKo7] loaded module [lang-mustache]
[2018-09-26T09:53:35,405][INFO ][o.e.p.PluginsService     ] [KGDjKo7] loaded module [lang-painless]
[2018-09-26T09:53:35,405][INFO ][o.e.p.PluginsService     ] [KGDjKo7] loaded module [percolator]
[2018-09-26T09:53:35,406][INFO ][o.e.p.PluginsService     ] [KGDjKo7] loaded module [reindex]
[2018-09-26T09:53:35,406][INFO ][o.e.p.PluginsService     ] [KGDjKo7] loaded module [transport-netty3]
[2018-09-26T09:53:35,406][INFO ][o.e.p.PluginsService     ] [KGDjKo7] loaded module [transport-netty4]
[2018-09-26T09:53:35,407][INFO ][o.e.p.PluginsService     ] [KGDjKo7] loaded plugin [analysis-ik]
[2018-09-26T09:53:48,420][INFO ][o.e.n.Node               ] initialized
[2018-09-26T09:53:51,433][INFO ][o.e.n.Node               ] [KGDjKo7] starting ...
[2018-09-26T09:53:51,848][INFO ][i.n.u.i.PlatformDependent] Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system unstability.
[2018-09-26T09:53:52,531][INFO ][o.e.t.TransportService   ] [KGDjKo7] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2018-09-26T09:53:54,796][WARN ][o.e.b.BootstrapChecks    ] [KGDjKo7] initial heap size [132120576] not equal to maximum heap size [2111832064]; this can cause resize pauses and prevents mlockall from locking the entire heap
[2018-09-26T09:54:01,373][INFO ][o.e.c.s.ClusterService   ] [KGDjKo7] new_master {KGDjKo7}{KGDjKo7fQG-xJnVpmi4qQw}{R-HsP79lRG2eDodz5RUxjA}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-09-26T09:54:16,223][INFO ][o.e.g.GatewayService     ] [KGDjKo7] recovered [0] indices into cluster_state
[2018-09-26T09:54:16,392][INFO ][o.e.h.HttpServer         ] [KGDjKo7] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2018-09-26T09:54:16,392][INFO ][o.e.n.Node               ] [KGDjKo7] started

• 后续待更新
  • TransportService(利用Transport进行数据传输,使用netty作为数据传输层,后面会详细讲解数据传输)
  • HttpServerTransport(使用netty接收用户请求,转发给HttpServer#dispatchRequest,进而利用RestController的RestHandler处理相应的用户请求,后面也会详细讲解)
  • 插件机制/搜索流程/索引流程也都会讲解