鍓嶈█
鍙湁鍏夊ご鎵嶈兘鍙樺己銆�
鏂囨湰宸叉敹褰曡嚦鎴戠殑GitHub浠撳簱锛屾杩嶴tar锛�https://github.com/ZhongFuCheng3y/3y
鍦ㄨ繖绡囦箣鍓嶅凡缁忓啓杩囦袱绡囧熀纭�鏂囩珷浜嗭紝寮虹儓寤鸿鍏堝幓闃呰锛�
- 浠�涔堟槸ZooKeeper锛�
- 浠�涔堟槸娑堟伅闃熷垪锛�
浼楁墍鍛ㄧ煡锛屾秷鎭槦鍒楃殑浜у搧鏈夊ソ鍑犵锛岃繖閲屾垜閫夋嫨瀛︿範Kafka鐨勫師鍥狅紝鏃犱粬锛屽叕鍙稿湪鐢ㄣ��
鎴戝徃浣跨敤鐨勬槸Kafka鍜岃嚜鐮旂殑娑堟伅闃熷垪(Kafka鍜孯ocketMQ)鏀圭増锛屼簬鏄垜灏辨兂瀛﹀Kafka杩欐娑堟伅闃熷垪鍟︺�傛湰绡囨枃绔犲Kafka鍏ラ棬锛屽笇鏈涘澶у鏈夋墍甯姪銆�
鏈枃鐭ヨ瘑鐐规彁鍓嶉瑙堬細
杩欑瘒鏂囩珷鑺变簡鎴戝緢闀挎椂闂寸敾鍥撅紝鐩殑鏄笇鏈涗互鏈�閫氫織鏄撴噦鐨勬柟寮忓甫澶у鍏ラ棬锛屽鏋滆寰椾笉閿欙紝甯屾湜鑳界粰鎴戠偣涓禐锛�
涓�銆佷粈涔堟槸Kafka锛�
棣栧厛鎴戜滑寰楀幓瀹樼綉鐪嬬湅鏄�庝箞浠嬬粛Kafka鐨勶細
- https://kafka.apache.org/intro
鍦ㄦ敹闆嗚祫鏂欏涔犵殑鏃跺�欙紝宸茬粡鍙戠幇鏈変笉灏戠殑鍓嶈緢瀵瑰畼缃戠殑浠嬬粛杩涜缈昏瘧鍜屾�荤粨浜嗭紝鎵�浠ユ垜杩欓噷灏变笉閲嶅浜嗭紝璐翠笅鍦板潃澶у鑷鍘诲涔犲暒锛�
- https://scala.cool/2018/03/learning-kafka-1/
- https://colobu.com/2014/08/06/kafka-quickstart/
鎴戜箣鍓嶅啓杩囩殑娑堟伅闃熷垪鍏ラ棬鏂囩珷涔熸彁鍒颁簡锛岃鍋氫竴涓秷鎭槦鍒楀彲鑳借鑰冭檻鍒颁互涓嬬殑闂锛�
- 浣跨敤娑堟伅闃熷垪涓嶅彲鑳芥槸鍗曟満鐨勶紙蹇呯劧鏄垎甯冨紡or闆嗙兢锛�
- 鏁版嵁鍐欏埌娑堟伅闃熷垪锛屽彲鑳戒細瀛樺湪鏁版嵁涓㈠け闂锛屾暟鎹湪娑堟伅闃熷垪闇�瑕�鎸佷箙鍖�(纾佺洏锛熸暟鎹簱锛烺edis锛熷垎甯冨紡鏂囦欢绯荤粺锛�)
- 鎯宠淇濊瘉娑堟伅锛堟暟鎹級鏄湁搴忕殑锛屾�庝箞鍋氾紵
- 涓轰粈涔堝湪娑堟伅闃熷垪涓噸澶嶆秷璐逛簡鏁版嵁
涓嬮潰鎴戜互Kafka涓轰緥瀵硅繖浜涢棶棰樿繘琛岀畝鍗曠殑瑙ｇ瓟锛岃繘鑰屽叆闂↘afka銆�
1.1 Kafka鍏ラ棬
浼楁墍鍛ㄧ煡锛孠afka鏄竴涓秷鎭槦鍒楋紝鎶婃秷鎭斁鍒伴槦鍒楅噷杈圭殑鍙�鐢熶骇鑰�锛屼粠闃熷垪閲岃竟娑堣垂鐨勫彨娑堣垂鑰�銆�
涓�涓秷鎭腑闂翠欢锛岄槦鍒椾笉鍗曞崟鍙湁涓�涓紝鎴戜滑寰�寰�浼氭湁澶氫釜闃熷垪锛岃�屾垜浠敓浜ц�呭拰娑堣垂鑰呭氨寰楃煡閬擄細鎶婃暟鎹涪缁欏摢涓槦鍒楋紝浠庡摢涓槦鍒楁秷鎭�傛垜浠渶瑕佺粰闃熷垪鍙栧悕瀛楋紝鍙仛topic(鐩稿綋浜庢暟鎹簱閲岃竟琛�鐨勬蹇�)
鐜板湪鎴戜滑缁欓槦鍒楀彇浜嗗悕瀛椾互鍚庯紝鐢熶骇鑰呭氨鐭ラ亾寰�鍝釜闃熷垪涓㈡暟鎹簡锛屾秷璐硅�呬篃鐭ラ亾寰�鍝釜闃熷垪鎷挎暟鎹簡銆傛垜浠彲浠ユ湁澶氫釜鐢熶骇鑰�寰�鍚屼竴涓槦鍒�(topic)涓㈡暟鎹紝澶氫釜娑堣垂鑰�寰�鍚屼竴涓槦鍒�(topic)鎷挎暟鎹�
涓轰簡鎻愰珮涓�涓槦鍒�(topic)鐨�鍚炲悙閲�锛孠afka浼氭妸topic杩涜鍒嗗尯(Partition)
鎵�浠ワ紝鐢熶骇鑰呭疄闄呬笂鏄線涓�涓猼opic鍚嶄负Java3y涓殑鍒嗗尯(Partition)涓㈡暟鎹紝娑堣垂鑰呭疄闄呬笂鏄線涓�涓猼opic鍚嶄负Java3y鐨勫垎鍖�(Partition)鍙栨暟鎹�
涓�鍙癒afka鏈嶅姟鍣ㄥ彨鍋�Broker锛孠afka闆嗙兢灏辨槸澶氬彴Kafka鏈嶅姟鍣細
涓�涓猼opic浼氬垎涓哄涓猵artition锛屽疄闄呬笂partition浼�鍒嗗竷鍦ㄤ笉鍚岀殑broker涓紝涓句釜渚嬪瓙锛�
鐢辨寰楃煡锛�Kafka鏄ぉ鐒跺垎甯冨紡鐨�銆�
濡傛灉涓嶄簡瑙ｅ垎甯冨紡/闆嗙兢锛屼互鍙婂熀鏈殑鍒嗗竷寮忔蹇电殑鍚屽锛屽彲浠ュ叧娉ㄦ垜鐨凣itHub锛�https://github.com/ZhongFuCheng3y/3y
鍏抽敭瀛楋細鍒嗗竷寮忋�丼pringCloud 淇濊瘉鑳借浣犳悶鎳傘�傝寰楁垜鍐欏緱涓嶉敊锛屽氨缁欐垜鐐逛釜璧�锛�
鐜板湪鎴戜滑宸茬粡鐭ラ亾浜嗗線topic閲岃竟涓㈡暟鎹紝瀹為檯涓婅繖浜涙暟鎹細鍒嗗埌涓嶅悓鐨刾artition涓婏紝杩欎簺partition瀛樺湪涓嶅悓鐨刡roker涓娿�傚垎甯冨紡鑲畾浼氬甫鏉ラ棶棰橈細鈥滀竾涓�鍏朵腑涓�鍙癰roker(Kafka鏈嶅姟鍣�)鍑虹幇缃戠粶鎶栧姩鎴栬�呮寕浜嗭紝鎬庝箞鍔烇紵鈥�
Kafka鏄繖鏍峰仛鐨勶細鎴戜滑鏁版嵁瀛樺湪涓嶅悓鐨刾artition涓婏紝閭afka灏辨妸杩欎簺partition鍋�澶囦唤銆傛瘮濡傦紝鐜板湪鎴戜滑鏈変笁涓猵artition锛屽垎鍒瓨鍦ㄤ笁鍙癰roker涓娿�傛瘡涓猵artition閮戒細澶囦唤锛岃繖浜涘浠芥暎钀藉湪涓嶅悓鐨刡roker涓娿��
绾㈣壊鍧楃殑partition浠ｈ〃鐨勬槸涓�鍒嗗尯锛岀传鑹茬殑partition鍧椾唬琛ㄧ殑鏄�澶囦唤鍒嗗尯銆傜敓浜ц�呭線topic涓㈡暟鎹紝鏄笌涓�鍒嗗尯浜や簰锛屾秷璐硅�呮秷璐箃opic鐨勬暟鎹紝涔熸槸涓庝富鍒嗗尯浜や簰銆�
澶囦唤鍒嗗尯浠呬粎鐢ㄤ綔浜庡浠斤紝涓嶅仛璇诲啓銆�濡傛灉鏌愪釜Broker鎸備簡锛岄偅灏变細閫変妇鍑哄叾浠朆roker鐨刾artition鏉ヤ綔涓轰富鍒嗗尯锛岃繖灏卞疄鐜颁簡楂樺彲鐢�銆�
鍙﹀鍊煎緱涓�鎻愮殑鏄細褰撶敓浜ц�呮妸鏁版嵁涓㈣繘topic鏃讹紝鎴戜滑鐭ラ亾鏄啓鍦╬artition涓婄殑锛岄偅partition鏄�庝箞灏嗗叾鎸佷箙鍖栫殑鍛紵锛堜笉鎸佷箙鍖栧鏋淏roker涓�旀寕浜嗭紝閭ｈ偗瀹氫細涓㈡暟鎹槢)銆�
Kafka鏄皢partition鐨勬暟鎹啓鍦�纾佺洏鐨�(娑堟伅鏃ュ織)锛屼笉杩嘖afka鍙厑璁�杩藉姞鍐欏叆(椤哄簭璁块棶)锛岄伩鍏嶇紦鎱㈢殑闅忔満 I/O 鎿嶄綔銆�
- Kafka涔熶笉鏄痯artition涓�鏈夋暟鎹氨绔嬮┈灏嗘暟鎹啓鍒扮鐩樹笂锛屽畠浼氬厛缂撳瓨涓�閮ㄥ垎锛岀瓑鍒拌冻澶熷鏁版嵁閲忔垨绛夊緟涓�瀹氱殑鏃堕棿鍐嶆壒閲忓啓鍏�(flush)銆�
涓婇潰balabala鍦伴兘鏄鐢熶骇鑰呮妸鏁版嵁涓㈣繘topic鏄�庝箞鏍风殑锛屼笅闈㈡潵璁茶娑堣垂鑰呮槸鎬庝箞娑堣垂鐨勩�傛棦鐒舵暟鎹槸淇濆瓨鍦╬artition涓殑锛岄偅涔�娑堣垂鑰呭疄闄呬笂涔熸槸浠巔artition涓彇鏁版嵁銆�
鐢熶骇鑰呭彲浠ユ湁澶氫釜锛屾秷璐硅�呬篃鍙互鏈夊涓�傚儚涓婇潰鍥剧殑鎯呭喌锛屾槸涓�涓秷璐硅�呮秷璐逛笁涓垎鍖虹殑鏁版嵁銆傚涓秷璐硅�呭彲浠ョ粍鎴愪竴涓�娑堣垂鑰呯粍銆�
鏈潵鏄竴涓秷璐硅�呮秷璐逛笁涓垎鍖虹殑锛岀幇鍦ㄦ垜浠湁娑堣垂鑰呯粍锛屽氨鍙互姣忎釜娑堣垂鑰呭幓娑堣垂涓�涓垎鍖�锛堜篃鏄负浜嗘彁楂樺悶鍚愰噺锛�
鎸夊浘涓婃墍绀虹殑鎯呭喌锛岃繖閲屾兂瑕佽鏄庣殑鏄細
- 濡傛灉娑堣垂鑰呯粍涓殑鏌愪釜娑堣垂鑰呮寕浜嗭紝閭ｄ箞鍏朵腑涓�涓秷璐硅�呭彲鑳藉氨瑕佹秷璐逛袱涓猵artition浜�
- 濡傛灉鍙湁涓変釜partition锛岃�屾秷璐硅�呯粍鏈�4涓秷璐硅�咃紝閭ｄ箞涓�涓秷璐硅�呬細绌洪棽
- 濡傛灉澶氬姞鍏ヤ竴涓�娑堣垂鑰呯粍锛屾棤璁烘槸鏂板鐨勬秷璐硅�呯粍杩樻槸鍘熸湰鐨勬秷璐硅�呯粍锛岄兘鑳芥秷璐箃opic鐨勫叏閮ㄦ暟鎹�傦紙娑堣垂鑰呯粍涔嬮棿浠庨�昏緫涓婂畠浠槸鐙珛鐨勶級
鍓嶉潰璁茶В鍒颁簡鐢熶骇鑰呭線topic閲屼涪鏁版嵁鏄瓨鍦╬artition涓婄殑锛岃�宲artition鎸佷箙鍖栧埌纾佺洏鏄疘O椤哄簭璁块棶鐨勶紝骞朵笖鏄厛鍐欑紦瀛橈紝闅斾竴娈垫椂闂存垨鑰呮暟鎹噺瓒冲澶х殑鏃跺�欐墠鎵归噺鍐欏叆纾佺洏鐨勩��
娑堣垂鑰呭湪璇荤殑鏃跺�欎篃寰堟湁璁茬┒锛氭甯哥殑璇荤鐩樻暟鎹槸闇�瑕佸皢鍐呮牳鎬佹暟鎹嫹璐濆埌鐢ㄦ埛鎬佺殑锛岃�孠afka 閫氳繃璋冪敤sendfile()鐩存帴浠庡唴鏍哥┖闂达紙DMA鐨勶級鍒板唴鏍哥┖闂达紙Socket鐨勶級锛�灏戝仛浜嗕竴姝ユ嫹璐�鐨勬搷浣溿��
鏈夌殑鍚屽鍙兘浼氫骇鐢熺枒闂細娑堣垂鑰呮槸鎬庝箞鐭ラ亾鑷繁娑堣垂鍒板摢閲岀殑鍛�锛烱afka涓嶆槸鏀寔鍥炴函鍚楋紵閭ｆ槸鎬庝箞鍋氱殑鍛�锛�
- 姣斿涓婇潰涔熸彁鍒帮細濡傛灉涓�涓秷璐硅�呯粍涓殑鏌愪釜娑堣垂鑰呮寕浜嗭紝閭ｆ寕鎺夌殑娑堣垂鑰呮墍娑堣垂鐨勫垎鍖哄彲鑳藉氨鐢卞瓨娲荤殑娑堣垂鑰呮秷璐广�傞偅瀛樻椿鐨勬秷璐硅�呮槸闇�瑕佺煡閬撴寕鎺夌殑娑堣垂鑰呮秷璐瑰埌鍝簡锛屼笉鐒舵�庝箞鐜┿��
杩欓噷瑕佸紩鍑�offset浜嗭紝Kafka灏辨槸鐢�offset鏉ヨ〃绀烘秷璐硅�呯殑娑堣垂杩涘害鍒板摢浜嗭紝姣忎釜娑堣垂鑰呬細閮芥湁鑷繁鐨�offset銆傝鐧戒簡offset灏辨槸琛ㄧず娑堣垂鑰呯殑娑堣垂杩涘害銆�
鍦ㄤ互鍓嶇増鏈殑Kafka锛岃繖涓�offset鏄敱Zookeeper鏉ョ鐞嗙殑锛屽悗鏉afka寮�鍙戣�呰涓篫ookeeper涓嶅悎閫傚ぇ閲忕殑鍒犳敼鎿嶄綔锛屼簬鏄妸offset鍦╞roker浠ュ唴閮╰opic(__consumer_offsets)鐨勬柟寮忔潵淇濆瓨璧锋潵銆�
姣忔娑堣垂鑰呮秷璐圭殑鏃跺�欙紝閮戒細鎻愪氦杩欎釜offset锛孠afka鍙互璁╀綘閫夋嫨鏄嚜鍔ㄦ彁浜よ繕鏄墜鍔ㄦ彁浜ゃ��
鏃㈢劧鎻愬埌浜哯ookeeper锛岄偅灏卞璇翠竴鍙ャ�俍ookeeper铏界劧鍦ㄦ柊鐗堢殑Kafka涓病鏈夌敤浣滀簬淇濆瓨瀹㈡埛绔殑offset锛屼絾鏄痁ookeeper鏄疜afka涓�涓噸瑕佺殑渚濊禆銆�
- 鎺㈡祴broker鍜宑onsumer鐨勬坊鍔犳垨绉婚櫎銆�
- 璐熻矗缁存姢鎵�鏈塸artition鐨勯瀵艰��/浠庡睘鑰呭叧绯伙紙涓诲垎鍖哄拰澶囦唤鍒嗗尯锛夛紝濡傛灉涓诲垎鍖烘寕浜嗭紝闇�瑕侀�変妇鍑哄浠藉垎鍖轰綔涓轰富鍒嗗尯銆�
- 缁存姢topic銆乸artition绛夊厓閰嶇疆淇℃伅
- ....
鏈�鍚�
閫氳繃杩欑瘒鏂囩珷锛屾枃绔犲紑澶撮偅鍑犱釜闂浼拌澶氬灏戝皯閮芥噦涓�浜涘暒銆傛垜鏉ョ畝瑕佸洖绛斾竴涓嬶細
浣跨敤娑堟伅闃熷垪涓嶅彲鑳芥槸鍗曟満鐨勶紙蹇呯劧鏄垎甯冨紡or闆嗙兢锛�
Kafka澶╃劧鏄垎甯冨紡鐨勶紝寰�涓�涓猼opic涓㈡暟鎹紝瀹為檯涓婂氨鏄線澶氫釜broker鐨刾artition瀛樺偍鏁版嵁
鏁版嵁鍐欏埌娑堟伅闃熷垪锛屽彲鑳戒細瀛樺湪鏁版嵁涓㈠け闂锛屾暟鎹湪娑堟伅闃熷垪闇�瑕�鎸佷箙鍖�(纾佺洏锛熸暟鎹簱锛烺edis锛熷垎甯冨紡鏂囦欢绯荤粺锛�)
Kafka浼氬皢partition浠ユ秷鎭棩蹇楃殑鏂瑰紡(钀界鐩�)瀛樺偍璧锋潵锛岄�氳繃 椤哄簭璁块棶IO鍜岀紦瀛�(绛夊埌涓�瀹氱殑閲忔垨鏃堕棿)鎵嶇湡姝ｆ妸鏁版嵁鍐欏埌纾佺洏涓婏紝鏉ユ彁楂橀�熷害銆�
鎯宠淇濊瘉娑堟伅锛堟暟鎹級鏄湁搴忕殑锛屾�庝箞鍋氾紵
Kafka浼氬皢鏁版嵁鍐欏埌partition锛屽崟涓猵artition鐨勫啓鍏ユ槸鏈夐『搴忕殑銆傚鏋滆淇濊瘉鍏ㄥ眬鏈夊簭锛岄偅鍙兘鍐欏叆涓�涓猵artition涓�傚鏋滆娑堣垂涔熸湁搴忥紝娑堣垂鑰呬篃鍙兘鏈変竴涓��
涓轰粈涔堝湪娑堟伅闃熷垪涓噸澶嶆秷璐逛簡鏁版嵁
鍑℃槸鍒嗗竷寮忓氨鏃犳硶閬垮厤缃戠粶鎶栧姩/鏈哄櫒瀹曟満绛夐棶棰樼殑鍙戠敓锛屽緢鏈夊彲鑳芥秷璐硅�匒璇诲彇浜嗘暟鎹紝杩樻病鏉ュ緱鍙婃秷璐癸紝灏辨寕鎺変簡銆俍ookeeper鍙戠幇娑堣垂鑰匒鎸備簡锛岃娑堣垂鑰匓鍘绘秷璐瑰師鏈秷璐硅�匒鐨勫垎鍖猴紝绛夋秷璐硅�匒閲嶈繛鐨勬椂鍊欙紝鍙戠幇宸茬粡閲嶅娑堣垂鍚屼竴鏉℃暟鎹簡銆�(鍚勭鍚勬牱鐨勬儏鍐碉紝娑堣垂鑰呰秴鏃剁瓑绛夐兘鏈夊彲鑳�...)
濡傛灉涓氬姟涓婁笉鍏佽閲嶅娑堣垂鐨勯棶棰橈紝鏈�濂芥秷璐硅�呴偅绔仛涓氬姟涓婄殑鏍￠獙锛堝鏋滃凡缁忔秷璐硅繃浜嗭紝灏变笉娑堣垂浜嗭級
杩欑瘒鏂囩珷涓昏鏄疜afka鍏ラ棬锛孠afka杩樻秹鍙婂埌鍒殑姒傚康锛屼互鍙婅繕鏈夊埆鐨勪笢瑗裤�傚湪鎴戞劅瑙変腑锛屽緢澶氱殑闈㈣瘯棰橀兘璺熼厤缃湁鍏筹紝鎵�浠ュ湪瑙ｅ喅鏌愪簺闂鐨勬椂鍊欙紝鍏堢湅鐪嬭兘涓嶈兘閫氳繃鐜版湁閰嶇疆瑙ｅ喅鎺�锛堝澶氫簡妗嗘灦锛屼綘灏变細鍙戠幇寰堝瀹樻柟鐨勫氨宸茬粡鏀寔瑙ｅ喅浜嗭紝浣犲仛鐨勫彲鑳芥敼鏀归厤缃�/鍙傛暟灏卞畬浜嬩簡锛夈��
鏈凡鏀跺綍鑷虫垜鐨凣itHub绮鹃�夋枃绔狅紝娆㈣繋Star锛�https://github.com/ZhongFuCheng3y/3y
涔愪簬杈撳嚭骞茶揣鐨凧ava鎶�鏈叕浼楀彿锛�Java3y銆傚叕浼楀彿鍐�鏈�300澶氱瘒鍘熷垱鎶�鏈枃绔犮�佹捣閲忚棰戣祫婧愩�佺簿缇庤剳鍥撅紝鍏虫敞鍗冲彲鑾峰彇锛�
闈炲父鎰熻阿浜烘墠浠兘鐪嬪埌杩欓噷锛屽鏋滆繖涓枃绔犲啓寰楄繕涓嶉敊锛岃寰椼�屼笁姝�嶆垜鏈夌偣涓滆タ鐨勮瘽 姹傜偣璧� 姹傚叧娉笍 姹傚垎浜煈� 姹傜暀瑷�馃挰 瀵规殩鐢锋垜鏉ヨ鐪熺殑 闈炲父鏈夌敤锛侊紒锛�
鍒涗綔涓嶆槗锛屽悇浣嶇殑鏀寔鍜岃鍙紝灏辨槸鎴戝垱浣滅殑鏈�澶у姩鍔涳紝鎴戜滑涓嬬瘒鏂囩珷瑙侊紒