添加echo服务与分析

【在Cloud Foundry 上添加ECHO服务】

官方教程：https://github.com/cloudfoundry/oss-docs/tree/master/vcap/adding_a_system_service

不清楚是CF版本更新的原因还是别的什么，反正按照官方给出的教程添加echo是无法成功的。问题就在于官方提供配置文件缺少一些参数造成的。

下面的内容重点是熟悉这些配置信息是怎么回事。

 

首先我准备了两个节点：

【Node1】

预安装：cloud_controller、router 、health_manager、mongodb_gateway 、mysql_gateway、redis_gateway、mongodb_node、stager、redis_node。

IP地址：192.168.1.166

准备添加：echo_gateway

【Node2】

预安装：mysql、dea

IP地址：192.168.1.101

准备添加：echo_node

 

 

 

 

 

【添加gateway】

【步骤1】

内容：修改~/cloudfoundry/.deployments/devbox/config/vcap_components.json文件，在其中添加echo_gateway，如下

{"components":["router","health_manager","mongodb_gateway","cloud_controller","mysql_gateway","redis_gateway","mongodb_node","stager","redis_node","echo_gateway"]}

功能：这个文件夹下存储的就是“开机启动项”，每次重启CF，它都是根据这个文件中的内容选择开启哪些内容。

 

【步骤2】

内容：修改vim cloudfoundry/.deployments/center/config/cloud_controller.yml文件，添加如下内容：

# Services we provide, and their tokens. Avoids bootstrapping DB. builtin_services:   redis:     token: changeredistoken   mongodb:     token: changemongodbtoken   mysql:     token: changemysqltoken   echo:     token: changeechotoken service_proxy:   token: ["changebrokertoken"]

功能：CC在与Services的gateway交互期间，需要添加1个令牌，两者在初始化配置文件中都需要一致。可以查看【步骤6】中echo_gateway的token配置信息。

 

【步骤3】

内容：修改~/cloudfoundry/vcap/services/tools/misc/bin/nuke_service.rb，添加如下内容：

default_configs = {   :mongodb => File.expand_path("../../mongodb/config/mongodb_gateway.yml", __FILE__),   :redis   => File.expand_path("../../redis/config/redis_gateway.yml", __FILE__),   :mysql   => File.expand_path("../../mysql/config/mysql_gateway.yml", __FILE__),   :neo4j   => File.expand_path("../../neo4j/config/neo4j_gateway.yml", __FILE__),   :vblob   => File.expand_path("../../vblob/config/vblob_gateway.yml", __FILE__),   :echo    => File.expand_path("../../echo/config/echo_gateway.yml", __FILE__), }

功能：这个是一个工具，用于从CF上删除一个service ，不能算作是添加Service中的一个环节。

 

【步骤4】

内容：编辑~/cloudfoundry/vcap/dev_setup/lib/vcap_components.rb文件，添加如下内容

## services: gateways & nodes %w(redis mysql mongodb rabbitmq postgresql vblob neo4j memcached couchdb elasticsearch filesystem echo).each do |service|   ServiceComponent.register("#{service}_gateway") end   %w(redis mysql mongodb rabbitmq postgresql vblob neo4j memcached couchdb elasticsearch ).each do |service|  ServiceComponent.register("#{service}_node") end   %w(redis mysql mongodb postgresql).each do |service|   ServiceComponent.register("#{service}_worker") end

功能：看语句可以猜到，在启动过程中需要对每个组件进行注册操作。我们查看~/cloudfoundry/vcap/dev_setup/bin/vcap，可以看到这样一句，在【步骤1】中我们也看到启动的时候使用的vcap，而vcap又需要vcap_components库，这个库就是我们这个步骤中修改的文件：

require File.expand_path(File.join(“..”, “lib”, “vcap_components”), File.dirname(__FILE__))

 

【步骤5】

内容：执行下列命令

cd ~/cloudfoundry/vcap/services/echo;source $HOME/.cloudfoundry_deployment_profile && bundle package

功能：【--？--应该是编译echo，或者是在解决依赖包关系，这部分内容暂时不太明白】

注：这部分内容必须分析，以后我们需要手动写这部分的内容

 

【步骤6】

内容：将 ~/cloudfoundry/vcap/services/echo/config/echo_gateway.yml文件拷贝到 ~/cloudfoundry/.deployments/mysqldea/config/目录下。并如下修改

--- cloud_controller_uri: api.vcap.me service:   name: echo   version: "1.0"   description: 'Echo service'   plans: ['free']   default_plan: free   tags: ['echo', 'echo-1.0', 'echobased', 'demo']   timeout: 15   supported_versions: ['1.0']   version_aliases:     "current": "1.0" ip_route: 192.168.1.166 index: 0 token: changeechotoken logging:   level: debug mbus: nats://nats:[email protected]:4222 pid: /var/vcap/sys/run/echo_service.pid node_timeout: 2

功能：第一、如下所示，启动时候使用的配置信息在~/cloudfoundry/.deployments/mysqldea/config/目录下，所以我们需要将配置文件复制到该目录下。

Executing /home/vm/cloudfoundry/.deployments/center/deploy/rubies/ruby-1.9.2-p180/bin/ruby /home/vm/cloudfoundry/vcap/dev_setup/bin/vcap restart cloud_controller router health_manager mongodb_gateway mysql_gateway redis_gateway mongodb_node stager redis_node echo_gateway -c /home/vm/cloudfoundry/.deployments/center/config -v /home/vm/cloudfoundry -l /home/vm/cloudfoundry/.deployments/center/log

第二、在配置信息方面，官网上给的有误，少了几个参数，后面我们会介绍如果缺少某些参数的时候我们要如何排错

 

【添加echo_node】

【步骤1】

内容：重复【添加echo_node】中的【步骤1】，但是这次添加的是“echo_node”

{"components":["dea","mysql_node","echo_node"]}

功能：添加启动项

 

【步骤2】

内容：重复【添加echo_node】中的【步骤3】。

功能：与【添加echo_node】中的【步骤3】一致

 

【步骤3】

内容：重复【添加echo_node】中的【步骤4】，只是这次添加echo的位置不同

## services: gateways & nodes %w(redis mysql mongodb rabbitmq postgresql vblob neo4j memcached couchdb elasticsearch filesystem).each do |service|   ServiceComponent.register("#{service}_gateway") end   %w(redis mysql mongodb rabbitmq postgresql vblob neo4j memcached couchdb elasticsearch echo).each do |service|  ServiceComponent.register("#{service}_node") end   %w(redis mysql mongodb postgresql).each do |service|   ServiceComponent.register("#{service}_worker") end

功能：与【添加echo_node】中的【步骤4】一致

 

【步骤4】

内容：重复【添加echo_node】中的【步骤5】

功能：与【添加echo_node】中的【步骤5】一致

 

【步骤5】

内容：将 ~/cloudfoundry/vcap/services/echo/config/echo_node.yml文件拷贝到 ~/cloudfoundry/.deployments/mysqldea/config/目录下。并如下修改

--- plan: free capacity: 100 local_db: sqlite3:/var/vcap/services/echo/echo_node.db mbus: nats://nats:[email protected]:4222 base_dir: /var/vcap/services/echo/ index: 0 logging:   level: debug pid: /var/vcap/sys/run/echo_node.pid node_id: echo_node_0 port: 5002 host: 192.168.1.101

功能：与与【添加echo_node】中的【步骤6】一致

 

【参数分析】

【分析1】

考虑【添加gateway】中【步骤1】中添加的参数

准备工作：将下列中灰色部分删除

{"components":["router","health_manager","mongodb_gateway","cloud_controller","mysql_gateway","redis_gateway","mongodb_node","stager","redis_node","echo_gateway"]}

结果：启动时不会检查echo_gateway，直接忽略。

结果分析：我们查看vim /home/vm/cloudfoundry/vcap/dev_setup/bin/vcap_dev，在最底部就可以看到，启动过程中的参数就包含了vcap_components["components"].join(" ")

puts "Using cloudfoundry config from #{deployment_config_path}" exec_cmd("#{ruby_binary} #{vcap_launch} #{command} #{vcap_components["components"].join(" ")} -c #{deployment_config_path} -v #{vcap_home} -l #{deployment_info["deployment_log_path"]}")

而我们向面追查发现vcap_components的赋值

begin   vcap_components = JSON.parse(File.read(Deployment.get_vcap_config_file(deployment_config_path)))   deployment_info = JSON.parse(File.read(Deployment.get_deployment_info_file(deployment_config_path))) rescue => e   STDERR.puts "#{e.inspect}. Could not parse deployment config files . Please check your deployment."   exit 1 end

我们直接将其打印出来：vcap_config_file/home/vm/cloudfoundry/.deployments/center/config/vcap_components.json。所以我们现在就知道了这个文件下存放着启动项的信息

【分析2】

考虑【添加gateway】中【步骤1】中的token参数，我们将分析gateway与CC之间的token是怎么回事

【分析2.1】

准备：在center节点上，我们编辑文件~/ cloudfoundry/.deployments/center/config/cloud_controller.yml，将token部分去除

# Services we provide, and their tokens. Avoids bootstrapping DB. builtin_services:   redis:     token: changeredistoken   mongodb:     token: changemongodbtoken   mysql:     token: changemysqltoken #  echo: #    token: changeechotoken service_proxy:   token: ["changebrokertoken"]

结果：正常运行，【--？--现在仅仅是log日志看着没问题，不清楚会不会影响正常使用】

CC的日志信息：

DEBUG -- Create service request: {"label":"echo-1.0","url":"http://192.168.1.166:35980","plans":["free"],"cf_plan_id":null,"tags":["echo","echo-1.0","echobased","demo"],"active":true,"description":"Echo service","plan_options":null,"acls":null,"timeout":15,"provider":null,"default_plan":"free","supported_versions":["1.0"],"version_aliases":{"current":"1.0"}} Found svc = #<Service id: 4, label: "echo-1.0", url: "http://192.168.1.166:32781", token: "changeechotoken", name: "echo", version: "1.0", description: "Echo service", info_url: nil, tags: ["echo", "echo-1.0", "echobased", "demo"], plans: ["free"], plan_options: nil, binding_options: nil, acls: nil, active: true, created_at: "2012-10-30 04:51:15", updated_at: "2012-10-30 05:23:56", timeout: 15, cf_plan_id: nil, provider: nil, supported_versions: ["1.0"], version_aliases: {"current"=>"1.0"}, default_plan: "free">

echo_gateway的日志信息：

INFO -- Sending info to cloud controller: http://api.vcap.me/services/v1/offerings INFO -- Fetching handles from cloud controller @ http://api.vcap.me/services/v1/offerings/echo-1.0/handles DEBUG -- [EchoaaS-Provisioner] Connected to node mbus.. DEBUG -- [EchoaaS-Provisioner] Received node announcement: {"available_capacity":100,"capacity_unit":1,"id":"echo_node_0","plan":"free","supported_versions":["1.0"]} INFO -- Successfully registered with cloud controller INFO -- Successfully fetched handles

结果分析：说明修改~/ cloudfoundry/.deployments/center/config/cloud_controller.yml中的token没有起作 用。但是这个Token真的没有用么，见【分析1.3】

 

【分析2.2】

准备：修改vim cloudfoundry/.deployments/center/config/echo_gateway.yml，将token部分注释掉

--- cloud_controller_uri: api.vcap.me service:   name: echo   version: "1.0"   description: 'Echo service'   plans: ['free']   default_plan: free   tags: ['echo', 'echo-1.0', 'echobased', 'demo']   timeout: 15   supported_versions: ['1.0']   version_aliases:     "current": "1.0" ip_route: 192.168.1.166 index: 0 #token: changeechotoken logging:   level: debug mbus: nats://nats:[email protected]:4222 pid: /var/vcap/sys/run/echo_service.pid node_timeout: 2

结果：在启动过程中提示“Token missing”，这个的log是echo_gateway的日志里面提示的信息，而在CC的log中找不到任何echo相关的字眼。

vm@vm-virtual-machine:~$ ~/cloudfoundry/vcap/dev_setup/bin/vcap_dev restart Targeting deployment "center" with cloudfoundry home "/home/vm/cloudfoundry" ……………… echo_gateway                  :   STOPPED LOG:  Couldn't read config file: Token missing

结果分析：说明echo_gateway.yml中的token是必须写入的。

 

【分析2.3】

准备：修改vim cloudfoundry/.deployments/center/config/echo_gateway.yml，将token部分修改

--- cloud_controller_uri: api.vcap.me service:   name: echo   version: "1.0"   description: 'Echo service'   plans: ['free']   default_plan: free   tags: ['echo', 'echo-1.0', 'echobased', 'demo']   timeout: 15   supported_versions: ['1.0']   version_aliases:     "current": "1.0" ip_route: 192.168.1.166 index: 0 token: changemyechotoken logging:   level: debug mbus: nats://nats:[email protected]:4222 pid: /var/vcap/sys/run/echo_service.pid node_timeout: 2

结果：

我们查看echo_gateway的日志发现：

[2012-10-30 13:57:24.994541] echo_gateway - pid=21159 tid=63c7 fid=5643  ERROR -- Failed registering with cloud controller, status=403 [2012-10-30 13:57:25.002911] echo_gateway - pid=21159 tid=63c7 fid=5643  ERROR -- Failed fetching handles, status=403

结果分析：CC配置文件中的token是需要的，一旦不一致，echo_gateway就会出现错误。

至于为什么在【分析2.1】中却没有出现问题，这点就不得而知了

 

【分析3】

考虑【添加gateway】中【步骤3】中的添加的语句，我们将其注释掉；

准备工作：我们将下面灰色部分语句注释

default_configs = {   :mongodb => File.expand_path("../../mongodb/config/mongodb_gateway.yml", __FILE__),   :redis   => File.expand_path("../../redis/config/redis_gateway.yml", __FILE__),   :mysql   => File.expand_path("../../mysql/config/mysql_gateway.yml", __FILE__),   :neo4j   => File.expand_path("../../neo4j/config/neo4j_gateway.yml", __FILE__),   :vblob   => File.expand_path("../../vblob/config/vblob_gateway.yml", __FILE__),   #:echo    => File.expand_path("../../echo/config/echo_gateway.yml", __FILE__), }

结果：正常运行，查看了想到的日志文件，运行都正常。

结果分析：这个配置时不影响应用正常启动的，更加证明这个是一个工具。

 

【分析4】

考虑【添加gateway】中【步骤3】中的添加的参数，我们将其删除：

准备工作：如下面所示，将echo删除

## services: gateways & nodes %w(redis mysql mongodb rabbitmq postgresql vblob neo4j memcached couchdb elasticsearch filesystem echo).each do |service|   ServiceComponent.register("#{service}_gateway") end   %w(redis mysql mongodb rabbitmq postgresql vblob neo4j memcached couchdb elasticsearch).each do |service|   ServiceComponent.register("#{service}_node") end

结果：如下图所示，在启动echo_gateway的过程中，因为无法注册，所以直接忽略。

vm@vm-virtual-machine:~$ ~/cloudfoundry/vcap/dev_setup/bin/vcap_dev restart ……………… Executing /home/vm/cloudfoundry/.deployments/center/deploy/rubies/ruby-1.9.2-p180/bin/ruby /home/vm/cloudfoundry/vcap/dev_setup/bin/vcap restart cloud_controller router health_manager mongodb_gateway mysql_gateway redis_gateway mongodb_node stager redis_node echo_gateway -c /home/vm/cloudfoundry/.deployments/center/config -v /home/vm/cloudfoundry -l /home/vm/cloudfoundry/.deployments/center/log Skipping invalid component: echo_gateway_gateway Skipping invalid component: echo_gateway_node ……………… redis_node                           STOPPED Skipping invalid component: echo_gateway_gateway Skipping invalid component: echo_gateway_node Skipping invalid component: echo_gateway_gateway Skipping invalid component: echo_gateway_node cloud_controller                 RUNNING ……………… redis_node                           RUNNING

结果分析：从配置的文件中，我们可以发现，echo需要注册后才能启动。

 

【分析5】

考虑【添加gateway】中【步骤5】中的添加的参数，我们将其删除：

准备工作：我们在配置完参数后，不执行bundle语句，直接启动

结果：我们发现，有如下的提示信息

vm@vm-virtual-machine:~$ ~/cloudfoundry/vcap/dev_setup/bin/vcap_dev restart Targeting deployment "center" with cloudfoundry home "/home/vm/cloudfoundry" ……………… redis_node                          :        RUNNING echo_gateway                      :        STOPPED LOG:  /home/vm/cloudfoundry/.deployments/center/deploy/rubies/ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/bundler-1.1.3/lib/bundler/spec_set.rb:90:in `block in materialize': Could not find ci_reporter-1.7.0 in any of the sources (Bundler::GemNotFound) ……………… from /home/vm/cloudfoundry/vcap/bin/../services/echo/bin/echo_gateway:7:in `<main>'

结果分析：【--？--以后我们也要这样来？！】

【分析6】

考虑【添加gateway】中【步骤6】中的添加的参数，我们将试着一个个删除，看看会有什么样的表现

【分析6.1】

准备工作：删除ip_route: 192.168.1.166。

--- cloud_controller_uri: api.vcap.me service:   name: echo   version: "1.0"   description: 'Echo service'   plans: ['free']   default_plan: free   tags: ['echo', 'echo-1.0', 'echobased', 'demo']   timeout: 15   supported_versions: ['1.0']   version_aliases:     "current": "1.0" ip_route: 192.168.1.166 index: 0 token: changeechotoken logging:   level: debug mbus: nats://nats:[email protected]:4222 pid: /var/vcap/sys/run/echo_service.pid node_timeout: 2

结果：正常运行

结果分析：ip_route这个参数不是必须参数

 

【分析6.2】

准备工作：删除version部分内容

--- cloud_controller_uri: api.vcap.me service:   name: echo   version: "1.0"   description: 'Echo service'   plans: ['free']   default_plan: free   tags: ['echo', 'echo-1.0', 'echobased', 'demo']   timeout: 15   supported_versions: ['1.0']   version_aliases:     "current": "1.0" index: 0 token: changeechotoken logging:   level: debug mbus: nats://nats:[email protected]:4222 pid: /var/vcap/sys/run/echo_service.pid node_timeout: 2

结果：

在echo_gateway的日志中可以看到

ERROR -- Failed registering with cloud controller, status=400

在CC的日志文件中可以看到

DEBUG -- Create service request: {"label":"echo-1.0","url":"http://192.168.1.166:37472","plans":["free"],"cf_plan_id":null,"tags":["echo","echo-1.0","echobased","demo"],"active":true,"description":"Echo service","plan_options":null,"acls":null,"timeout":15,"provider":null,"default_plan":"free","supported_versions":["1.0"],"version_aliases":null} ERROR -- Failure decoding service offering request: Field: version_aliases, Error: Missing field version_aliases

结果分析：在添加服务的时候，需要确定参数：version_aliases。

 

【分析6.3】

准备工作

--- cloud_controller_uri: api.vcap.me service:   name: echo   version: "1.0"   description: 'Echo service'   plans: ['free']   default_plan: free   tags: ['echo', 'echo-1.0', 'echobased', 'demo']   timeout: 15   supported_versions: ['1.0']   version_aliases:     "current": "1.0" index: 0 token: changeechotoken logging:   level: debug mbus: nats://nats:[email protected]:4222 pid: /var/vcap/sys/run/echo_service.pid node_timeout: 2

结果：

在echo_gateway的日志中可以看到

ERROR -- Failed registering with cloud controller, status=400

在CC的日志文件中可以看到

DEBUG -- Create service request: {"label":"echo-1.0","url":"http://192.168.1.16 6:44294","plans":["free"],"cf_plan_id":null,"tags":["echo","echo-1.0","echobased","demo"],"active":true,"description":"Echo service","plan_op tions":null,"acls":null,"timeout":15,"provider":null,"default_plan":"free","supported_versions":null,"version_aliases":null} ERROR -- Failure decoding service offering request: Field: supported_versions, Error: Missing field supported_versions, Field: version_aliases, Error: Missing field version_aliases

结果分析：

 

【分析6.4】

--- cloud_controller_uri: api.vcap.me service:   name: echo   version: "1.0"   description: 'Echo service'   plans: ['free']   default_plan: free   tags: ['echo', 'echo-1.0', 'echobased', 'demo']   timeout: 15   supported_versions: ['1.0']   version_aliases:     "current": "1.0" index: 0 token: changeechotoken logging:   level: debug mbus: nats://nats:[email protected]:4222 mbus:nats://localhost:4222 pid: /var/vcap/sys/run/echo_service.pid node_timeout: 2

结果：在CC的日志中我们找不到任何和echo相关的内容，而在echo_gateway的日志文件中，我们可以找到

ERROR -- Exiting due to NATS error: Could not connect to server on nats://localhost:4222 ERROR -- Failed registering with cloud controller: ERROR -- Failed fetching handles:

结果分析：CC请求Service需要CC在NATS上订阅了该services，然后这个service的gateway需要在NATS上进行注册以后，CC订阅到该service的信息，然后才会发送请求该service的请求。

【分析6】中的内容都是因为gateway配置错误而引起的，而gateway的配置文件需要配置那些信息呢？其实我们可以通过CC发送的request信息可以看出它需要读取那些参数信息，下表是mongodb的request经过整理的表格，我们整理了一部分必须的参数【--？--不排除其他参数必须】。

键	值	说明
"label"	"mongodb-1.8"	请求的service的标签
"url"	"http://192.168.1.166:42208"	【必须】链接的地址
"plans"	["free"]
"cf_plan_id"	null
"tags"	["nosql"，"document"]	对于MongoDB的tags
"active"	true
"description"	"MongoDB NoSQL store"	描述
"plan_options"	null
"acls"	null
"timeout"	15
"provider"	null
"default_plan"	"free"
"supported_versions"	["1.8","2.0"]	【必须】可以支持的版本
"version_aliases"	{"next":"2.0","current":"1.8"}	【必须】当前使用的版本，和备用的版本

从上面的表我们可能看的还是有些模糊，可以参考/home/vm/cloudfoundry/.deployments/center/deploy/rubies/ruby-1.9.2-p180/lib/ruby/gems/1.9.1/bundler/gems/vcap-services-base-e54a821c72f1/lib/base/asynchronous_service_gateway.rb的setup函数。我们可以发现下面的语句，发现很多参数都是匹配的：

43   def setup(opts)  44     missing_opts = REQ_OPTS.select {|o| !opts.has_key? o}  45     raise ArgumentError, "Missing options: #{missing_opts.join(', ')}" unless missing_opts.empty?  46     @service      = opts[:service]  47     @token        = opts[:token]  48     @logger       = opts[:logger] || make_logger()  49     @cld_ctrl_uri = http_uri(opts[:cloud_controller_uri])  50     @offering_uri = "#{@cld_ctrl_uri}/services/v1/offerings"  51     @provisioner  = opts[:provisioner]  52     @hb_interval  = opts[:heartbeat_interval] || 60  53     @node_timeout = opts[:node_timeout]  54     @handles_uri = "#{@cld_ctrl_uri}/services/v1/offerings/#{@service[:label]}/handles"  55     @handle_fetch_interval = opts[:handle_fetch_interval] || 1  56     @check_orphan_interval = opts[:check_orphan_interval] || -1  57     @double_check_orphan_interval = opts[:double_check_orphan_interval] || 300  58     @handle_fetched = false  59     @fetching_handles = false  60     @version_aliases = @service[:version_aliases] || {}  61     @svc_json     = {  62       :label  => @service[:label],  63       :url    => @service[:url],  64       :plans  => @service[:plans],  65       :cf_plan_id => @service[:cf_plan_id],  66       :tags   => @service[:tags],  67       :active => true,  68       :description  => @service[:description],  69       :plan_options => @service[:plan_options],  70       :acls => @service[:acls],  71       :timeout => @service[:timeout],  72       :provider => @service[:provider],  73       :default_plan => @service[:default_plan],  74       :supported_versions => @service[:supported_versions],  75       :version_aliases => @service[:version_aliases]  76     }.to_json