Photon——Application - LoadBalancing 负载均衡应用

 

Application - LoadBalancing 负载均衡应用

 

     This article explains the server-side implementation of the LoadBalancing application.

     本文解释服务端是如何实现LoadBalancing应用程序的。

 

Content 目录

• Concept 概念
• Basic Workflow 基本工作流

• Master Server 主服务器
• Master: Handling Client Peers 处理客户端用户
• Master: Handling Game Server Peers 处理游戏服务端用户

• Game Server 游戏服务器
• Game Server: Handling Client Peers 处理客户端用户
• Game Server: Reporting Game States to the Master 报告游戏状态给主服务

• LoadBalancing implementation 负载均衡的实现
• Game Servers: Determine Workload 游戏服务：确定工作负荷
• Implementation Details 实现细节
• Master Server: LoadBalancing Algorithm 主服务：负载均衡算法

• Configuration and Deployment 配置和部署
• Deploying a Game Server 部署游戏服务
• Deploying a Master Server 部署主服务

Concept 概念

 

     The LoadBalancing Application (literally) extends the   Lite  Application. It provides all the well-known Lite functions - like Rooms, Events, Properties and so on, and adds a layer of scalability, that enables you to run the application on multiple servers.

     LoadBalancing应用扩展了Lite应用。它提供了所有的Lite的功能。例如，房间、事件、属性等等，另外添加了一个可扩展的层，使得你可以运行这个应用在多个服务器上。

 

     The basic setup is simple: There is always 1 Master Server and 1..N Game Servers.

     这基本配置是这样的：总有一个主服务和1到N个游戏服务

 

     The Master Server has two roles:

• keep track of the games that are currently open on the Game Servers.
• keep track of the workload of the connected Game Servers and assign peers to the appropriate Game Servers.

     主服务有2个角色：

• 保持跟踪当前打开着的游戏服务
• 保持跟踪连接着的游戏服务器的工作负荷和分配用户到对应的游戏服务器

 

     The Game Servers have two tasks as well:

• host the game rooms. They run a slightly modified version of the Lite Application for this purpose.
• regularly report their current work load and the list of their games to the Master Server.

     游戏服务器有2个任务：

• 承载游戏房间。为此运行一个略做修改版本的Lite
• 定期报告他们当前的工作负荷和游戏列表到主服务器上

Basic Workflow 基本工作流

 

     The workflow from a client side perspective is quite simple as well:

     这个工作流从客户端角度来看非常的简单：

 

     Clients connect to the Master Server, where they can join the lobby and retrieve a list of open games.

     客户端连接到主服务器，他们可以加入大厅和获取一个游戏列表

 

     When they call a   CreateGame  operation on the Master, the game is not actually created - the Master Server only determines the Game Server with the least workload and returns its IP to the client.

     这时他们在主服务器上调用了 CreateGame操作，这个游戏不是实际创建的，主服务器仅仅定义了游戏服务器的最少工作负荷和返回IP地址给客户端。

 

     When clients call a   JoinGame  or   JoinRandomGame  operation on the Master, the Master looks up the Game Server on which the game is running, and returns its IP to the client.

     当客户端在主服务器上调用 JoinGame或 JoinRandomGame操作时，主服务器查找运行着的游戏服务器，并返回IP地址给客户端

 

     The client disconnects from the Master Server, connects to the Game Server with the IP it just received, and calls the   CreateGame  or   JoinGame  operation again.

     客户端从主服务器断开，连接到获得的IP地址对应的游戏服务器，并再次调用 CreateGame或 JoinGame操作

 

     From then on, everything works the same way as with a   Lite  application.

     从这以后，工作方式和Lite应用一样。

 

Master Server 主服务器

 

     This section explains the Master Server implementation - see the LoadBalancing.MasterServer namespace in the \src-server\Loadbalancing\Loadbalancing.sln solution.

     本节将解释主服务器的实现

 

     The MasterApplication decides if incoming connections are originated by game clients (on the “client port”) or by game servers (on the “game server port”).

     这主应用程序定义传入的连接是来自客户端还是来自游戏服务端。

 

Master: Handling Client Peers 处理客户端用户

 

     The MasterClientPeer represents a client connection to the Master Server. The following operations are available for a MasterClientPeer:

     这MasterClientPeer代表一个客户端连接到主服务器。以下操作可用于MasterClientPeer

 

• Authenticate 验证

     The   Authenticate  operation has only a dummy implementation. The developer should use it as a starting point to implement his own authentication mechanism:

     验证操作只有一个伪实现。开发者需要使用它为开始点实现自己的身份验证机制

 

// MasterClientPeer.cs:

private OperationResponse HandleAuthenticate(OperationRequest operationRequest)

{

     OperationResponse response;

 

     var request = new AuthenticateRequest( this .Protocol, operationRequest);

     if (!OperationHelper.ValidateOperation(request, log, out response))

     {

         return response;

     }

     

     this .UserId = request.UserId;

 

     // publish operation response

     var responseObject = new AuthenticateResponse { QueuePosition = 0 };

     return new OperationResponse(operationRequest.OperationCode, responseObject);

}

 

• JoinLobby 加入大厅

     The   JoinLobby  operation is used to add the MasterClientPeer to the AppLobby, which contains a GameList - the list of all open games on any Game Server. The peer receives an initial   GameListEvent, which contains the current list of games in the GameList (filtered by the optional Properties of the   JoinLobby  operation):

     加入大厅操作添加MasterClientPeer到大厅里，包含一个开放游戏的列表。用户接收到一个初始化的 GameListEven t，包含一个当前的游戏列表：

 

     Afterwards, a GameListUpdateEvent is send to the client at regular intervals, which contains the list of changed games (also filtered by the optional Properties of the   JoinLobby  operation). The client will receive the update events as long as it is connected.

     之后，一个GameListUpdateEvent定期发送到客户端，它包含改变的游戏列表。客户端将接收更新事件只要它连接着。

 

// AppLobby.cs:

protected virtual OperationResponse HandleJoinLobby(MasterClientPeer peer, OperationRequest operationRequest, SendParameters sendParameters)

{

     // validate operation

     var operation = new JoinLobbyRequest(peer.Protocol, operationRequest);

     OperationResponse response;

     if (OperationHelper.ValidateOperation(operation, log, out response) == false )

     {

         return response;

     }

 

     peer.GameChannelSubscription = null ;

 

     var subscription = this .GameList.AddSubscription(peer, operation.GameProperties, operation.GameListCount);

     peer.GameChannelSubscription = subscription;

     peer.SendOperationResponse( new OperationResponse(operationRequest.OperationCode), sendParameters);

 

     // publish game list to peer after the response has been sent

     var gameList = subscription.GetGameList();

     var e = new GameListEvent { Data = gameList };

     var eventData = new EventData(( byte )EventCode.GameList, e);

     peer.SendEvent(eventData, new SendParameters());

 

     return null ;

}

• JoinGame / JoinRandomGame 加入游戏/加入随机游戏

     The   JoinGame  operation is called when a client wants to join an existing game that is listed in the AppLobby’s GameList, specified by a unique GameId. If the game exists and the peer is allowed to join, the Master Server returns the IP of the Game Server, on which the Game is actually running, to the client.

     当客户端想加入到游戏列表中一个指定游戏ID的已存在的游戏中时 JoinGame操作将被调用。如果游戏存在并允许用户加入，这主服务会返回游戏服务器的IP地址到客户端，在这个游戏服务器上游戏是真实运行的。

 

     The Master Server also updates the GameState and adds the peer to its list of “joining peers”. It will be removed once it has joined the game on the Game Server (or after a certain timeout). This way, the Master Server can keep track of peers that are in transition between the Master and the Game Server.

     主服务器也更新游戏状态和添加用户到 “joining peers”列表。当它加入到游戏服务器时将被移除。这样，主服务器可以保持跟踪那些传输在主服务器和游戏服务器两边的用户。

 

      JoinRandomGame  works in a similar way, except that the game is chosen by random by the Master Server and the GameId is returned to the client.

      JoinRandomGame工作方式类似，除了，游戏的选择是通过主服务器随机得到的并要返回游戏ID到客户端。

 

• CreateGame 创建游戏

     The   CreateGame  operation is called when a client wants to create a new game. The Master Server determines a Game Server on which the new game will be created and returns the Game Server’s IP to the client. See the “LoadBalancing Algorithm” section for more details.

     当客户端想要创建一个新的游戏时， CreateGame操作将被调用。这主服务器决定在一个游戏服务器上创建一个新的游戏并返回这个游戏服务器的IP地址到客户端。

 

     In addition, a GameState object is created and added to the GameList, and the peer is stored as a “joining peer”.

     此外，一个游戏状态对象被创建并添加到游戏列表，用户被储存为 “joining peer”。
     

     Note that this GameState is only used to keep track of the games - the game itself only exists on a Game Server.

     注意：游戏状态是仅仅被用于保持跟踪游戏服务器上存在的游戏。

Master: Handling Game Server Peers 处理游戏服务端用户

 

     The Master Server always knows which Game Servers are available, how many games they host and how the current workload is.

     主服务器总是知道哪些游戏服务器是可用的，多少游戏被托管，当前的工作负荷是多少。

 

     To achieve this, each Game Server connects to the Master Server on startup. The MasterApplication maintains a GameServerCollection, in which IncomingGameServerPeers are stored.

     为此，每个游戏服务器在启动时就要连接到主服务器。主应用程序包含了一个游戏集合，储存着传入游戏服务器用户。

 

     The Game Server can only call one operation:

     这游戏服务器能够调用的操作是：

• RegisterGameServer 注册游戏服务器 

     The Game Servers call the   RegisterGameServer  operation once after they are connected to the Master Server. The Game Server is added to the Master’s GameServerCollection and to its LoadBalancer (see the “LoadBalancing Algorithm” seection below). It will be removed from the GameServerCollection on disconnect.

     游戏服务器调用 RegisterGameServer操作在它连接到主服务器时。这游戏服务器添加到主服务的游戏服务器集合和负载均衡器中。当断开时它将被移除。

 

     Check the “Game Server” section to see how the Game Server sends further updates about its games and its workload to the Master.

     检查 “Game Server”，看看游戏服务器是如何发送进一步的游戏更新和它的工作负载到主服务器

Game Server 游戏服务器

 

     This section describes the Game Server implementation. See the LoadBalancing.GameServer namespace in the \src-server\Loadbalancing\Loadbalancing.sln solution.

     这节描述了游戏服务器的实现。

 

Game Server: Handling Client Peers 处理客户端用户

 

     The Game Server is derived from the Lite application. As soon as a client has received a Game Server address from the Master, the client can call any operation on the Game Server that is available in Lite. The only difference is that we have separate operation codes for   JoinGame  and   CreateGame  on the Game Server, while Lite handles both with   JoinGame.

     游戏服务器来源于Lite应用。当客户端从主服务器接收到游戏服务器的地址时，客户端可以在游戏服务器上调用任意可用的操作。唯一不同的是在游戏服务器上我们有单独的操作码 JoinGame和 CreateGame，而Lite的处理都是 JoinGame。

 

Game Server: Reporting Game States to the Master 报告游戏状态到主服务

 

     The connection to the Master server is represented as an OutgoingMasterServerPeer in the Game Server. Once the connection is established, the Game Server calls a   Register  operation on the Master Server. Afterwards, the Game Server publishes all existing game states to the master server:

     连接到主服务器将在游戏服务器上被表示为一个OutgoingMasterServerPeer。一旦连接被建立，游戏服务器将在主服务器上调用 Register操作。然后，游戏服务器将公布所有的游戏状态到主服务器上。

 

// OutgoingMasterServerPeer.cs:

protected virtual void HandleRegisterGameServerResponse(OperationResponse operationResponse)

{

     // [...]

     

     switch (operationResponse.ReturnCode)

     {

     case ( short )ErrorCode.Ok:

         {

             log.InfoFormat( "Successfully registered at master server: serverId={0}" , GameApplication.ServerId);

             this .IsRegistered = true ;

             this .UpdateAllGameStates();

             this .StartUpdateLoop();

             break ;

         }

     }

}

 

 

     This is done by sending a message to each Game that tells it to send it’s game state to the master:

     这是发送一个信息到每个游戏，告知他们发送游戏状态到主服务器

 

// OutgoingMasterServerPeer.cs:

public virtual void UpdateAllGameStates()

{

     // [...]

     

     foreach ( var gameId in GameCache.Instance.GetRoomNames())

     {

         Room room;

         if (GameCache.Instance.TryGetRoomWithoutReference(gameId, out room))

         {

             room.EnqueueMessage( new RoomMessage(( byte )GameMessageCodes.ReinitializeGameStateOnMaster));

         }               

     }

}

 

     The Game handles this in the ProcessMessage method and calls the UpdateGameStateOnMaster method to send an   UpdateGameEvent  to the master:

     游戏处理ProcessMessage方法和调用UpdateGameStateOnMaster方法去发送 UpdateGameEvent到主服务器

 

protected virtual void UpdateGameStateOnMaster(

             byte ? newMaxPlayer = null ,

             bool ? newIsOpen = null ,

             bool ? newIsVisble = null ,

             object [] lobbyPropertyFilter = null ,

             Hashtable gameProperties = null ,

             string newPeerId = null ,

             string removedPeerId = null ,

             bool reinitialize = false )

         {           

        // [...]

             

             var e = this .CreateUpdateGameEvent();

         e.Reinitialize = reinitialize;

             e.MaxPlayers = newMaxPlayer;           

         // [ ... more event data is set here ... ]

             

             var eventData = new EventData(( byte )ServerEventCode.UpdateGameState, e);

             GameApplication.Instance.MasterPeer.SendEvent(eventData, new SendParameters());

         }

}

 

 

     The game state is also updated on the master whenever a game is created, joined or left by a client or its properties are changed.

     在游戏被用户创建、加入、离开或者属性被更改时，游戏状态就会被更新到主服务器上。

LoadBalancing implementation 负载均衡的实现

 

     The next section describes how the game servers report their current work load to the Master Server, and how the Master Server determines the Game Server that is best suited to handle new CreateGame requests - the actual LoadBalancing algorithm.

     下一节描述的是游戏服务器如何报告他们当前的工作负荷到主服务器上，主服务器是如何决定哪个游戏服务器处理新的CreateGame请求（实际的负载均衡算法）。

 

Game Servers: Determine Workload 确定工作负荷

     See the LoadBalancing.LoadShedding namespace in the \src-server\Loadbalancing\Loadbalancing.sln solution for implementation details.

     The Game Servers regularly report their current work load to the master server. The work load includes, for example: - CPU usage - Bandwidth usage - some Photon-specific values, like ENet + Business Queue Length, the average time the server spends on each request, etc. - Latency (when sending requests to itself)

     游戏服务器定期报告他们当前的工作负荷到主服务器。这工作负荷包括：CPU的使用、带宽的使用、Photon的一些特殊值。

 

     The most important (and easiest to understand) factor is the CPU load, so we will focus on the CPU load in this documentation.

     最重要的也是最容易理解的是CPU的负载，所以在此我们聚焦于CPU

 

     All these factors are summarized in a single value - the “Load Level” of a Game Server, which is reported to the Master.

     所有的因素被概括在一个独立的值里，游戏的负载等级，这将被报告到主服务器上。

 

     The lower the load level, the better is the Game Server suited to host new games.

     较低的负载水平，是更适合于承载新游戏的游戏服务器。

 

Implementation Details 实现细节

 

     The Game Server collects “Feedback” about the above-mentioned factors. There is one FeedbackController object for each factor - it consists of a FeedbackName and a FeedbackLevel:

     游戏服务器收集上述因素的反馈“Feedback”，这为每个因素提供一个FeedbackController对象，它由一个FeedbackName和一个FeedbackLevel组成。

 

internal enum FeedbackName

{

     CpuUsage,  

     Bandwidth,

     TimeSpentInServer

}

     

public enum FeedbackLevel

{

     Highest = 4,

     High = 3,

     Normal = 2,

     Low = 1,

     Lowest = 0

}

 

 

     The DefaultConfiguration class defines the thresholds for each value - e.g., a server has the “lowest” FeedbackLevel up to 20% CPU usage, it reaches the “highest” FeedbackLevel at 90% CPU and so on.

     DefaultConfiguration类定义了每个值的阀值，一个服务器有 “lowest” 级别的FeedbackLevel 提升20%的CUP使用，它达到“highest” 级别的FeedbackLevel 时CPU已经使用了90%

 

// DefaultConfiguration.cs:

 

internal static List<FeedbackController> GetDefaultControllers()

{

     internal static List<FeedbackController> GetDefaultControllers()

     {

         var cpuController = new FeedbackController(

         FeedbackName.CpuUsage,

         new Dictionary<FeedbackLevel, int >

                 {

                     { FeedbackLevel.Lowest, 20 },

                     { FeedbackLevel.Low, 35 },

                     { FeedbackLevel.Normal, 50 },

                     { FeedbackLevel.High, 70 },

                     { FeedbackLevel.Highest, 90 }

                 },

         0,

         FeedbackLevel.Lowest);

 

     // [...]

}

 

 

     These values can also be configured in a workload.config file. The LoadBalancing.LoadShedding.Configuration namespaces takes care of reading values from a config file, or applies the DefaultConfiguration if no config exists.

     这些值可以被配置在workload.config文件中。 LoadBalancing.LoadShedding.Configuration命名空间关注于读取配置文件，或当配置文件不存在的时候应用默认值。

 

     At regular intervals, the Game Server checks some Windows Performance Counters, sets the current values for all its FeedbackControllers and calculates a new “overall feedback”.

     在一定的时间间隔内，游戏服务器会检查一些Windows性能计数器，为所有的FeedbackControllers设置当前值并计算一个新的“overall feedback”

 

     This is done in the WorkloadController class:

     WorkloadController类是这样的：

 

private void Update()

{

     FeedbackLevel oldValue = this .feedbackControlSystem.Output;

 

     if ( this .cpuCounter.InstanceExists)

     {

         var cpuUsage = ( int ) this .cpuCounter.GetNextAverage();

         Counter.CpuAvg.RawValue = cpuUsage;

         this .feedbackControlSystem.SetCpuUsage(cpuUsage);

     }

     

     // [...]

     

     if ( this .timeSpentInServerInCounter.InstanceExists && this .timeSpentInServerOutCounter.InstanceExists)

     {

         var timeSpentInServer = ( int ) this .timeSpentInServerInCounter.GetNextAverage() + ( int ) this .timeSpentInServerOutCounter.GetNextAverage();

         Counter.TimeInServerInAndOutAvg.RawValue = timeSpentInServer;

         this .feedbackControlSystem.SetTimeSpentInServer(timeSpentInServer);

     }

     

     this .FeedbackLevel = this .feedbackControlSystem.Output;

     Counter.LoadLevel.RawValue = ( byte ) this .FeedbackLevel;

 

     if (oldValue != this .FeedbackLevel)

     {

         if (log.IsInfoEnabled)

         {

             log.InfoFormat( "FeedbackLevel changed: old={0}, new={1}" , oldValue, this .FeedbackLevel);

         }

 

         this .RaiseFeedbacklevelChanged();

     }

}

 

     If the overall feedback level changes, the OutgoingMasterServerPeer will report the new server state to the Master:

     如果overall feedback等级改变，OutgoingMasterServerPeer 将报告新的服务器状态到主服务器：

 

public void UpdateServerState()

  {

     // [...]

     this .UpdateServerState(

         GameApplication.Instance.WorkloadController.FeedbackLevel,

         GameApplication.Instance.PeerCount,

         GameApplication.Instance.WorkloadController.ServerState);

}

 

Master Server: LoadBalancing Algorithm 主服务：负载均衡算法

     See the LoadBalancing.LoadBalancer class in the \src-server\Loadbalancing\Loadbalancing.sln solution for implementation details.

     The Master Server stores the LoadLevel for each Game Server in the LoadBalancer class. It also holds an additional list of all the servers that currently have the lowest load level.

     主服务器在LoadBalancer类中储存了每个游戏服务器的负载水平。它还附加处理一个包含所有目前最低负荷水平的服务器的列表。

 

     Whenever a client calls the   CreateGame  operation, the Master Server fetches the address of a server with the lowest load level from the LoadBalancer and returns it to the client, which then connects to that server.

     任何时候一个客户端调用 CreateGame操作时，主服务器会从LoadBalancer中获取一个最低负荷的服务器地址返回给客户端并连接到服务器。

Configuration and Deployment 配置和部署

 

     For demonstration purposes, the SDK contains a setup of 1 Master and 2 Game Servers in its deploy directory:

     出于演示的目的，SDK中包含1个主服务器与2个游戏服务器在deploy目录下

• • /deploy/LoadBalancing/Master
  • /deploy/LoadBalancing/GameServer1
  • /deploy/LoadBalancing/GameServer2

     This setup is only intended for local development.

     这个设置仅仅适用于本地开发

Deploying a Game Server 部署游戏服务

 

     When you deploy your LoadBalancing project to a production server, you should not host 2 Game Server applications on one server. Remove all settings for “GameServer2” from the PhotonServer.config, and delete the /deploy/LoadBalancing/GameServer2 directory.

     当你部署你的负载均衡项目到一个服务器上时，你不要承载2个游戏服务在一个服务器上，从PhotonServer.config中移除“GameServer2”的所有设置。并删除/deploy/LoadBalancing/GameServer2目录

 

     You need to make sure that the Game Servers can register at the Master Server. Set the MasterIPAddress in the Photon.LoadBalancing.dll.config to the Master’s public IP.

     你需要确保游戏服务器可以注册到主服务器上。在Photon.LoadBalancing.dll.config设置MasterIPAddress为主服务器的公共IP地址。

 

     You also need to make sure that the game clients can reach the Game Servers. On each Game Server, you need to set the Public IP address of that Game Server. If you leave the value empty, the Public IP Address will be auto detected.

     你还需要确保游戏客户端可以到达游戏服务器。在每个游戏服务器上，你需要设置公共的IP地址。如果你离开的值为空，这公共的IP地址将自动被检测到。

 

< Photon.LoadBalancing.GameServer.GameServerSettings >

       < setting name = "MasterIPAddress" serializeAs = "String" >

         < value >127.0.0.1</ value >

< setting >    

     < setting name = "PublicIPAddress" serializeAs="String”>

         < value >127.0.0.1</ value >

         

         <!-- use this to auto-detect the PublicIPAddress: -->

         <!-- <value> </value> -->

       </ setting >

       

       <!-- [...] -->

</ Photon.LoadBalancing.GameServer.GameServerSettings >

 

     You can also use the Photon Control to set a Public IP.

     你也可以使用Photon控制器来设置一个公共的IP。

Deploying a Master Server 部署主服务

 

     You need to make sure that you only have 1 Master server: either remove all settings for the “Master” application from the PhotonServer.config on your Game Servers, or at least make sure that your game servers and clients all use the same IP to connect to the same, single Master server.

     你需要确保你至少有一个主服务器，在你的游戏服务器上的PhotonServer.config中移除所有的 “Master”设置，或者至少确保你的游戏服务器和客户端使用的IP可以连接到独立的主服务器。

 

     Otherwise, no special configuration is required on the master server.

     不然，主服务器需要其他特殊的配置。