使用nginx对grpc进行反向代理和负载均衡
文章目录
-
- 前言
- 相关链接
- 先了解下grpc的一些前置条件
- 上配置
-
- 1. grpc
- 2. grpc + ssl
- 配置完之后验证
-
- 方法1，使用postman调用grpc
- 方法2，使用Java代码调用grpc
- 1.grpc客户端demo
-
- 可能遇到的报错：
- 原因：
- 2. grpc+ssl客户端demo
-
- 可能遇到的报错：
- 原因：
前言
最近项目中需要给我们的grpc服务进行反向代理和负载均衡。故将这期间查阅的资料进行了整理记录
相关链接
介绍nginx负载均衡：https://blog.csdn.net/qq_51574197/article/details/117408823
nginx对grpc进行反向代理：https://blog.csdn.net/Dearmark/article/details/80066819
netty中http2的设计：https://my.oschina.net/u/1261452/blog/2997252
springboot对http2的支持：https://blog.csdn.net/luxinfeng666/article/details/107067260/
tomcat对http2的支持：https://www.cnblogs.com/xifenglou/p/9394948.html
nginx配置ssl和http2：http://www.web3.xin/index/article/567.html
springboot配置https(SSL证书)：https://www.jb51.net/article/233123.htm#_lab2_1_4
springboot配置SSL：https://blog.csdn.net/csj50/article/details/123206007
keytool生成证书：https://blog.csdn.net/Guesshat/article/details/123024693
nginx对gprc进行负载均衡的java小demo：https://blog.csdn.net/weixin_42691149/article/details/119214146?utm_term=nginx-ingress%E8%A7%A3%E5%86%B3grpc%E9%95%BF%E8%BF%9E%E6%8E%A5%E8%B4%9F%E8%BD%BD%E4%B8%8D%E5%9D%87%E5%A4%B1%E8%B4%A5&utm_medium=distribute.pc_aggpage_search_result.none-task-blog-2allsobaiduweb~default-1-119214146-null-null&spm=3001.4430
nginx反向代理grpc长连接服务最佳实践：https://blog.csdn.net/zry5858/article/details/123609086
nginx安装http2模块：https://blog.csdn.net/ChinaHacker995/article/details/113538789
nginx安装http2模块：https://blog.csdn.net/weixin_43885541/article/details/121009451
postman支持grpc：https://blog.csdn.net/qq_60168783/article/details/123339858
先了解下grpc的一些前置条件
- 1.确认nginx是否安装了http2模块，没安装的话是不支持http2的，因为grpc是基于HTTP2来开发的，是需要nginx支持http2，验证方法nginx -V，注意v是大写的
nginx安装http2模块：https://blog.csdn.net/ChinaHacker995/article/details/113538789
nginx安装http2模块：https://blog.csdn.net/weixin_43885541/article/details/121009451 - 2.nginx从1.13.10开始支持grpc
- 3.tomcat 从 8.0以上支持http2 及 h2c
上配置
1. grpc
#此配置为负载均衡配置
upstream datacenter_load {
server 127.0.0.1:9090;
}
server {
#此处配置上http2，因为gprc是基于http2的
listen 88 http2;
server_name test.datacenter.technocore.network;
#此处貌似不能像http转https一样，会报错，所以注释掉单独再写个grpcs的server配置
#return 301 grpc://$server_name:880/$request_uri;
location / {
#此处配置grpc服务的反向代理
grpc_pass grpc://datacenter_load;
grpc_set_header Host $host;
grpc_set_header X-Real-IP $remote_addr;
grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
2. grpc + ssl
#此配置为负载均衡配置
upstream datacenter_load2 {
server 127.0.0.1:9090;
}
server {
#此处配置上http2和ssl，因为gprc是基于http2的
listen 880 ssl http2;
server_name test.technocore.network;
#此处配置ssl证书的路径
ssl_certificate /opt/metisnetwork/nginx/ssl/test.technocore.network.pem;
ssl_certificate_key /opt/metisnetwork/nginx/ssl/test.technocore.network.key;
location / {
#此处配置grpc服务的反向代理
grpc_pass grpc://datacenter_load2;
grpc_set_header Host $host;
grpc_set_header X-Real-IP $remote_addr;
grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
配置完之后验证
方法1，使用postman调用grpc
postman支持grpc：https://blog.csdn.net/qq_60168783/article/details/123339858
可能会遇到的报错信息：
Internal errorOperation couldn’t be completed because of an internal error at the server.Received RST_STREAM with code 2 triggered by internal client error: Protocol error
分析：
1.博主现在postman版本是Postman v9.19.0，暂时是不支持grpc+ssl的
2.可能是端口错误了
方法2，使用Java代码调用grpc
1.grpc客户端demo
package com.platon.metis.storage.grpc;
import com.platon.metis.storage.grpc.lib.api.ListPowerSummaryResponse;
import com.platon.metis.storage.grpc.lib.api.ResourceServiceGrpc;
import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;
import lombok.extern.slf4j.Slf4j;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.test.context.junit4.SpringRunner;
@RunWith(SpringRunner.class)
@Slf4j
public class GrpcClientDemo {
@Test
public void listPowerSummary() {
log.info("start to test listPowerSummary()...");
//普通的grpc需要使用明文，所以usePlaintext参数，表示明文进行请求。
/**
* 否则会报错：
*/
ManagedChannel channel = ManagedChannelBuilder.forAddress("test.technocore.network",88)
.usePlaintext()
.build();
com.google.protobuf.Empty request = com.google.protobuf.Empty.newBuilder().build();
ResourceServiceGrpc.ResourceServiceBlockingStub resourceServiceBlockingStub = ResourceServiceGrpc.newBlockingStub(channel);
ListPowerSummaryResponse response = resourceServiceBlockingStub.listPowerSummary(request);
log.info("listPowerSummary(), response:{}", response);
}
}
可能遇到的报错：
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
原因：
构建channel的时候需要加上usePlaintext(),即表示通讯要使用明文通讯。
/*
* io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
* Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
*
* at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
* at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
* at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
* at com.platon.metis.storage.grpc.lib.api.ResourceServiceGrpc$ResourceServiceBlockingStub.listPowerSummary(ResourceServiceGrpc.java:522)
* at com.platon.metis.storage.grpc.GrpcClientDemo.listPowerSummary(GrpcClientDemo.java:39)
* at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
* at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
* at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
* at java.lang.reflect.Method.invoke(Method.java:498)
* at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
* at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
* at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
* at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
* at org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:74)
* at org.springframework.test.context.junit4.statements.RunAfterTestExecutionCallbacks.evaluate(RunAfterTestExecutionCallbacks.java:84)
* at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
* at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
* at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
* at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
* at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:251)
* at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
* at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
* at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
* at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
* at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
* at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
* at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
* at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
* at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
* at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
* at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
* at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
* at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
* at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
* at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
* at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
* Caused by: io.grpc.netty.shaded.io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 000012040000000000000300000080000400010000000500ffffff0000040800000000007fff00000000080700000000000000000000000001
* at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1232)
* at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1300)
* at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
* at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
* at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
* at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
* at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
* at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
* at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
* at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
* at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
* at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
* at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
* at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
* at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
* at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
* at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
* at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
* at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
* at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
* at java.lang.Thread.run(Thread.java:748)
*
*/
2. grpc+ssl客户端demo
package com.platon.metis.storage.grpc;
import com.platon.metis.storage.grpc.lib.api.ListPowerSummaryResponse;
import com.platon.metis.storage.grpc.lib.api.ResourceServiceGrpc;
import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;
import lombok.extern.slf4j.Slf4j;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.test.context.junit4.SpringRunner;
@RunWith(SpringRunner.class)
@Slf4j
public class GrpcClientDemo {
@Test
public void listPowerSummary() {
log.info("start to test listPowerSummary()...");
//grpcs需要使用密文，所以要去掉usePlaintext
/**
* 否则会报错：
*/
ManagedChannel channel = ManagedChannelBuilder.forAddress("test.technocore.network",880)
// .usePlaintext()
.build();
com.google.protobuf.Empty request = com.google.protobuf.Empty.newBuilder().build();
ResourceServiceGrpc.ResourceServiceBlockingStub resourceServiceBlockingStub = ResourceServiceGrpc.newBlockingStub(channel);
ListPowerSummaryResponse response = resourceServiceBlockingStub.listPowerSummary(request);
log.info("listPowerSummary(), response:{}", response);
}
}
可能遇到的报错：
io.grpc.StatusRuntimeException: INTERNAL: http2 exception
原因：
构建channel的时候需要去掉usePlaintext(),即表示通讯要密文加密。
/**
* 否则会报错：
* io.grpc.StatusRuntimeException: INTERNAL: http2 exception
*
* at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
* at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
* at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
* at com.platon.metis.storage.grpc.lib.api.ResourceServiceGrpc$ResourceServiceBlockingStub.listPowerSummary(ResourceServiceGrpc.java:522)
* at com.platon.metis.storage.grpc.GrpcClientDemo.listPowerSummary(GrpcClientDemo.java:111)
* at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
* at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
* at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
* at java.lang.reflect.Method.invoke(Method.java:498)
* at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
* at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
* at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
* at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
* at org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:74)
* at org.springframework.test.context.junit4.statements.RunAfterTestExecutionCallbacks.evaluate(RunAfterTestExecutionCallbacks.java:84)
* at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
* at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
* at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
* at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
* at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:251)
* at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
* at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
* at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
* at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
* at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
* at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
* at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
* at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
* at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
* at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
* at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
* at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
* at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
* at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
* at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
* at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
* Caused by: io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
* at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:108)
* at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.verifyFirstFrameIsSettings(Http2ConnectionHandler.java:338)
* at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:239)
* at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
* at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
* at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
* at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
* at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
* at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
* at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
* at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
* at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
* at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
* at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
* at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
* at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
* at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
* at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
* at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
* at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
* at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
* at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
* at java.lang.Thread.run(Thread.java:748)
*/