Envoy proxy 源代码解读 - original_dst cluster
original_dst cluster 定义如下：
- name: cluster1
type: ORIGINAL_DST
lb_policy: ORIGINAL_DST_LB
type 设置成 ORIGINAL_DST，同时 lb_policy 需要设置为 ORIGINAL_DST_LB，否则配置解析的时候会报错：
[critical][main] [external/envoy/source/server/server.cc:92] error initializing configuration 'echo2_server.yaml': cluster: cluster type 'original_dst' may only be used with LB type 'original_dst_lb'
orignal_dst cluster 获取 upstream 的 ip 信息时来源有两个，一个是由设置了 use_original_dst: true 的 listener 所初始化 Envoy::Extensions::ListenerFilters::OriginalDst::OriginalDstFilter 解析请求的 original_dst（通常是 iptable redirect 而来）：
// envoy/source/extensions/filters/listener/original_dst.cc
Network::FilterStatus OriginalDstFilter::onAccept(Network::ListenerFilterCallbacks& cb) {
ENVOY_LOG(debug, "original_dst: New connection accepted");
Network::ConnectionSocket& socket = cb.socket();
const Network::Address::Instance& local_address = *socket.localAddress();
if (local_address.type() == Network::Address::Type::Ip) {
Network::Address::InstanceConstSharedPtr original_local_address =
getOriginalDst(socket.ioHandle().fd());
// A listener that has the use_original_dst flag set to true can still receive
// connections that are NOT redirected using iptables. If a connection was not redirected,
// the address returned by getOriginalDst() matches the local address of the new socket.
// In this case the listener handles the connection directly and does not hand it off.
if (original_local_address) {
// Restore the local address to the original one.
socket.restoreLocalAddress(original_local_address);
}
}
return Network::FilterStatus::Continue;
}
另外一个是，如果 cluster 设置了 use_http_header: true：
- name: cluster1
type: ORIGINAL_DST
lb_policy: ORIGINAL_DST_LB
original_dst_lb_config:
use_http_header: true
则直接从 x-envoy-original-dst-host 头信息里面读取 ip 信息：
// envoy/source/common/upstream/original_dst_cluster.cc
HostConstSharedPtr OriginalDstCluster::LoadBalancer::chooseHost(LoadBalancerContext* context) {
if (context) {
// Check if override host header is present, if yes use it otherwise check local address.
Network::Address::InstanceConstSharedPtr dst_host = nullptr;
if (use_http_header_) {
dst_host = requestOverrideHost(context);
}
...
}
⚠️注意：x-envoy-original-dst-host 只能设置 IP，否则会报错：
original_dst_load_balancer: invalid override header value.
listener 重定向
listener 如果设置了 use_original_dst: true，在接收到网络请求的时候，会根据原始的请求地址查找到匹配的 listener，然后重定向到过去，这是实现的代码：
// envoy/source/server/connection_handler_impl.cc
// Check if the socket may need to be redirected to another listener.
ActiveListenerBase* new_listener = nullptr;
if (hand_off_restored_destination_connections_ && socket_->localAddressRestored()) {
// Find a listener associated with the original destination address.
new_listener = listener_.parent_.findActiveListenerByAddress(*socket_->localAddress());
}
if (new_listener != nullptr) {
// TODO(sumukhs): Try to avoid dynamic_cast by coming up with a better interface design
ActiveTcpListener* tcp_listener = dynamic_cast(new_listener);
ASSERT(tcp_listener != nullptr, "ActiveSocket listener is expected to be tcp");
// Hands off connections redirected by iptables to the listener associated with the
// original destination address. Pass 'hand_off_restored_destination_connections' as false to
// prevent further redirection.
tcp_listener->onAccept(std::move(socket_),
false /* hand_off_restored_destination_connections */);
}
为什么不能把 envoy 当作通用的 http proxy?
Envoy 需要明确知道 upstream 的 ip 信息，只能处理以下几种情况：
- 在配置里面以
cluster形式存在，由 Envoy 自己来解析 IP 地址 - iptable forward，
upstream的 ip 可以通过original_dst拿到，因为下游发起请求时已经解析出来了upstream的 ip - 通过
x-envoy-original-dst头设置 IP 和端口
对于如下配置：
static_resources:
clusters:
- name: cluster1
type: ORIGINAL_DST
lb_policy: ORIGINAL_DST_LB
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 15001
use_original_dst: true
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
deprecated_v1: true
value:
stat_prefix: ingress_http
route_config:
virtual_hosts:
- routes:
- prefix: "/"
timeout_ms: 0
cluster: cluster1
auto_host_rewrite: true
domains:
- "*"
name: local_service
filters:
- name: router
config: {}
codec_type: auto
下面的使用方式会被告知 no health upstream，因为 cluster1 没有配置任何 host:
# 假设 envoy 监听 15001 端口
curl -H "Host: www.baidu.com" http://localhost:15001
而以下方式则会被告知 404:
# 假设 envoy 监听 15001 端口
http_proxy=localhost:15001 curl -v http://www.baidu.com
因为 path 信息不是相对路径，这是 curl 发出的命令：
GET http://www.baidu.com/ HTTP/1.1
Host: www.baidu.com
User-Agent: curl/7.54.0
Accept: */*
Proxy-Connection: Keep-Alive
而 Envoy 目前只支持 :path 设置为相对路径的情况，遇到 :path 不是以 / 开头的情况，则支持返回 404：
// envoy/source/common/http/conn_manager_impl.cc
void ConnectionManagerImpl::ActiveStream::decodeHeaders(HeaderMapPtr&& headers, bool end_stream) {
...
// Currently we only support relative paths at the application layer. We expect the codec to have
// broken the path into pieces if applicable. NOTE: Currently the HTTP/1.1 codec only does this
// when the allow_absolute_url flag is enabled on the HCM.
// https://tools.ietf.org/html/rfc7230#section-5.3 We also need to check for the existence of
// :path because CONNECT does not have a path, and we don't support that currently.
if (!request_headers_->Path() || request_headers_->Path()->value().getStringView().empty() ||
request_headers_->Path()->value().getStringView()[0] != '/') {
const bool has_path =
request_headers_->Path() && !request_headers_->Path()->value().getStringView().empty();
connection_manager_.stats_.named_.downstream_rq_non_relative_path_.inc();
sendLocalReply(Grpc::Common::hasGrpcContentType(*request_headers_), Code::NotFound, "", nullptr,
is_head_request_, absl::nullopt,
has_path ? StreamInfo::ResponseCodeDetails::get().AbsolutePath
: StreamInfo::ResponseCodeDetails::get().MissingPath);
return;
}
...
}