13、Kubernetes核心技术 - Ingress
目录
一、概述
二、Ingress工作原理
三、Ingress 使用
3.1)、Ingress-http方式
3.1.1)、创建Ingress Controller和对应的Service
3.1.2)、创建tomcat的Pod和Service
3.1.3)、创建nginx的Pod和Service
3.1.4)、创建ingress http代理
3.1.5)、配置本地host文件
3.1.6)、访问nginx
3.1.7)、访问tomcat
3.2)、Ingress-https方式
3.2.1)、生成证书
3.2.2)、生成秘钥
3.2.3)、创建ingress https代理
3.2.4)、测试
一、概述
通过前面的文章，我们知道k8s暴露服务常见的方式主要有两种：
- 1、NodePort
- 2、LoadBalancer
但是这两种方式在集群规模比较大的时候，都会存在一些问题。
- NodePort：每个主机节点都要暴露出一个端口，如果Service很多的话，会占用很多集群机器的端口，并且端口过多，也不便于管理；
- LoadBalancer：每一个Service 都需要一个 LB（负载均衡器），如果Service很多的话，将会造成LB浪费，并且LoadBalancer需要外部的负载均衡设备进行支持；
为了解决上述的问题，k8s抽象了Ingress的概念，通过配置 Ingress和Ingress Controller 来通过匹配 URL 的方式实现 HTTP/HTTPS 代理，只需要一个NodePort或者一个LB就可以满足暴露多个Service 需求。
二、Ingress工作原理
Ingress包含Ingress和Ingress Controller两大组件。
- Ingress：k8s中的一个对象，主要是用来定义请求如何转发到Service的规则；
- Ingress Controller：具体实现反向代理及负载均衡的程序，核心是一个Deployment，对Ingress定义的规则进行解析，根据配置的规则来实现请求转发，实现的方式有很多，比如Nginx，Haproxy，Istio等，Ingress Controller需要编写的yaml资源清单主要有：Deployment, Service, ConfigMap, ServiceAccount（Auth），其中Service的类型可以是NodePort或者LoadBalancer。
Ingress的工作流程大体如下：
- 1、用户编写 Ingress 规则，说明哪个域名对应哪个Service；
- 2、Ingress Contoller 动态感知 ingress 编写的规则，通过跟 Ingress 交互得知某个域名对应哪个Service，跟 Kubernetes API 交互获取到Service 地址等信息后，然后生成对应的反向代理规则；
- 3、Ingress Controller通过Kubernetes API写入到负载均衡器中，然后负载均衡器 reload 该规则，就可以实现服务发现；
- 4、客户端请求负载均衡器，由负载均衡器转发到后端Pod节点；
三、Ingress 使用
3.1)、Ingress-http方式
3.1.1)、创建Ingress Controller和对应的Service
vim nginx-ingress.yaml
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
---
# Source: ingress-nginx/templates/controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
automountServiceAccountToken: true
---
# Source: ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
data:
---
# Source: ingress-nginx/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
name: ingress-nginx
rules:
- apiGroups:
- ''
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ''
resources:
- nodes
verbs:
- get
- apiGroups:
- ''
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
---
# Source: ingress-nginx/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
name: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
rules:
- apiGroups:
- ''
resources:
- namespaces
verbs:
- get
- apiGroups:
- ''
resources:
- configmaps
- pods
- secrets
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- configmaps
resourceNames:
- ingress-controller-leader
verbs:
- get
- update
- apiGroups:
- ''
resources:
- configmaps
verbs:
- create
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
---
# Source: ingress-nginx/templates/controller-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-service-webhook.yaml
apiVersion: v1
kind: Service
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller-admission
namespace: ingress-nginx
spec:
type: ClusterIP
ports:
- name: https-webhook
port: 443
targetPort: webhook
appProtocol: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
annotations:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
type: NodePort
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
appProtocol: http
- name: https
port: 443
protocol: TCP
targetPort: https
appProtocol: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
revisionHistoryLimit: 10
minReadySeconds: 0
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
spec:
dnsPolicy: ClusterFirst
containers:
- name: controller
image: k8s.gcr.io/ingress-nginx/controller:v1.0.0@sha256:0851b34f69f69352bf168e6ccf30e1e20714a264ab1ecd1933e4d8c0fc3215c6
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
args:
- /nginx-ingress-controller
- --election-id=ingress-controller-leader
- --controller-class=k8s.io/ingress-nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
runAsUser: 101
allowPrivilegeEscalation: true
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: LD_PRELOAD
value: /usr/local/lib/libmimalloc.so
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
ports:
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
protocol: TCP
- name: webhook
containerPort: 8443
protocol: TCP
volumeMounts:
- name: webhook-cert
mountPath: /usr/local/certificates/
readOnly: true
resources:
requests:
cpu: 100m
memory: 90Mi
nodeSelector:
kubernetes.io/os: linux
serviceAccountName: ingress-nginx
terminationGracePeriodSeconds: 300
volumes:
- name: webhook-cert
secret:
secretName: ingress-nginx-admission
---
# Source: ingress-nginx/templates/controller-ingressclass.yaml
# We don't support namespaced ingressClass yet
# So a ClusterRole and a ClusterRoleBinding is required
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: nginx
namespace: ingress-nginx
spec:
controller: k8s.io/ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
# before changing this value, check the required kubernetes version
# https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
name: ingress-nginx-admission
webhooks:
- name: validate.nginx.ingress.kubernetes.io
matchPolicy: Equivalent
rules:
- apiGroups:
- networking.k8s.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- ingresses
failurePolicy: Fail
sideEffects: None
admissionReviewVersions:
- v1
clientConfig:
service:
namespace: ingress-nginx
name: ingress-nginx-controller-admission
path: /networking/v1/ingresses
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ingress-nginx-admission
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
rules:
- apiGroups:
- admissionregistration.k8s.io
resources:
- validatingwebhookconfigurations
verbs:
- get
- update
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ingress-nginx-admission
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
rules:
- apiGroups:
- ''
resources:
- secrets
verbs:
- get
- create
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: ingress-nginx-admission-create
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
template:
metadata:
name: ingress-nginx-admission-create
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
containers:
- name: create
image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0@sha256:f3b6b39a6062328c095337b4cadcefd1612348fdd5190b1dcbcb9b9e90bd8068
imagePullPolicy: IfNotPresent
args:
- create
- --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
- --namespace=$(POD_NAMESPACE)
- --secret-name=ingress-nginx-admission
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
nodeSelector:
kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: ingress-nginx-admission-patch
namespace: ingress-nginx
annotations:
helm.sh/hook: post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
template:
metadata:
name: ingress-nginx-admission-patch
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
containers:
- name: patch
image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0@sha256:f3b6b39a6062328c095337b4cadcefd1612348fdd5190b1dcbcb9b9e90bd8068
imagePullPolicy: IfNotPresent
args:
- patch
- --webhook-name=ingress-nginx-admission
- --namespace=$(POD_NAMESPACE)
- --patch-mutating=false
- --secret-name=ingress-nginx-admission
- --patch-failure-policy=Fail
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
nodeSelector:
kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 2000安装ingress controller：
$ kubectl create -f nginx-ingress.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created查看Pod以及Service，注意，需要通过-n指定ingress-nginx命名空间：
$ kubectl get pod -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-gbkfg 0/1 Completed 0 67s
ingress-nginx-admission-patch-hc69f 0/1 Completed 0 67s
ingress-nginx-controller-78dccfdb9-tsxt7 1/1 Running 0 67s
$ kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.96.214.135 80:31132/TCP,443:30503/TCP 73s
ingress-nginx-controller-admission ClusterIP 10.111.41.178 443/TCP 73s 可以看到，ingress-nginx-controller这个Service，通过NodePort方式暴露出端口，我们在外部就能访问到。
3.1.2)、创建tomcat的Pod和Service
vim tomcat.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat
spec:
replicas: 3
selector:
matchLabels:
app: tomcat
template:
metadata:
labels:
app: tomcat
spec:
containers:
- name: tomcat
image: tomcat:8.5-jre10-slim
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: tomcat
spec:
selector:
app: tomcat
clusterIP: None
type: ClusterIP
ports:
- port: 8080
targetPort: 8080$ vim tomcat.yaml
$ kubectl create -f tomcat.yaml
deployment.apps/tomcat created
service/tomcat created
$ kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/tomcat-ff7c8b896-kdtss 1/1 Running 0 14s
pod/tomcat-ff7c8b896-tsxhj 1/1 Running 0 14s
pod/tomcat-ff7c8b896-wmhh7 1/1 Running 0 14s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 443/TCP 20d
service/tomcat ClusterIP None 8080/TCP 14s 3.1.3)、创建nginx的Pod和Service
vim nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.17.1
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
selector:
app: nginx
clusterIP: None
type: ClusterIP
ports:
- port: 80
targetPort: 80$ vim nginx.yaml
$ kubectl create -f nginx.yaml
deployment.apps/nginx created
service/nginx created
$ kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/nginx-54c4864cd8-dj58r 1/1 Running 0 7s
pod/nginx-54c4864cd8-fdgsv 1/1 Running 0 7s
pod/nginx-54c4864cd8-xcmsx 1/1 Running 0 7s
pod/tomcat-ff7c8b896-kdtss 1/1 Running 0 47s
pod/tomcat-ff7c8b896-tsxhj 1/1 Running 0 47s
pod/tomcat-ff7c8b896-wmhh7 1/1 Running 0 47s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 443/TCP 20d
service/nginx ClusterIP None 80/TCP 7s
service/tomcat ClusterIP None 8080/TCP 47s 3.1.4)、创建ingress http代理
vim ingress-proxy-http.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-http
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: nginx.test.com
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: nginx # 需要对应前面我们自己定义的service的名称
port:
number: 80
- host: tomcat.test.com
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: tomcat # 需要对应前面我们自己定义的service的名称
port:
number: 8080创建ingress：
$ vim ingress-proxy-http.yaml
$ kubectl create -f ingress-proxy-http.yaml
ingress.networking.k8s.io/ingress-http created
$ kubectl get ingress ingress-http -o wide
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-http nginx.test.com,tomcat.test.com 172.30.2.2 80 7s 查看ingress详情：
controlplane $ kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-54c4864cd8-dj58r 1/1 Running 0 116s 192.168.0.9 controlplane
nginx-54c4864cd8-fdgsv 1/1 Running 0 116s 192.168.1.12 node01
nginx-54c4864cd8-xcmsx 1/1 Running 0 116s 192.168.1.13 node01
tomcat-ff7c8b896-kdtss 1/1 Running 0 2m36s 192.168.0.8 controlplane
tomcat-ff7c8b896-tsxhj 1/1 Running 0 2m36s 192.168.1.10 node01
tomcat-ff7c8b896-wmhh7 1/1 Running 0 2m36s 192.168.1.11 node01
$ kubectl describe ingress ingress-http
Name: ingress-http
Labels:
Namespace: default
Address: 172.30.2.2
Ingress Class:
Default backend:
Rules:
Host Path Backends
---- ---- --------
nginx.test.com
/ nginx:80 (192.168.0.9:80,192.168.1.12:80,192.168.1.13:80)
tomcat.test.com
/ tomcat:8080 (192.168.0.8:8080,192.168.1.10:8080,192.168.1.11:8080)
Annotations: kubernetes.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 67s (x2 over 73s) nginx-ingress-controller Scheduled for sync 
可以看到，在Ingress资源清单中，我们定义了域名跟Service的映射规则，当访问nginx.test.com这个域名的时候，后端Service就是nginx，后端Pod就对应着我们前面定义好的三个nginx的Pod，并且可以看到Pod的IP地址，端口信息；当访问tomcat.test.com这个域名的时候，后端Service就是tomcat，后端Pod就对应着我们前面定义好的三个tomcat的Pod，并且可以看到Pod的IP地址，端口信息。
3.1.5)、配置本地host文件
因为我们这个域名是自定义配置的，所以需要配置 Hosts 解析（本地解析）。
vim /etc/hosts
[Master节点的IP地址] nginx.test.com
[Master节点的IP地址] tomcat.test.com3.1.6)、访问nginx
$ kubectl get svc -n ingress-nginx ingress-nginx-controller
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.96.214.135 80:31132/TCP,443:30503/TCP 25m
$ kubectl get ingress ingress-http -o wide
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-http nginx.test.com,tomcat.test.com 172.30.2.2 80 6m48s
$ curl nginx.test.com:31132
Welcome to nginx!
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.
Thank you for using nginx.
3.1.7)、访问tomcat
$ curl tomcat.test.com:31132
Apache Tomcat/8.5.35
Apache Tomcat/8.5.35
If you're seeing this, you've successfully installed Tomcat. Congratulations!
![[tomcat logo]](tomcat.png)
Developer Quick Start
Managing Tomcat
For security, access to the manager webapp is restricted.
Users are defined in:
$CATALINA_HOME/conf/tomcat-users.xml
In Tomcat 8.5 access to the manager application is split between
different users. Read more...
Release Notes
Changelog
Migration Guide
Security Notices
Documentation
Tomcat 8.5 Documentation
Tomcat 8.5 Configuration
Tomcat Wiki
Find additional important configuration information in:
$CATALINA_HOME/RUNNING.txt
Developers may be interested in:
Getting Help
FAQ and Mailing Lists
The following mailing lists are available:
- tomcat-announce
Important announcements, releases, security vulnerability notifications. (Low volume).
- tomcat-users
User support and discussion
- taglibs-user
User support and discussion for Apache Taglibs
- tomcat-dev
Development mailing list, including commit messages
Copyright ©1999-2023 Apache Software Foundation. All Rights Reserved
3.2)、Ingress-https方式
3.2.1)、生成证书
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/C=CN/ST=BJ/L=BJ/O=nginx/CN=test.com"$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/C=CN/ST=BJ/L=BJ/O=nginx/CN=test.com"
Generating a RSA private key
.............................................................................+++++
...................................+++++
writing new private key to 'tls.key'
-----3.2.2)、生成秘钥
$ kubectl create secret tls tls-secret --key tls.key --cert tls.crt
secret/tls-secret created
$ kubectl get secret/tls-secret -o yaml
apiVersion: v1
data:
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURkVENDQWwyZ0F3SUJBZ0lVRS9ub2tFQWVqczhURmlRS0hmN0hLY0JZR0Ywd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1NqRUxNQWtHQTFVRUJoTUNRMDR4Q3pBSkJnTlZCQWdNQWtKS01Rc3dDUVlEVlFRSERBSkNTakVPTUF3RwpBMVVFQ2d3RmJtZHBibmd4RVRBUEJnTlZCQU1NQ0hSbGMzUXVZMjl0TUI0WERUSXpNREV4TWpBNE1ETXpObG9YCkRUSTBNREV4TWpBNE1ETXpObG93U2pFTE1Ba0dBMVVFQmhNQ1EwNHhDekFKQmdOVkJBZ01Ba0pLTVFzd0NRWUQKVlFRSERBSkNTakVPTUF3R0ExVUVDZ3dGYm1kcGJuZ3hFVEFQQmdOVkJBTU1DSFJsYzNRdVkyOXRNSUlCSWpBTgpCZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF6UW5Pa29WWGVkQW9UTWVHQ1hGTURWV05GNHBPCnNmNm9vdmxNUlVTa3BoV3ZpeFhVMHJJTXZMbkRnVkY2ZGt5M05oS1ViQU5NZ01YV2t5U041NnNnOFc3bTJ4MHoKYW41UHBwWmlnVTM0SWRLcnlEMEdhcnlOa2FoQ0xJcHRnN1ViRmo2TkpwaWl4bFJQMEhRMUhqTGVsQ2E2NFpCbApYcTJFa0tnL1JKalpKUDgxVnFOR3AyT2ZibjYreFpLTmMzMTVhOThndGpUbHkvczNaYXBiYTZXOGFlajhob3IyCmZGVG1uYnNGVzdEREFoWGVEY1ZDZ1VYK20vQ216RHJzSUxnRGl6Q0pGV2s1cTQ1bG02RmlwWnFoZnRIWkxzenoKeUZDT0FrVkkrMHRYYktvcHNKaUJSKzlzZGV3RTI1VXBUMGdTSys0ZFR4eWhHeTlSQXBUWm5CTmVqd0lEQVFBQgpvMU13VVRBZEJnTlZIUTRFRmdRVXpqOWdFa3lvOU54TzluK1dra2VoQnpuQ0txMHdId1lEVlIwakJCZ3dGb0FVCnpqOWdFa3lvOU54TzluK1dra2VoQnpuQ0txMHdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QU5CZ2txaGtpRzl3MEIKQVFzRkFBT0NBUUVBaCtyWGI4cHhONG90ZEpVRHhCbURBaHVlTDJyU2QzdjlJVGYxQlQvMllaTnlmckNVZXl3NQpnVWtlTkpMTksxYzhlQkpIT2h3dEphUEJxQk9vcTNDQmUrVFduZkthMjRadE9HcUVxSkhwQVIvb0gzRExVaWpaCjRHeDN2NnRXZGdqMkpQS25iSmVpZENhQUxMblRoSG9RQk9sWTVRV1hDZHJsYmlsakNKWFIwOEZ6M2JTSGZXeGEKQ1RjUmhicUhmbVBNN1QwcU9RQkJjUGtKSUQ0MVRLS1k3cFNiSWtHSkdxcThEeTh3amFrbm56M0lMV0xocDFZWApuQWFIRnhuUUtIck5KSWJ6b3pEeTVwLzNaUmU2RlIxK2xWMXIybFZlTjQ4WkpNa3FUZDZaK2pVS3p1bzh0TEFoCm01WWJzMDdUSDFuYlJDM1lqTXI1clhua1RXMmNyUG5teXc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2Z0lCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktnd2dnU2tBZ0VBQW9JQkFRRE5DYzZTaFZkNTBDaE0KeDRZSmNVd05WWTBYaWs2eC9xaWkrVXhGUktTbUZhK0xGZFRTc2d5OHVjT0JVWHAyVExjMkVwUnNBMHlBeGRhVApKSTNucXlEeGJ1YmJIVE5xZmsrbWxtS0JUZmdoMHF2SVBRWnF2STJScUVJc2ltMkR0UnNXUG8wbW1LTEdWRS9RCmREVWVNdDZVSnJyaGtHVmVyWVNRcUQ5RW1Oa2svelZXbzBhblk1OXVmcjdGa28xemZYbHIzeUMyTk9YTCt6ZGwKcWx0cnBieHA2UHlHaXZaOFZPYWR1d1Zic01NQ0ZkNE54VUtCUmY2YjhLYk1PdXdndUFPTE1Ja1ZhVG1yam1XYgpvV0tsbXFGKzBka3V6UFBJVUk0Q1JVajdTMWRzcWltd21JRkg3MngxN0FUYmxTbFBTQklyN2gxUEhLRWJMMUVDCmxObWNFMTZQQWdNQkFBRUNnZ0VBSzUyYmNMV21yd1hsbkxlUXZxOTRJeG9Bd25Jc1hWYXpucjl6UndUY0ZXeGwKaEN1bnN3NG5BWHB1cVRLZzMwdGdhY000Q1dZR1B4cFdrOFJoMzFjYTBlRHFnTS9NWVNudlhadDk1czJrSytXNQpud1hFT0s1Vld4ZTB3KzdLYndyMmNSdnJRTHVWWEtVUXVXSU83cGFabkdCRXZyak5wMzgzeE9IVlc0MDhHODlBCk9VaExOTXJnVXFxNk14OXhMMTREbHBzU3VDOHJEWGN5L3RpMVBZY1Q4elhNaGlFaWhkSkt6bi9FN2ZPakRFZW4KaGIyZW85ckU2Z2xXNmQxaEVwTEUwczNpcGEzRU9IVCtYdFl3K2xuc1FGVi92Z1g0eWp3ckJGaCt1UE5MQytLRAo0QW1vQU9CdTZia3dqVFJHVzduK3RBTkRkcFQzSFEvSUQrVzQwWFVPd1FLQmdRRDZXbmg4MldPUjVMdHRTOTNSCkl1Ui8rQWpIZGxOUm13KzdId3VxQkhUemhvbkhNSlRJc3Y2ZnRYaUJjT21GcENDRi9BNk1VMEIzNGNHY2VyRGgKMDhNYi9wbHJCUzJkRGttYkJCb0hsekFMN0VaQ2xIR212eElJVTVJOWVlU0o2R21jRm5UdG1KQXdMV3huRVdHZwo2SXNBcnBiQUNPcVlnOGxNb05STFFSYlZJUUtCZ1FEUnFiQmU4cjVHTGFjWXZIWmpJUFhEbFFhUWJRbkF3OXRWCitEY0RhSXNvdlA5bEFmbWVmLytKV1VyaGlZMjUrd1lBdzRWaG9KbU9OR0t5Vms3cGhmb2N3VUxTai9XQTJHQW4KMHhIb29EaWdKUlpnRGhGYkxBR2dJbHZ5ajJtT3JNQTN3MGZxcmZ5Y2NxaUdPUWFiT1RXK0k2ekRMSVFWNHNNNQoxNzB6YXdVTnJ3S0JnUUMvcGF3Z251V3JGSGNxRnU2cUgxVGVaWWlLOEppZG10OHk5blFlWmFVRm0xTXphWXdzCkMySUI3cEJlUG9wMVpJd0ZKVXdadmMxWE9EblhTTkRKUHl0WDdkR1I3SGZiaDhsdTRuZnVvWjBGTTQwaGdMTnoKMjBhNW13Z1RDWkdheEpDOERURGJoWE96ZmNKRE1Eb1ZUSTdFS2VKN0JSaFpRbUZCeU1oYkIydXNJUUtCZ0dxUgpKSXR1T0paMUYreXp4anptZ09TVGQ4MnJEbjBEVUJYblh4Y0ZhVTRnbW5PZHlHSHV6MVRocFBUME5LeG1VZ3ZMClZaMWt3d2dMeW1xNUFjMGlneGxnaGZWK3BKdUQzNFArZmFqTzMxbE80SXVjT3VncmV2d0kydEYwWVlQYWZLMGIKUGcranRiVlRkaU1iRXllTXNhdGVlSzZQanVlTTVnd2RneGFDZkFRVEFvR0JBTTEvcEJ6eEwwSmFOMVRVcStrNwpVUXpsUEFObzMwbU1Sd1R4b1N0ODQwWk5Ibmk2RUNWVDg1amFKUWhKd0FFcnVsenQ3TERMUCsvWTlRVHZRMDhRCkdIVFRNYU9xYUJ2Rk4rYmxldlUyaERzNzY1dlBlelNoUHdoRHdKcWlSaTFLbHJBa0ZwaldkT2oxV3pDRzhSN0QKZjFxYWpDdUZiTWMrNm5BZXA0U1NvZC9OCi0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
kind: Secret
metadata:
creationTimestamp: "2023-01-12T08:04:00Z"
name: tls-secret
namespace: default
resourceVersion: "4758"
uid: 0ef46f60-cf7b-443e-ab3f-92b9bfd87c4f
type: kubernetes.io/tls3.2.3)、创建ingress https代理
vim ingress-proxy-https.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-https
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
tls:
- hosts:
- nginx.test.com
- tomcat.test.com
secretName: tls-secret
rules:
- host: nginx.test.com
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: nginx
port:
number: 80
- host: tomcat.test.com
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: tomcat
port:
number: 8080$ kubectl create -f ingress-proxy-https.yaml
ingress.networking.k8s.io/ingress-https created
$ kubectl get ingress ingress-https -o wide
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-https nginx.test.com,tomcat.test.com 172.30.2.2 80, 443 15s
$ kubectl describe ingress ingress-https
Name: ingress-https
Labels:
Namespace: default
Address: 172.30.2.2
Ingress Class:
Default backend:
TLS:
tls-secret terminates nginx.test.com,tomcat.test.com
Rules:
Host Path Backends
---- ---- --------
nginx.test.com
/ nginx:80 (192.168.0.9:80,192.168.1.12:80,192.168.1.13:80)
tomcat.test.com
/ tomcat:8080 (192.168.0.8:8080,192.168.1.10:8080,192.168.1.11:8080)
Annotations: kubernetes.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 17s (x2 over 27s) nginx-ingress-controller Scheduled for sync 
3.2.4)、测试
$ curl -H "Host: nginx.test.com" https://10.96.214.135 -v -k
* Trying 10.96.214.135:443...
* TCP_NODELAY set
* Connected to 10.96.214.135 (10.96.214.135) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* start date: Jan 12 07:32:36 2023 GMT
* expire date: Jan 12 07:32:36 2024 GMT
* issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55cb6ff13210)
> GET / HTTP/2
> Host: nginx.test.com
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< date: Thu, 12 Jan 2023 08:26:51 GMT
< content-type: text/html
< content-length: 612
< last-modified: Tue, 25 Jun 2019 12:19:45 GMT
< etag: "5d121161-264"
< accept-ranges: bytes
< strict-transport-security: max-age=15724800; includeSubDomains
<
Welcome to nginx!
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.
Thank you for using nginx.
* Connection #0 to host 10.96.214.135 left intact
$ curl -H "Host: tomcat.test.com" https://10.96.214.135 -v -k
* Trying 10.96.214.135:443...
* TCP_NODELAY set
* Connected to 10.96.214.135 (10.96.214.135) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* start date: Jan 12 07:32:36 2023 GMT
* expire date: Jan 12 07:32:36 2024 GMT
* issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x555d40f23210)
> GET / HTTP/2
> Host: tomcat.test.com
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< date: Thu, 12 Jan 2023 08:27:39 GMT
< content-type: text/html;charset=UTF-8
< strict-transport-security: max-age=15724800; includeSubDomains
<
Apache Tomcat/8.5.35
Apache Tomcat/8.5.35
If you're seeing this, you've successfully installed Tomcat. Congratulations!
Developer Quick Start
Managing Tomcat
For security, access to the manager webapp is restricted.
Users are defined in:
$CATALINA_HOME/conf/tomcat-users.xml
In Tomcat 8.5 access to the manager application is split between
different users. Read more...
Release Notes
Changelog
Migration Guide
Security Notices
Documentation
Tomcat 8.5 Documentation
Tomcat 8.5 Configuration
Tomcat Wiki
Find additional important configuration information in:
$CATALINA_HOME/RUNNING.txt
Developers may be interested in:
Getting Help
FAQ and Mailing Lists
The following mailing lists are available:
- tomcat-announce
Important announcements, releases, security vulnerability notifications. (Low volume).
- tomcat-users
User support and discussion
- taglibs-user
User support and discussion for Apache Taglibs
- tomcat-dev
Development mailing list, including commit messages
Copyright ©1999-2023 Apache Software Foundation. All Rights Reserved
* Connection #0 to host 10.96.214.135 left intact