helm3-安装logstash

前言

本文介绍如何在k8s集群中使用helm来创建logstash，供大家参考学习。

准备

  • 阿里云K8S集群
  • 安装helm

安装Kafka

我们首先添加一下helm库，并且搜索到logstash

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm search repo logstash
NAME                        CHART VERSION    APP VERSION    DESCRIPTION
bitnami/logstash            5.4.1            8.7.1          Logstash is an open source data processing engi...
stable/dmarc2logstash       1.3.1            1.0.3          DEPRECATED Provides a POP3-polled DMARC XML rep...
stable/logstash             2.4.3            7.1.1          DEPRECATED - Logstash is an open source, server...
bitnami/dataplatform-bp2    12.0.5           1.0.1          DEPRECATED This Helm chart can be used for the ...

我们安装的Chart版本：5.4.1 ，App版本：8.7.1，接着我们把源码pull下来，如下：

$ helm pull bitnami/logstash

解压一下下载后的logstash-5.4.1.tgz文件，如下所示 ：

image.png

之后我们打开values.yaml文件，如下所示：

## @section Global parameters
## Global Docker image parameters
## Please, note that this will override the image parameters, including dependencies, configured to use the global value
## Current available global Docker image parameters: imageRegistry, imagePullSecrets and storageClass

## @param global.imageRegistry Global Docker image registry
## @param global.imagePullSecrets Global Docker registry secret names as an array
## @param global.storageClass Global StorageClass for Persistent Volume(s)
##
global:
  imageRegistry: ""
  ## E.g.
  ## imagePullSecrets:
  ##   - myRegistryKeySecretName
  ##
  imagePullSecrets: []
  storageClass: ""

## @section Common parameters

## @param kubeVersion Force target Kubernetes version (using Helm capabilities if not set)
##
kubeVersion: ""
## @param nameOverride String to partially override logstash.fullname template (will maintain the release name)
##
nameOverride: ""
## @param fullnameOverride String to fully override logstash.fullname template
##
fullnameOverride: ""
## @param clusterDomain Default Kubernetes cluster domain
##
clusterDomain: cluster.local
## @param commonAnnotations Annotations to add to all deployed objects
##
commonAnnotations: {}
## @param commonLabels Labels to add to all deployed objects
##
commonLabels: {}
## @param extraDeploy Array of extra objects to deploy with the release (evaluated as a template).
##
extraDeploy: []

## Enable diagnostic mode in the deployment
##
diagnosticMode:
  ## @param diagnosticMode.enabled Enable diagnostic mode (all probes will be disabled and the command will be overridden)
  ##
  enabled: false
  ## @param diagnosticMode.command Command to override all containers in the deployment
  ##
  command:
    - sleep
  ## @param diagnosticMode.args Args to override all containers in the deployment
  ##
  args:
    - infinity

## @section Logstash parameters

## Bitnami Logstash image
## ref: https://hub.docker.com/r/bitnami/logstash/tags/
## @param image.registry Logstash image registry
## @param image.repository Logstash image repository
## @param image.tag Logstash image tag (immutable tags are recommended)
## @param image.digest Logstash image digest in the way sha256:aa.... Please note this parameter, if set, will override the tag
## @param image.pullPolicy Logstash image pull policy
## @param image.pullSecrets Specify docker-registry secret names as an array
## @param image.debug Specify if debug logs should be enabled
##
image:
  registry: docker.io
  repository: bitnami/logstash
  tag: 8.6.0-debian-11-r0
  digest: ""
  ## Specify a imagePullPolicy. Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
  ## ref: https://kubernetes.io/docs/user-guide/images/#pre-pulling-images
  ##
  pullPolicy: IfNotPresent
  ## Optionally specify an array of imagePullSecrets (secrets must be manually created in the namespace)
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ## Example:
  ## pullSecrets:
  ##   - myRegistryKeySecretName
  ##
  pullSecrets: []
  ## Set to true if you would like to see extra information on logs
  ##
  debug: false
## @param hostAliases Add deployment host aliases
## https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/
##
hostAliases: []
## @param configFileName Logstash configuration file name. It must match the name of the configuration file mounted as a configmap.
##
configFileName: logstash.conf
## @param enableMonitoringAPI Whether to enable the Logstash Monitoring API or not  Kubernetes cluster domain
##
enableMonitoringAPI: true
## @param monitoringAPIPort Logstash Monitoring API Port
##
monitoringAPIPort: 9600
## @param extraEnvVars Array containing extra env vars to configure Logstash
## For example:
## extraEnvVars:
##  - name: ELASTICSEARCH_HOST
##    value: "x.y.z"
##
extraEnvVars: []
## @param extraEnvVarsSecret To add secrets to environment
##
extraEnvVarsSecret: ""
## @param extraEnvVarsCM To add configmaps to environment
##
extraEnvVarsCM: ""
## @param input [string] Input Plugins configuration
## ref: https://www.elastic.co/guide/en/logstash/current/input-plugins.html
##
input: |-
  # udp {
  #   port => 1514
  #   type => syslog
  # }
  # tcp {
  #   port => 1514
  #   type => syslog
  # }
  http { port => 8080 }
## @param filter Filter Plugins configuration
## ref: https://www.elastic.co/guide/en/logstash/current/filter-plugins.html
## e.g:
## filter: |-
##   grok {
##     match => { "message" => "%{COMBINEDAPACHELOG}" }
##   }
##   date {
##     match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
##   }
##
filter: ""
## @param output [string] Output Plugins configuration
## ref: https://www.elastic.co/guide/en/logstash/current/output-plugins.html
##
output: |-
  # elasticsearch {
  #   hosts => ["${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}"]
  #   manage_template => false
  #   index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  # }
  # gelf {
  #   host => "${GRAYLOG_HOST}"
  #   port => ${GRAYLOG_PORT}
  # }
  stdout {}
## @param existingConfiguration Name of existing ConfigMap object with the Logstash configuration (`input`, `filter`, and `output` will be ignored).
##
existingConfiguration: ""
## @param enableMultiplePipelines Allows user to use multiple pipelines
## ref: https://www.elastic.co/guide/en/logstash/master/multiple-pipelines.html
##
enableMultiplePipelines: false
## @param extraVolumes Array to add extra volumes (evaluated as a template)
## extraVolumes:
##   - name: myvolume
##     configMap:
##       name: myconfigmap
##
extraVolumes: []
## @param extraVolumeMounts Array to add extra mounts (normally used with extraVolumes, evaluated as a template)
## extraVolumeMounts:
##   - mountPath: /opt/bitnami/desired-path
##     name: myvolume
##     readOnly: true
##
extraVolumeMounts: []

 ## ServiceAccount for Logstash
 ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
 ##
serviceAccount:
  ## @param serviceAccount.create Enable creation of ServiceAccount for Logstash pods
  ##
  create: true
  ## @param serviceAccount.name The name of the service account to use. If not set and `create` is `true`, a name is generated
  ## If not set and create is true, a name is generated using the logstash.serviceAccountName template
  ##
  name: ""
  ## @param serviceAccount.automountServiceAccountToken Allows automount of ServiceAccountToken on the serviceAccount created
  ## Can be set to false if pods using this serviceAccount do not need to use K8s API
  ##
  automountServiceAccountToken: true
  ## @param serviceAccount.annotations Additional custom annotations for the ServiceAccount
  ##
  annotations: {}

## @param containerPorts [array] Array containing the ports to open in the Logstash container (evaluated as a template)
##
containerPorts:
  - name: http
    containerPort: 8080
    protocol: TCP
  - name: monitoring
    containerPort: 9600
    protocol: TCP
  ## - name: syslog-udp
  ##   containerPort: 1514
  ##   protocol: UDP
  ## - name: syslog-tcp
  ##   containerPort: 1514
  ##   protocol: TCP
  ##
## @param initContainers Add additional init containers to the Logstash pod(s)
## ref: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
## e.g:
## initContainers:
##  - name: your-image-name
##    image: your-image
##    imagePullPolicy: Always
##    command: ['sh', '-c', 'echo "hello world"']
##
initContainers: []
## @param sidecars Add additional sidecar containers to the Logstash pod(s)
## e.g:
## sidecars:
##   - name: your-image-name
##     image: your-image
##     imagePullPolicy: Always
##     ports:
##       - name: portname
##         containerPort: 1234
##
sidecars: []
## @param replicaCount Number of Logstash replicas to deploy
##
replicaCount: 1
## @param updateStrategy.type Update strategy type (`RollingUpdate`, or `OnDelete`)
## ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets
##
updateStrategy:
  type: RollingUpdate
## @param podManagementPolicy Pod management policy
## https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#pod-management-policies
##
podManagementPolicy: OrderedReady
## @param podAnnotations Pod annotations
## Ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
##
podAnnotations: {}
## @param podLabels Extra labels for Logstash pods
## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
##
podLabels: {}
## @param podAffinityPreset Pod affinity preset. Ignored if `affinity` is set. Allowed values: `soft` or `hard`
## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity
##
podAffinityPreset: ""
## @param podAntiAffinityPreset Pod anti-affinity preset. Ignored if `affinity` is set. Allowed values: `soft` or `hard`
## Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity
##
podAntiAffinityPreset: soft
## Node affinity preset
## Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity
##
nodeAffinityPreset:
  ## @param nodeAffinityPreset.type Node affinity preset type. Ignored if `affinity` is set. Allowed values: `soft` or `hard`
  ##
  type: ""
  ## @param nodeAffinityPreset.key Node label key to match. Ignored if `affinity` is set.
  ## E.g.
  ## key: "kubernetes.io/e2e-az-name"
  ##
  key: ""
  ## @param nodeAffinityPreset.values Node label values to match. Ignored if `affinity` is set.
  ## E.g.
  ## values:
  ##   - e2e-az1
  ##   - e2e-az2
  ##
  values: []
## @param affinity Affinity for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
## Note: podAffinityPreset, podAntiAffinityPreset, and nodeAffinityPreset will be ignored when it's set
##
affinity: {}
## @param nodeSelector Node labels for pod assignment
## ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}
## @param tolerations Tolerations for pod assignment
## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
tolerations: []
## @param priorityClassName Pod priority
## ref: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/
##
priorityClassName: ""
## @param schedulerName Name of the k8s scheduler (other than default)
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
schedulerName: ""
## @param terminationGracePeriodSeconds In seconds, time the given to the Logstash pod needs to terminate gracefully
## ref: https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
##
terminationGracePeriodSeconds: ""
## @param topologySpreadConstraints Topology Spread Constraints for pod assignment
## https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/
## The value is evaluated as a template
##
topologySpreadConstraints: []

## K8s Security Context for Logstash pods
## Configure Pods Security Context
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod
## @param podSecurityContext.enabled Enabled Logstash pods' Security Context
## @param podSecurityContext.fsGroup Set Logstash pod's Security Context fsGroup
##
podSecurityContext:
  enabled: true
  fsGroup: 1001
## Configure Container Security Context
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod
## @param containerSecurityContext.enabled Enabled Logstash containers' Security Context
## @param containerSecurityContext.runAsUser Set Logstash containers' Security Context runAsUser
## @param containerSecurityContext.runAsNonRoot Set Logstash container's Security Context runAsNonRoot
##
containerSecurityContext:
  enabled: true
  runAsUser: 1001
  runAsNonRoot: true
## @param command Override default container command (useful when using custom images)
##
command: []
## @param args Override default container args (useful when using custom images)
##
args: []
## @param lifecycleHooks for the Logstash container(s) to automate configuration before or after startup
##
lifecycleHooks: {}
## Logstash containers' resource requests and limits
## ref: https://kubernetes.io/docs/user-guide/compute-resources/
## We usually recommend not to specify default resources and to leave this as a conscious
## choice for the user. This also increases chances charts run on environments with little
## resources, such as Minikube. If you do want to specify resources, uncomment the following
## lines, adjust them as necessary, and remove the curly braces after 'resources:'.
## @param resources.limits The resources limits for the Logstash container
## @param resources.requests The requested resources for the Logstash container
##
resources:
  ## Example:
  ## limits:
  ##    cpu: 100m
  ##    memory: 128Mi
  limits: {}
  ## Examples:
  ## requests:
  ##    cpu: 100m
  ##    memory: 128Mi
  requests: {}
## Configure extra options for Logstash containers' liveness, readiness and startup probes
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes
## @param startupProbe.enabled Enable startupProbe
## @param startupProbe.initialDelaySeconds Initial delay seconds for startupProbe
## @param startupProbe.periodSeconds Period seconds for startupProbe
## @param startupProbe.timeoutSeconds Timeout seconds for startupProbe
## @param startupProbe.failureThreshold Failure threshold for startupProbe
## @param startupProbe.successThreshold Success threshold for startupProbe
##
startupProbe:
  enabled: false
  initialDelaySeconds: 60
  periodSeconds: 10
  timeoutSeconds: 5
  successThreshold: 1
  failureThreshold: 6
## @param livenessProbe.enabled Enable livenessProbe
## @param livenessProbe.initialDelaySeconds Initial delay seconds for livenessProbe
## @param livenessProbe.periodSeconds Period seconds for livenessProbe
## @param livenessProbe.timeoutSeconds Timeout seconds for livenessProbe
## @param livenessProbe.failureThreshold Failure threshold for livenessProbe
## @param livenessProbe.successThreshold Success threshold for livenessProbe
##
livenessProbe:
  enabled: true
  initialDelaySeconds: 60
  periodSeconds: 10
  timeoutSeconds: 5
  successThreshold: 1
  failureThreshold: 6
## @param readinessProbe.enabled Enable readinessProbe
## @param readinessProbe.initialDelaySeconds Initial delay seconds for readinessProbe
## @param readinessProbe.periodSeconds Period seconds for readinessProbe
## @param readinessProbe.timeoutSeconds Timeout seconds for readinessProbe
## @param readinessProbe.failureThreshold Failure threshold for readinessProbe
## @param readinessProbe.successThreshold Success threshold for readinessProbe
##
readinessProbe:
  enabled: true
  initialDelaySeconds: 60
  periodSeconds: 10
  timeoutSeconds: 5
  successThreshold: 1
  failureThreshold: 6
## @param customStartupProbe Custom startup probe for the Web component
##
customStartupProbe: {}
## @param customLivenessProbe Custom liveness probe for the Web component
##
customLivenessProbe: {}
## @param customReadinessProbe Custom readiness probe for the Web component
##
customReadinessProbe: {}
## Service parameters
##
service:
  ## @param service.type Kubernetes service type (`ClusterIP`, `NodePort`, or `LoadBalancer`)
  ##
  type: ClusterIP
  ## @param service.ports [array] Logstash service ports (evaluated as a template)
  ##
  ports:
    - name: http
      port: 8080
      targetPort: http
      protocol: TCP
    ## - name: syslog-udp
    ##   port: 1514
    ##   targetPort: syslog-udp
    ##   protocol: UDP
    ## - name: syslog-tcp
    ##   port: 1514
    ##   targetPort: syslog-tcp
    ##   protocol: TCP
    ##
  ## @param service.loadBalancerIP loadBalancerIP if service type is `LoadBalancer`
  ##
  loadBalancerIP: ""
  ## @param service.loadBalancerSourceRanges Addresses that are allowed when service is LoadBalancer
  ## https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service
  ## e.g:
  ## loadBalancerSourceRanges:
  ## - 10.10.10.0/24
  ##
  loadBalancerSourceRanges: []
  ## @param service.externalTrafficPolicy External traffic policy, configure to Local to preserve client source IP when using an external loadBalancer
  ## ref https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip
  ##
  externalTrafficPolicy: ""
  ## @param service.clusterIP Static clusterIP or None for headless services
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#choosing-your-own-ip-address
  ## e.g:
  ## clusterIP: None
  ##
  clusterIP: ""
  ## @param service.annotations Annotations for Logstash service
  ##
  annotations: {}
  ## @param service.sessionAffinity Session Affinity for Kubernetes service, can be "None" or "ClientIP"
  ## If "ClientIP", consecutive client requests will be directed to the same Pod
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies
  ##
  sessionAffinity: None
  ## @param service.sessionAffinityConfig Additional settings for the sessionAffinity
  ## sessionAffinityConfig:
  ##   clientIP:
  ##     timeoutSeconds: 300
  ##
  sessionAffinityConfig: {}
## Persistence parameters
##
persistence:
  ## @param persistence.enabled Enable Logstash data persistence using PVC
  ##
  enabled: false
  ## @param persistence.existingClaim A manually managed Persistent Volume and Claim
  ## If defined, PVC must be created manually before volume will be bound
  ## The value is evaluated as a template
  ##
  existingClaim: ""
  ## @param persistence.storageClass PVC Storage Class for Logstash data volume
  ## If defined, storageClassName: 
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ## set, choosing the default provisioner.
  ##
  storageClass: ""
  ## @param persistence.accessModes PVC Access Mode for Logstash data volume
  ##
  accessModes:
    - ReadWriteOnce
  ## @param persistence.size PVC Storage Request for Logstash data volume
  ##
  size: 2Gi
  ## @param persistence.annotations Annotations for the PVC
  ##
  annotations: {}
  ## @param persistence.mountPath Mount path of the Logstash data volume
  ##
  mountPath: /bitnami/logstash/data
  ## @param persistence.selector Selector to match an existing Persistent Volume for WordPress data PVC
  ## If set, the PVC can't have a PV dynamically provisioned for it
  ## E.g.
  ## selector:
  ##   matchLabels:
  ##     app: my-app
  ##
  selector: {}
## Init Container parameters
## Change the owner and group of the persistent volume(s) mountpoint(s) to 'runAsUser:fsGroup' on each component
## values from the securityContext section of the component
##
volumePermissions:
  ## @param volumePermissions.enabled Enable init container that changes the owner and group of the persistent volume(s) mountpoint to `runAsUser:fsGroup`
  ##
  enabled: false
  ## The security context for the volumePermissions init container
  ## @param volumePermissions.securityContext.runAsUser User ID for the volumePermissions init container
  ##
  securityContext:
    runAsUser: 0
  ## @param volumePermissions.image.registry Init container volume-permissions image registry
  ## @param volumePermissions.image.repository Init container volume-permissions image repository
  ## @param volumePermissions.image.tag Init container volume-permissions image tag (immutable tags are recommended)
  ## @param volumePermissions.image.digest Init container volume-permissions image digest in the way sha256:aa.... Please note this parameter, if set, will override the tag
  ## @param volumePermissions.image.pullPolicy Init container volume-permissions image pull policy
  ## @param volumePermissions.image.pullSecrets Specify docker-registry secret names as an array
  ##
  image:
    registry: docker.io
    repository: bitnami/bitnami-shell
    tag: 11-debian-11-r70
    digest: ""
    ## Specify a imagePullPolicy
    ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
    ## ref: https://kubernetes.io/docs/user-guide/images/#pre-pulling-images
    ##
    pullPolicy: IfNotPresent
    ## Optionally specify an array of imagePullSecrets (secrets must be manually created in the namespace)
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    ## Example:
    ## pullSecrets:
    ##   - myRegistryKeySecretName
    ##
    pullSecrets: []
  ## Init Container resource requests and limits
  ## ref: https://kubernetes.io/docs/user-guide/compute-resources/
  ## We usually recommend not to specify default resources and to leave this as a conscious
  ## choice for the user. This also increases chances charts run on environments with little
  ## resources, such as Minikube. If you do want to specify resources, uncomment the following
  ## lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  ## @param volumePermissions.resources.limits Init container volume-permissions resource limits
  ## @param volumePermissions.resources.requests Init container volume-permissions resource requests
  ##
  resources:
    ## Example:
    ## limits:
    ##    cpu: 100m
    ##    memory: 128Mi
    limits: {}
    ## Examples:
    ## requests:
    ##    cpu: 100m
    ##    memory: 128Mi
    requests: {}
## Configure the ingress resource that allows you to access the
## Logstash installation. Set up the URL
## ref: https://kubernetes.io/docs/user-guide/ingress/
##
ingress:
  ## @param ingress.enabled Enable ingress controller resource
  ##
  enabled: false
  ## @param ingress.selfSigned Create a TLS secret for this ingress record using self-signed certificates generated by Helm
  ##
  selfSigned: false
  ## @param ingress.pathType Ingress Path type
  ##
  pathType: ImplementationSpecific
  ## @param ingress.apiVersion Override API Version (automatically detected if not set)
  ##
  apiVersion: ""
  ## @param ingress.hostname Default host for the ingress resource
  ##
  hostname: logstash.local
  ## @param ingress.path The Path to Logstash. You may need to set this to '/*' in order to use this with ALB ingress controllers.
  ##
  path: /
  ## @param ingress.annotations Additional annotations for the Ingress resource. To enable certificate autogeneration, place here your cert-manager annotations.
  ## For a full list of possible ingress annotations, please see
  ## ref: https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md
  ## Use this parameter to set the required annotations for cert-manager, see
  ## ref: https://cert-manager.io/docs/usage/ingress/#supported-annotations
  ##
  ## e.g:
  ## annotations:
  ##   kubernetes.io/ingress.class: nginx
  ##   cert-manager.io/cluster-issuer: cluster-issuer-name
  ##
  annotations: {}
  ## @param ingress.tls Enable TLS configuration for the hostname defined at ingress.hostname parameter
  ## TLS certificates will be retrieved from a TLS secret with name: {{- printf "%s-tls" .Values.ingress.hostname }}
  ## You can use the ingress.secrets parameter to create this TLS secret or relay on cert-manager to create it
  ##
  tls: false
  ## @param ingress.extraHosts The list of additional hostnames to be covered with this ingress record.
  ## Most likely the hostname above will be enough, but in the event more hosts are needed, this is an array
  ## extraHosts:
  ## - name: logstash.local
  ##   path: /
  ##
  extraHosts: []
  ## @param ingress.extraPaths Any additional arbitrary paths that may need to be added to the ingress under the main host.
  ## For example: The ALB ingress controller requires a special rule for handling SSL redirection.
  ## extraPaths:
  ## - path: /*
  ##   backend:
  ##     serviceName: ssl-redirect
  ##     servicePort: use-annotation
  ##
  extraPaths: []
  ## @param ingress.extraRules The list of additional rules to be added to this ingress record. Evaluated as a template
  ## Useful when looking for additional customization, such as using different backend
  ##
  extraRules: []
  ## @param ingress.extraTls The tls configuration for additional hostnames to be covered with this ingress record.
  ## see: https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
  ## extraTls:
  ## - hosts:
  ##     - logstash.local
  ##   secretName: logstash.local-tls
  ##
  extraTls: []
  ## @param ingress.secrets If you're providing your own certificates, please use this to add the certificates as secrets
  ## key and certificate should start with -----BEGIN CERTIFICATE----- or
  ## -----BEGIN RSA PRIVATE KEY-----
  ##
  ## name should line up with a tlsSecret set further up
  ## If you're using cert-manager, this is unneeded, as it will create the secret for you if it is not set
  ##
  ## It is also possible to create and manage the certificates outside of this helm chart
  ## Please see README.md for more information
  ##
  ## secrets:
  ##   - name: logstash.local-tls
  ##     key:
  ##     certificate:
  ##
  secrets: []
  ## @param ingress.ingressClassName IngressClass that will be be used to implement the Ingress (Kubernetes 1.18+)
  ## This is supported in Kubernetes 1.18+ and required if you have more than one IngressClass marked as the default for your cluster .
  ## ref: https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/
  ##
  ingressClassName: ""

## Pod disruption budget configuration
## ref: https://kubernetes.io/docs/concepts/workloads/pods/disruptions/
## @param pdb.create If true, create a pod disruption budget for pods.
## @param pdb.minAvailable Minimum number / percentage of pods that should remain scheduled
## @param pdb.maxUnavailable Maximum number / percentage of pods that may be made unavailable
##
pdb:
  create: false
  minAvailable: 1
  maxUnavailable: ""
  • 配置一：选择StorageClass

因为我们使用的是阿里云，所以这里我们只能使用阿里云存储，另外阿里云存储分为云盘和NAS、OSS等，因为云盘最低要20Gi我们用不到这么大的，所以我们使用NAS即可，而NAS服务要开通和创建StorageClass，这里前期我写了一篇如何创建阿里云NAS的StorageClass文章，参考上面的来操作即可创建名为alicloud-nas-fs的StorageClass，这样系统就会自动通过PVC(存储卷声明)绑定择StorageClass动态创建出PV(存储卷)。

persistence:
  enabled: true
  # NAS
  storageClass: alibabacloud-cnfs-nas
  size: 2Gi

  • 配置二：配置service
    设置service成外部也可以访问的NodePort模式，然后把官方注释的syslog-udp和syslog-tcp全部放开，如下所示：

    service:
type: NodePort
ports:
  - name: http
    port: 8080
    targetPort: http
    protocol: TCP
  - name: syslog-udp
    port: 1514
    targetPort: syslog-udp
    protocol: UDP
  - name: syslog-tcp
    port: 1514
    targetPort: syslog-tcp
    protocol: TCP
  • 配置三：配置containerPorts

把cotainerPorts注释部分全部放开，如下所示：

containerPorts:
  - name: http
    containerPort: 8080
    protocol: TCP
  - name: monitoring
    containerPort: 9600
    protocol: TCP
  - name: syslog-udp
    containerPort: 1514
    protocol: UDP
  - name: syslog-tcp
    containerPort: 1514
    protocol: TCP
  • 配置四：配置input和output

这里要配置input中codec为json_lines格式，并且type=syslog(这个是自定义标签可以是任何名称，如：abc)，配置output中elasticsearch的连接信息，如下所示：

input: |-
  udp {
   port => 1514
   type => syslog
   codec => json_lines
  }
  tcp {
   port => 1514
   type => syslog
   codec => json_lines
  }
  http { port => 8080 }
output: |-  
  if [active] != "" {
    elasticsearch {
      hosts => ["xxx.xxx.xxx.xxx:xxxx"]
      index => "%{active}-logs-%{+YYYY.MM.dd}"
    }
  } else {
    elasticsearch {
      hosts => ["xxx.xxx.xxx.xxx:xxxx"]
      index => "ignore-logs-%{+YYYY.MM.dd}"
    }
  }
  stdout { }

配置完成之后，完整的values.yaml文件如下所示 ：

values.yaml

global:
  storageClass: alibabacloud-cnfs-nas
service:
  type: NodePort
  ports:
    - name: http
      port: 8080
      targetPort: http
      protocol: TCP
    - name: syslog-udp
      port: 1514
      targetPort: syslog-udp
      protocol: UDP
    - name: syslog-tcp
      port: 1514
      targetPort: syslog-tcp
      protocol: TCP
persistence:
  enabled: true
  # NAS
  storageClass: alibabacloud-cnfs-nas
  size: 2Gi
containerPorts:
  - name: http
    containerPort: 8080
    protocol: TCP
  - name: monitoring
    containerPort: 9600
    protocol: TCP
  - name: syslog-udp
    containerPort: 1514
    protocol: UDP
  - name: syslog-tcp
    containerPort: 1514
    protocol: TCP
input: |-
  udp {
   port => 1514
   type => syslog
   codec => json
  }
  tcp {
   port => 1514
   type => syslog
   codec => json
  }
  http { port => 8080 }
output: |-  
  if [active] != "" {
    elasticsearch {
      hosts => ["47.97.208.153:31474"]
      index => "%{active}-logs-%{+YYYY.MM.dd}"
    }
  } else {
    elasticsearch {
      hosts => ["47.97.208.153:31474"]
      index => "logs-%{+YYYY.MM.dd}"
    }
  }
  stdout { }

接着，我们来通过如下helm命令来创建logstash，

$ kubectl create namespace logstash
$ helm install -f values.yaml logstash bitnami/logstash --namespace logstash

创建成功之后，会有如下输出，

NAME: logstash
LAST DEPLOYED: Wed Sep 27 10:58:35 2023
NAMESPACE: logstash
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: logstash
CHART VERSION: 5.4.1
APP VERSION: 8.7.1

** Please be patient while the chart is being deployed **

Logstash can be accessed through following DNS names from within your cluster:

    Logstash: logstash.logstash.svc.cluster.local

To access Logstash from outside the cluster execute the following commands:

    export NODE_PORT=$(kubectl get --namespace logstash -o jsonpath="{.spec.ports[0].nodePort}" services logstash)
    export NODE_IP=$(kubectl get nodes --namespace logstash -o jsonpath="{.items[0].status.addresses[0].address}")
    echo "http://${NODE_IP}:${NODE_PORT}"

这样等待logstash的创建了，我们也可以通过命令查看是否创建成功：

☁ kubectl get all -n logstash
NAME             READY   STATUS    RESTARTS   AGE
pod/logstash-0   1/1     Running   0          34m

NAME                        TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)                                        AGE
service/logstash            NodePort    192.168.132.104           8080:32345/TCP,1514:30227/UDP,1514:30227/TCP   34m
service/logstash-headless   ClusterIP   None                      8080/TCP,1514/UDP,1514/TCP                     34m

NAME                        READY   AGE
statefulset.apps/logstash   1/1     34m

获取ip和port

通过下面命令获取：

 export NODE_PORT=$(kubectl get --namespace logstash -o jsonpath="{.spec.ports[0].nodePort}" services logstash)
    export NODE_IP=$(kubectl get nodes --namespace logstash -o jsonpath="{.items[0].status.addresses[0].address}")
    echo "http://${NODE_IP}:${NODE_PORT}"

输出：

http://xxx.xxx.xxx.xxx:xxxx

使用curl测试logstash是否安装成功

1、首先把logstash上面的http端口8080映射成本地，如下所示：

kubectl port-forward service/logstash 8080:8080 -nlogstash

2、使用curl测试是否成功，如下所示：

curl -X POST -d '{"message": "Hello World","env": "dev"}' http://localhost:8080

注意：上面的32345对应的是http协议，而30227对应的是TCP协议，这个不要弄错了。

注意：如果是本地环境，需要配置brokers: xxx.xxx.xxx.xxx:30227，如果是线上并且跟kafka是同一个k8s服务的话，直接使用域名形式访问，如：brokers: logstash-0.logstash-headless.logstash.svc.cluster.local:30227

删除

1、如果想删除，直接到阿里云的后台选择helm应用点击删除即可
2、记得还要把PVC也要删除，否则一直在记费。

总结

1、使用helm安装logstash集群非常方便
2、如果想开发本地访问，可以配置service使用NodePort类型，如果是生产首选默认的ClusterIP模式
3、因为阿里云云盘最低要求20G，而我们不需要这么大的，所以使用NAS更划算(有钱当我没说)
4、logback是使用tcp连接的logstash，测试的时候我使用的是http协议，大家不要搞乱了！

你可能感兴趣的:(helm3logstash)

  • java工厂模式 3213213333332132 java抽象工厂
    工厂模式有 1、工厂方法 2、抽象工厂方法。 下面我的实现是抽象工厂方法, 给所有具体的产品类定一个通用的接口。 package 工厂模式; /** * 航天飞行接口 * * @Description * @author FuJianyong * 2015-7-14下午02:42:05 */ public interface SpaceF
  • nginx频率限制+python测试 ronin47 nginx 频率 python
           部分内容参考：http://www.abc3210.com/2013/web_04/82.shtml 首先说一下遇到这个问题是因为网站被攻击，阿里云报警，想到要限制一下访问频率，而不是限制ip（限制ip的方案稍后给出）。nginx连接资源被吃空返回状态码是502，添加本方案限制后返回599，与正常状态码区别开。步骤如下：
  • java线程和线程池的使用 dyy_gusi ThreadPoolthreadRunnabletimer
    java线程和线程池 一、创建多线程的方式     java多线程很常见，如何使用多线程，如何创建线程，java中有两种方式，第一种是让自己的类实现Runnable接口，第二种是让自己的类继承Thread类。其实Thread类自己也是实现了Runnable接口。具体使用实例如下： 1、通过实现Runnable接口方式 1 2
  • Linux 171815164 linux
    ubuntu kernel http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1.2-unstable/ 安卓sdk代理 mirrors.neusoft.edu.cn        80 输入法和jdk sudo apt-get install fcitx su
  • Tomcat JDBC Connection Pool g21121 Connection
           Tomcat7 抛弃了以往的DBCP 采用了新的Tomcat Jdbc Pool 作为数据库连接组件，事实上DBCP已经被Hibernate 所抛弃，因为他存在很多问题，诸如：更新缓慢，bug较多，编译问题，代码复杂等等。        Tomcat Jdbc P
  • 敲代码的一点想法 永夜-极光 java随笔感想
                  入门学习java编程已经半年了,一路敲代码下来,现在也才1w+行代码量,也就菜鸟水准吧,但是在整个学习过程中,我一直在想,为什么很多培训老师,网上的文章都是要我们背一些代码?比如学习Arraylist的时候,教师就让我们先参考源代码写一遍,然
  • jvm指令集 程序员是怎么炼成的 jvm 指令集
    转自：http://blog.csdn.net/hudashi/article/details/7062675#comments       将值推送至栈顶时 const ldc  push   load指令 const系列 该系列命令主要负责把简单的数值类型送到栈顶。(从常量池或者局部变量push到栈顶时均使用) 0x02 &nbs
  • Oracle字符集的查看查询和Oracle字符集的设置修改 aijuans oracle
    本文主要讨论以下几个部分：如何查看查询oracle字符集、 修改设置字符集以及常见的oracle utf8字符集和oracle exp 字符集问题。 一、什么是Oracle字符集 Oracle字符集是一个字节数据的解释的符号集合,有大小之分,有相互的包容关系。ORACLE 支持国家语言的体系结构允许你使用本地化语言来存储，处理，检索数据。它使数据库工具，错误消息，排序次序，日期，时间，货
  • png在Ie6下透明度处理方法 antonyup_2006 css浏览器FirebugIE
    由于之前到深圳现场支撑上线，当时为了解决个控件下载，我机器上的IE8老报个错，不得以把ie8卸载掉，换个Ie6,问题解决了，今天出差回来，用ie6登入另一个正在开发的系统，遇到了Png图片的问题，当然升级到ie8(ie8自带的开发人员工具调试前端页面JS之类的还是比较方便的，和FireBug一样，呵呵)，这个问题就解决了，但稍微做了下这个问题的处理。 我们知道PNG是图像文件存储格式，查询资
  • 表查询常用命令高级查询方法(二) 百合不是茶 oracle分页查询分组查询联合查询
    ----------------------------------------------------分组查询 group by    having   --平均工资和最高工资   select avg(sal)平均工资,max(sal)  from emp ;  --每个部门的平均工资和最高工资
  • uploadify3.1版本参数使用详解 bijian1013 JavaScriptuploadify3.1
    使用：     绑定的界面元素<input id='gallery'type='file'/>$("#gallery").uploadify({设置参数，参数如下}); 设置的属性： id: jQuery(this).attr('id'),//绑定的input的ID langFile: 'http://ww
  • 精通Oracle10编程SQL(17)使用ORACLE系统包 bijian1013 oracle数据库plsql
    /* *使用ORACLE系统包 */ --1.DBMS_OUTPUT --ENABLE:用于激活过程PUT,PUT_LINE,NEW_LINE,GET_LINE和GET_LINES的调用 --语法：DBMS_OUTPUT.enable(buffer_size in integer default 20000); --DISABLE:用于禁止对过程PUT,PUT_LINE,NEW
  • 【JVM一】JVM垃圾回收日志 bit1129 垃圾回收
    将JVM垃圾回收的日志记录下来，对于分析垃圾回收的运行状态，进而调整内存分配(年轻代，老年代，永久代的内存分配)等是很有意义的。JVM与垃圾回收日志相关的参数包括： -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc -XX:+PrintGC 通
  • Toast使用 白糖_ toast
    Android中的Toast是一种简易的消息提示框，toast提示框不能被用户点击，toast会根据用户设置的显示时间后自动消失。     创建Toast 两个方法创建Toast makeText(Context context, int resId, int duration)     参数：context是toast显示在
  • angular.identity boyitech AngularJSAngularJS API
    angular.identiy 描述: 返回它第一参数的函数. 此函数多用于函数是编程. 使用方法: angular.identity(value); 参数详解: Param Type Details value * to be returned. 返回值: 传入的value 实例代码: <!DOCTYPE HTML>
  • java-两整数相除，求循环节 bylijinnan java
    import java.util.ArrayList; import java.util.List; public class CircleDigitsInDivision { /** * 题目：求循环节，若整除则返回NULL，否则返回char*指向循环节。先写思路。函数原型：char*get_circle_digits(unsigned k,unsigned j)
  • Java 日期 周 年 Chen.H javaC++cC#
    /** * java日期操作(月末、周末等的日期操作) * * @author * */ public class DateUtil { /** */ /** * 取得某天相加(减)後的那一天 * * @param date * @param num *
  • [高考与专业]欢迎广大高中毕业生加入自动控制与计算机应用专业 comsci 计算机
          不知道现在的高校还设置这个宽口径专业没有,自动控制与计算机应用专业,我就是这个专业毕业的,这个专业的课程非常多,既要学习自动控制方面的课程,也要学习计算机专业的课程,对数学也要求比较高.....如果有这个专业,欢迎大家报考...毕业出来之后,就业的途径非常广.....      以后
  • 分层查询（Hierarchical Queries） daizj oracle递归查询层次查询
    Hierarchical Queries If a table contains hierarchical data, then you can select rows in a hierarchical order using the hierarchical query clause: hierarchical_query_clause::= start with condi
  • 数据迁移 daysinsun 数据迁移
    最近公司在重构一个医疗系统，原来的系统是两个.Net系统，现需要重构到java中。数据库分别为SQL Server和Mysql，现需要将数据库统一为Hana数据库，发现了几个问题，但最后通过努力都解决了。 1、原本通过Hana的数据迁移工具把数据是可以迁移过去的，在MySQl里面的字段为TEXT类型的到Hana里面就存储不了了，最后不得不更改为clob。 2、在数据插入的时候有些字段特别长
  • C语言学习二进制的表示示例 dcj3sjt126com cbasic
    进制的表示示例 # include <stdio.h> int main(void) { int i = 0x32C; printf("i = %d\n", i); /* printf的用法 %d表示以十进制输出 %x或%X表示以十六进制的输出 %o表示以八进制输出 */ return 0; }
  • NsTimer 和 UITableViewCell 之间的控制 dcj3sjt126com ios
    情况是这样的: 一个UITableView, 每个Cell的内容是我自定义的 viewA viewA上面有很多的动画, 我需要添加NSTimer来做动画, 由于TableView的复用机制, 我添加的动画会不断开启, 没有停止, 动画会执行越来越多.   解决办法: 在配置cell的时候开始动画, 然后在cell结束显示的时候停止动画   查找cell结束显示的代理
  • MySql中case when then 的使用 fanxiaolong casewhenthenend
    select "主键", "项目编号", "项目名称","项目创建时间", "项目状态","部门名称","创建人" union (select pp.id as "主键", pp.project_number as &
  • Ehcache（01）——简介、基本操作 234390216 cacheehcache简介CacheManagercrud
    Ehcache简介 目录 1       CacheManager 1.1      构造方法构建 1.2      静态方法构建 2       Cache 2.1&
  • 最容易懂的javascript闭包学习入门 jackyrong JavaScript
    http://www.ruanyifeng.com/blog/2009/08/learning_javascript_closures.html 闭包（closure）是Javascript语言的一个难点，也是它的特色，很多高级应用都要依靠闭包实现。 下面就是我的学习笔记，对于Javascript初学者应该是很有用的。 一、变量的作用域 要理解闭包，首先必须理解Javascript特殊
  • 提升网站转化率的四步优化方案 php教程分享 数据结构PHP数据挖掘Google活动
    网站开发完成后,我们在进行网站优化最关键的问题就是如何提高整体的转化率，这也是营销策略里最最重要的方面之一，并且也是网站综合运营实例的结果。文中分享了四大优化策略：调查、研究、优化、评估，这四大策略可以很好地帮助用户设计出高效的优化方案。 PHP开发的网站优化一个网站最关键和棘手的是，如何提高整体的转化率，这是任何营销策略里最重要的方面之一，而提升网站转化率是网站综合运营实力的结果。今天，我就分
  • web开发里什么是HTML5的WebSocket？ naruto1990 Webhtml5浏览器socket
    当前火起来的HTML5语言里面，很多学者们都还没有完全了解这语言的效果情况，我最喜欢的Web开发技术就是正迅速变得流行的 WebSocket API。WebSocket 提供了一个受欢迎的技术，以替代我们过去几年一直在用的Ajax技术。这个新的API提供了一个方法，从客户端使用简单的语法有效地推动消息到服务器。让我们看一看6个HTML5教程介绍里 的 WebSocket API：它可用于客户端、服
  • Socket初步编程——简单实现群聊 Everyday都不同 socket网络编程初步认识
    初次接触到socket网络编程，也参考了网络上众前辈的文章。尝试自己也写了一下，记录下过程吧：   服务端：（接收客户端消息并把它们打印出来）   public class SocketServer { private List<Socket> socketList = new ArrayList<Socket>(); public s
  • 面试：Hashtable与HashMap的区别（结合线程） toknowme
    昨天去了某钱公司面试，面试过程中被问道 Hashtable与HashMap的区别？当时就是回答了一点，Hashtable是线程安全的，HashMap是线程不安全的，说白了，就是Hashtable是的同步的，HashMap不是同步的，需要额外的处理一下。   今天就动手写了一个例子，直接看代码吧 package com.learn.lesson001; import java
  • MVC设计模式的总结 xp9802 设计模式mvc框架IOC
    随着Web应用的商业逻辑包含逐渐复杂的公式分析计算、决策支持等，使客户机越 来越不堪重负，因此将系统的商业分离出来。单独形成一部分，这样三层结构产生了。 其中‘层’是逻辑上的划分。 三层体系结构是将整个系统划分为如图2.1所示的结构[3] （1）表现层（Presentation layer）：包含表示代码、用户交互GUI、数据验证。 该层用于向客户端用户提供GUI交互，它允许用户
按字母分类： ABCDEFGHIJKLMNOPQRSTUVWXYZ其他