MinIO创建bucket notificaion

因为要给外国同事做sharing，所以用英文总结的，大家将就看下哈。

Refer:

https://docs.min.io/docs/minio-bucket-notification-guide.html

1.prepare the docker-compose file

version: '3'
services:
  kafka:
    image: wurstmeister/kafka
    container_name: kafka_yuker_test3
    volumes: 
      - /etc/localtime:/etc/localtime
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: "localhost"
      KAFKA_ZOOKEEPER_CONNECT: myhbase:2181
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_LOG_RETENTION_HOURS: 120
      KAFKA_MESSAGE_MAX_BYTES: 10000000
      KAFKA_REPLICA_FETCH_MAX_BYTES: 10000000
      KAFKA_GROUP_MAX_SESSION_TIMEOUT_MS: 60000
      KAFKA_NUM_PARTITIONS: 3
      KAFKA_DELETE_RETENTION_MS: 1000 
    networks:
      - minionw
  hbase:
    image: harisekhon/hbase:latest
    hostname: myhbase
    container_name: hbase_yuker3
    ports:
      - 2181:2181
      - 16020:16020
      - 16010:16010
      - 16000:16000
    networks:
      - minionw
      
  minio:
    image: 'docker.io/bitnami/minio:2020.12.29'
    ports:
      - '9000:9000'
    environment:
      MINIO_ACCESS_KEY: minio
      MINIO_SECRET_KEY: minio123
    networks:
      - minionw
      
  mc:
    image: minio/mc
    entrypoint: ["/bin/sh"]
    stdin_open: true
    tty: true
    volumes:
      - ./.mc:/root/.mc
    networks:
      - minionw
      
networks:
  minionw: 
    driver: "bridge"

1. We added hbase, Because that hbase docker image contains zookeeper，kafka depends on zookeeper。

2. We added mc--minio client，It is used to access minio server.

3. They share one network，The driver of the network is bridge, The 3 container can access each other using servicename+port through this config.

2.Config the kafka related info in minioServer

There is 2 method to config the kafka related info in minio server.

1. environment variables.

2. Config the configuration by mc admin config set command.

We choose the second workaround, because we may need to config the kafka endpoint automaticly if the kafka endpoint changed.

2.1 Create a minio client by mc config command, so that we can connect to minio server using this client.

mc config host add local http://minio:9000 minio minio123

Notice:

• The 'local' is the name of minio client, you can change it to every word you want.
• The http://minio:9000 is the endpoint of minio server.
• The 'minio minio123' is the username and password of the minio server.

2.2 Check the minio client we created just now.

mc admin info local

2.3 Get the kafka configuration in minio server

mc admin config get local/ notify_kafka

2.4 Set kafka related configuration to minio server

mc admin config set local notify_kafka:idl tls_skip_verify="off" queue_dir="" queue_limit="0" sasl="off" sasl_password="" sasl_username="" tls_client_auth="0" tls="off" client_tls_cert="" client_tls_key="" brokers="kafka:9092" topic="bucketevents" version=""

Notice:

• The most import config is 'brokers', We should config kafka:9092 instead of localhost:9092.
• If the kafka endpoint cann't be reachable, there'll be a failure notification to tell you set the configuration failure.
• We also configed the kafka topic.

2.5 Restart the minio server

mc admin service restart local

2.6 Check all configurations of minio server

mc admin info local --json

We can see the bucket notification related configuration. And we can see the arn of bucket notification, it is arn:minio:sqs::idl:kafka.

2.7 Create a bucket

mc mb local/images

Alternatively, you can also create bucket by minio broswer.

2.8 Add bucket notification

mc event add local/images arn:minio:sqs::idl:kafka --suffix .jpg

This config means that minio will send a notification to the topic of bucketevents when we upload or remove a .jpg file in the bucket of images.

Alternatively, we can create the bucket notification by Amazon S3 SDK.

Filter filter =
        new Filter()
                .withS3KeyFilter(
                        new S3KeyFilter()
                                .withFilterRules(
                                        new FilterRule().withName("suffix").withValue(".jpg")));
BucketNotificationConfiguration notificationConfiguration =
        new BucketNotificationConfiguration();

notificationConfiguration.addConfiguration(
        "DL_NOTIFICATION_" + "pvtcloud",
        new com.amazonaws.services.s3.model.QueueConfiguration(
                "arn:minio:sqs::idl:kafka", EnumSet.of(S3Event.ObjectRemoved, S3Event.ObjectCreated))
                .withFilter(filter));

// Create the notification configuration request and set the bucket notification configuration.
SetBucketNotificationConfigurationRequest request =
        new SetBucketNotificationConfigurationRequest("images", notificationConfiguration);
amazonS3.setBucketNotificationConfiguration(request);

Important:

Minio only support QueueConfiguration currently, so Don't use other kind of configuration, e.g TopicNotification and so on.

2.9 List the bucket notification

mc event list local/images --json

We can also list the bucket notification we created just now.

2.10 Trigger the bucket Notification

We can upload a .jpg file to the bucket of images, Then consume the kafka topic of bucketevents by kafka-console-consumer.sh. We can get the bucket notificaiton.