Logstash之4 对于logstash-filter-cipher plugin关于AES-GCM算法的支持的改进
1. logstash-filter-cipher 4.0.0中延续了以往对AES-GCM算法的使用。但是在新的ruby版本中AES-GCM算法引入了对于auth_data和auth_flag的使用，因此Logstash版本在从5.x升级到6.x或者7.x时，ruby同时进行了升级，使用高版本的openssl, 无意中引入了auth_data和auth_flag。因此使用logstash-filter-cipher 4.0.0对数据加密依然是成功的，但是无法解密，因为解密过程无法获取auth_data和auth_flag。
2. 因此对于AES-GCM算法的支持可以将加密数据中保存auth_data和auth_flag。这是非常必须的，否则解密就无法实现。
新的GCM算法使用ruby加密解密。参见：
https://ruby-doc.org/stdlib-2.3.1/libdoc/openssl/rdoc/OpenSSL/Cipher.html#class-OpenSSL::Cipher-label-Instantiating+a+Cipher
https://pycryptodome.readthedocs.io/en/latest/src/cipher/modern.html#gcm-mode
3. 对cipher.rb源码文件（./vendor/bundle/jruby/2.5.0/gems/logstash-filter-cipher-4.0.0/lib/logstash/filters/cipher.rb）可以进行改写，具体实现如下：
# encoding: utf-8
require "logstash/filters/base"
require "openssl"
# This filter parses a source and apply a cipher or decipher before
# storing it in the target.
#
class LogStash::Filters::Cipher < LogStash::Filters::Base
config_name "cipher"
# The field to perform filter
#
# Example, to use the @message field (default) :
# [source,ruby]
# filter { cipher { source => "message" } }
config :source, :validate => :string, :default => "message"
# The name of the container to put the result
#
# Example, to place the result into crypt :
# [source,ruby]
# filter { cipher { target => "crypt" } }
config :target, :validate => :string, :default => "message"
# Do we have to perform a `base64` decode or encode?
#
# If we are decrypting, `base64` decode will be done before.
# If we are encrypting, `base64` will be done after.
#
config :base64, :validate => :boolean, :default => true
# The key to use
#
# NOTE: If you encounter an error message at runtime containing the following:
#
# "java.security.InvalidKeyException: Illegal key size: possibly you need to install
# Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files for your JRE"
#
# Please read the following: https://github.com/jruby/jruby/wiki/UnlimitedStrengthCrypto
#
config :key, :validate => :string
# The key size to pad
#
# It depends of the cipher algorithm. If your key doesn't need
# padding, don't set this parameter
#
# Example, for AES-128, we must have 16 char long key. AES-256 = 32 chars
# [source,ruby]
# filter { cipher { key_size => 16 }
#
config :key_size, :validate => :number, :default => 16
# The character used to pad the key
config :key_pad, :default => "\0"
# The cipher algorithm
#
# A list of supported algorithms can be obtained by
# [source,ruby]
# puts OpenSSL::Cipher.ciphers
config :algorithm, :validate => :string, :required => true
# Encrypting or decrypting some data
#
# Valid values are encrypt or decrypt
config :mode, :validate => :string, :required => true
# Cipher padding to use. Enables or disables padding.
#
# By default encryption operations are padded using standard block padding
# and the padding is checked and removed when decrypting. If the pad
# parameter is zero then no padding is performed, the total amount of data
# encrypted or decrypted must then be a multiple of the block size or an
# error will occur.
#
# See EVP_CIPHER_CTX_set_padding for further information.
#
# We are using Openssl jRuby which uses default padding to PKCS5Padding
# If you want to change it, set this parameter. If you want to disable
# it, Set this parameter to 0
# [source,ruby]
# filter { cipher { cipher_padding => 0 }}
config :cipher_padding, :validate => :string
# Force an random IV to be used per encryption invocation and specify
# the length of the random IV that will be generated via:
#
# OpenSSL::Random.random_bytes(int_length)
#
# Enabling this will force the plugin to generate a unique
# random IV for each encryption call. This random IV will be prepended to the
# encrypted result bytes and then base64 encoded. On decryption "iv_random_length" must
# also be set to utilize this feature. Random IV's are better than statically
# hardcoded IVs
#
# For AES algorithms you can set this to a 16
# [source,ruby]
# filter { cipher { iv_random_length => 16 }}
config :iv_random_length, :validate => :number, :required => true
# If this is set the internal Cipher instance will be
# re-used up to @max_cipher_reuse times before being
# reset() and re-created from scratch. This is an option
# for efficiency where lots of data is being encrypted
# and decrypted using this filter. This lets the filter
# avoid creating new Cipher instances over and over
# for each encrypt/decrypt operation.
#
# This is optional, the default is no re-use of the Cipher
# instance and max_cipher_reuse = 1 by default
# [source,ruby]
# filter { cipher { max_cipher_reuse => 1000 }}
config :max_cipher_reuse, :validate => :number, :default => 1
#Add auth_data for gcm algorithm
config :auth_data, :validate => :string, :default => ""
def register
require 'base64' if @base64
init_cipher
end # def register
def filter(event)
#If decrypt or encrypt fails, we keep it it intact.
begin
if (event.get(@source).nil? || event.get(@source).empty?)
@logger.debug("Event to filter, event 'source' field: " + @source + " was null(nil) or blank, doing nothing")
return
end
#@logger.debug("Event to filter", :event => event)
data = event.get(@source)
if @mode == "decrypt"
data = Base64.strict_decode64(data) if @base64 == true
@random_iv = data.byteslice(0,@iv_random_length)
if @algorithm.include? "gcm"
auth_tag_length = 16
@cipher.auth_tag = data.byteslice(data.length-@iv_random_length..data.length)
data = data.byteslice(@iv_random_length..data.length-auth_tag_length-1)
else
data = data.byteslice(@iv_random_length..data.length)
end
end
if @mode == "encrypt"
@random_iv = OpenSSL::Random.random_bytes(@iv_random_length)
end
@cipher.iv = @random_iv
result = @cipher.update(data)[email protected]
if @mode == "encrypt"
# if we have a random_iv, prepend that to the crypted result
if !@random_iv.nil?
result = @random_iv + result
end
if @algorithm.downcase.include? "gcm" or @algorithm.downcase.include? "ccm"
result = result + @cipher.auth_tag
@logger.debug("Cipher auth_tag length: ", :auth_tag_length => @cipher.auth_tag.length)
end
result = Base64.strict_encode64(result).encode("utf-8") if @base64 == true
end
@logger.debug("Cipher algorithm: ", :algorithm => @algorithm)
rescue => e
@logger.warn("Exception catch on cipher filter", :event => event, :error => e)
# force a re-initialize on error to be safe
init_cipher
else
@total_cipher_uses += 1
result = result.force_encoding("utf-8") if @mode == "decrypt"
event.set(@target, result)
#Is it necessary to add 'if !result.nil?' ? exception have been already catched.
#In doubt, I keep it.
filter_matched(event) if !result.nil?
if !@max_cipher_reuse.nil? and @total_cipher_uses >= @max_cipher_reuse
@logger.debug("max_cipher_reuse["+@max_cipher_reuse.to_s+"] reached, total_cipher_uses = "+@total_cipher_uses.to_s)
init_cipher
end
end
end # def filter
def init_cipher
if [email protected]?
@cipher.reset
@cipher = nil
end
@cipher = OpenSSL::Cipher.new(@algorithm)
@total_cipher_uses = 0
if @mode == "encrypt"
@cipher.encrypt
elsif @mode == "decrypt"
@cipher.decrypt
else
@logger.error("Invalid cipher mode. Valid values are \"encrypt\" or \"decrypt\"", :mode => @mode)
raise "Bad configuration, aborting."
end
if @key.length != @key_size
@logger.debug("key length is " + @key.length.to_s + ", padding it to " + @key_size.to_s + " with '" + @key_pad.to_s + "'")
@key = @key[0,@key_size].ljust(@key_size,@key_pad)
end
@cipher.key = @key
@cipher.padding = @cipher_padding if @cipher_padding
if @algorithm.downcase.include? "gcm"
@cipher.auth_data = @auth_data
@logger.debug("Cipher initialisation auth_data: ", :auth_data => @auth_data)
end
@logger.debug("Cipher initialisation done", :mode => @mode, :key => @key, :iv_random_length => @iv_random_length, :iv_random => @iv_random, :cipher_padding => @cipher_padding)
end # def init_cipher
end # class LogStash::Filters::Cipher
4. AES-GCM加密文件、解密文件配置：
input { stdin { } }
filter {
cipher {
mode => "encrypt"
algorithm => "aes-128-gcm"
key_size => 16
iv_random_length => 16
cipher_padding => 0
key => "1234567890ABCDEF0"
source => "message"
target => "ciphertext"
enable_metric => false
remove_field => [ "message" ]
}
}
output { stdout {} }
input { stdin { } }
filter {
cipher {
mode => "decrypt"
algorithm => "aes-128-gcm"
key_size => 16
iv_random_length => 16
cipher_padding => 0
key => "1234567890ABCDEF0"
source => "message"
target => "plaintext"
enable_metric => false
remove_field => [ "message" ]
}
}
output { stdout {} }
5. 加密解密验证：
a. 加密前输入:
abcdefghijklmnopqrst加密后输出：
{
"ciphertext" => "6wu/hk+aKC4KC4E5ESwCxaVowyOlHvMEisxNYjhEZzZ2lDYaMArkn4zZur9t1wYczA/sdw==",
"host" => "px-VirtualBox",
"@version" => "1",
"@timestamp" => 2020-01-05T11:45:51.401Z
}
b. 解密前输入：
6wu/hk+aKC4KC4E5ESwCxaVowyOlHvMEisxNYjhEZzZ2lDYaMArkn4zZur9t1wYczA/sdw==解密后输出：
{
"host" => "px-VirtualBox",
"plaintext" => "abcdefghijklmnopqrst",
"@timestamp" => 2020-01-05T12:02:09.492Z,
"@version" => "1"
}
6. 源码改进参见：https://github.com/AndrewPanB/logstash-filter-cipher