rsyslog changelog
Scheduled Release 8.2002.0 (aka 2020.02) 2020-02-25
- 2020-02-25: imfile: add per minute rate limiting
Add MaxBytesPerMinute and MaxLinesPerMinute options.
These take integer values and, respectively, limit the number
of bytes or lines that may be sent in a minute.
This can be used to put a limit on the count or volume of logs
that may be sent for an imfile.
Thanks to Greg Farrell for the patch. - 2020-02-24: core: add global parameter “security.abortOnIDResolutionFail”
This parameter controls whether or not rsyslog aborts when a name ID
lookup fails (for user and group names). This is necessary as a security
measure, as otherwise the wrong permissions can be assigned or privileges
are not dropped.
CHANGE OF BEHAVIOR
The default for this parameter is “on”. In previous versions, the default
was “off” (by virtue of this parameter not existing). As such, existing
configurations may now error out.
We have decided to accept this change of behavior because of the potential
security implications.
closes https://github.com/rsyslog/rsyslog/issues/4164 - 2020-02-24: openssl TLS driver bugfix: chained certificates were not accepted
This was supported since always inside GnuTLS driver, but was missing for openssl one. - 2020-02-24: core bugfix: too early parsing of incoming messages
In theory, rsyslog should call parsers on the queue worker threads whenever
possible. This enables the parsers to be executed in parallel. There are
some cases where parsers needs to be called earlier, namely when parsed
data is needed for rate-limiting.
The logic to do this previously did not work correctly and was fixed six
years ago (!) by b51dd22. Unfortunately, b51dd22 was overly agressive:
it actually makes the early parser call now mandatory, effectively moving
parsing to the input side where there is no to little concurrency.
We still do not need to call the parser when all messages, regardless of
severity, need to be rate-limited. This is the default and very frequent
case. This patch introduces support for this and as such makes parsers
able to run in parallel in the frequent case again.
closes https://github.com/rsyslog/rsyslog/issues/4187 - 2020-02-20: testbench bugfix: two minor issues in omkafkadynakey.sh test
lead to false positives during test runs (depending on circumstances)
closes: https://github.com/rsyslog/rsyslog/issues/4134 - 2020-02-20: testbench: set max extra data length for tcpflood from 200 to 512KiB
Added a imrelp test for big messages (256KB).
closes: https://github.com/rsyslog/rsyslog/issues/4158 - 2020-02-20: config system bugfix: ‘config.enabled’ directive oddities
Previously the directive was processed way too late which caused false
errors whenever it was set to ‘off’ and possibly other problems.
Thanks to Jiri Vymazal for the patch. - 2020-02-09: 修复超时功能在高负载系统上不起作用的问题
imfile bugfix: timeout did not work on very busy system
The timeout feature was soley based on timeouts of the poll()
system call. On a very busy system, this would probably happen
very seldomly. Moreover, the timeout could occur later than
expected on any system with high load.
The issue was not reported from practice but discovered during
CI system improvements. - 2020-01-30: build system: change --enable-imfile-tests default to “yes”
This was accidentally set to “no” some time ago (actual commit unknown). Tests for
imfile should by default run when imfile is enabled.
see also https://github.com/rsyslog/rsyslog/issues/4120 - 2020-01-27: build system: add option --enable-gnutls-tests
This enables us to build GNUtls support but not necessarily
test it in CI. This is useful for some specialised subcomponent
test. The default is enabled if gnutls is enabled and disabled if not. - 2020-01-26: testbench: new test for loadbalancing via global vars
This is a popular functionality which had not been routinely tested
in the past. - 2020-01-26: mmdblookup bugfix: invalid data returned when no entry found
Since the upgrade of the package libmaxminddb on FreeBSD (1.3.2_2 -> 1.4.2),
the module mmdblookup returns the first entry of the mmdb database even if the entry
is not found. After some debug, I found the solution in the official maxminddb
repository : to check if the entry is in database, we must check the found_entry
attribute, otherwise the function MMDB_get_entry_data_list will return the first
entry of the database if the entry is not found in it.
Thanks to Kevin Guillemot for the patch. - 2020-01-23: oversize message log bugfix: do not close fd -1
The oversize message log fd is always closed on HUP, even if it never
was opened (and thus has -1 value). This patch corrects the issue.
The bug had no know-bad effect in practice other than getting an
(ignored) error status from close(). However, it introduced warnings
in test runs (e.g. when running under valgrind). - 2020-01-22: imfile bugfix: saving of old file_id for statefiles
Previously we saved old file_id unconditionally, which led to not
deleting old statefiles if files changes without rsyslog running.
Now it should work correctly.
Thanks to Jiri Vymazal for the patch. - 2020-01-22: imfile bugfix: 修复潜在的内存覆盖的问题
misadressing and potential segfault
Commit 3f72e8c introduced an invalid memory allocation size. This lead to
too-short alloc and thus to overwrite of non-owned memory. That in turn
could lead to segfaults or other hard to find problems.
The issue was detected by our upgraded CI system. We did not receive
any problem reports in practice. Nevertheless, the problem is real and
people should update affected versions to patched ones.
The bug was present in scheduled stable release 8.1911.0 and 8.2001.0.
see also: https://github.com/rsyslog/rsyslog/issues/4120
see also: https://github.com/rsyslog/rsyslog/pull/4141 - 2020-01-20: core bugfix: 修复“在启动后立即将rsyslog进行HUP并完全初始化之前，如果在HUP期间发生了潜在的竞争，则与已加载模块的列表可能会发生竞争。”的问题
potential race during HUP
when rsyslog is HUPed immediately after startup and before it is fully
initialized, there is a potential race with the list of loaded modules.
This patch ensures no bad things can happen in that case.
Detected by LLVM TSAN, not seen in practice. - 2020-01-20: testbench improvements and fixes
modernize tests, reduce robustness against slow machines, provide some
test framework functional enhancements, and optimize some tests.
Also includes some code changes to C testing components. Among others,
tests have slightly been speeded up by reducing the wait time at queue
shutdown. This is possible because of better overall completion checks.
Scheduled Release 8.2001.0 (aka 2020.01) 2020-01-14
- 2020-01-12: core bugfix: 解决“在写入磁盘队列时libfastjson发生竞争条件”的问题。
race condition related to libfastjson when using DA queue
Rsyslogd aborts when writing to disk queue from multiple workers simultaneously.
It is assumed that libfastjson is not thread-safe.
Resolve libfastjson race condition when writing to disk queue.
see also https://github.com/rsyslog/rsyslog/issues/4041
Thanks to MIZUTA Takeshi for the fix. - 2020-01-12: omfwd bugfix: parameter streamdriver.permitexpiredcerts did not work
closes https://github.com/rsyslog/rsyslog/issues/4098 - 2020-01-11: Bugfix: KSI module + dynafile in asynchronous mode fixed
Thanks to Taavi Valjaots for the patch - 2020-01-08: tls driver: 加入streamdriver.TlsVerifyDepth参数以设置配置证书验证深度
add support to configure certificate verify depth
Support added in omfwd as instance parameter:
streamdriver.TlsVerifyDepth
Support added in imtcp as module parameter:
streamdriver.TlsVerifyDepth
Can be 2 or higher.
Support added into ossl driver
Support added into gtls driver
Added testcases for both drivers.
closes: https://github.com/rsyslog/rsyslog/issues/4035 - 2020-01-08: modernization of testbench
moved some tests to newer standards, hardened them against slow testbench machines,
kafka component download improvements, and prevent dangling left-over test tool
instances from aborted tests - 2020-01-07: tls subsystem bugfix: default for permitExpiredCerts was invalidly “on”
The problem occured with commit 3d9b8df in December 2018 and went into
scheduled stable 8.1901.0. Unfortunately, the change in default was not detected
until a year later. This commit re-enables the previous default (“off”), which is
also the only sensible default from a security PoV. Unfortunately, new 2019
deployments may begin to see connection rejection when usin expired certs. As
expired certs should not be used, this hopefully will not cause problems in
practice.
Thanks to Jiri Vymazal for the patch. - 2020-01-01: testbench: improve ElasticSearch test speed
We now support re-using suitable running ES instances, which reduces the
number of restarts. - 2019-12-31: omelasticsearch: improve curl reply buffer handling
The curl reply buffer (pWrkrData->reply) was allocated, realloced and freed with
each request. This has now been reduced to once per module, slightly increasing
overall performance.
closes https://github.com/rsyslog/rsyslog/issues/1964 - 2019-12-31: config system: emit proper error message on $ in double-quoted string
closes https://github.com/rsyslog/rsyslog/issues/2869 - 2019-12-30: core bugfix: rsyslog aborts when config parse error is detected
In defaut settings, rsyslog tries to continue to run, but some data
structures are not properly initialized due to the config parsing error.
This causes a segfault.
In the following tracker, this is the root cause of the abort:
see also https://github.com/rsyslog/rsyslog/issues/2869 - 2019-12-30: fix some alignment issues
So far, this worked everywhere (for years). But it may still have
caused issues on some platforms.
closes https://github.com/rsyslog/rsyslog/issues/2608 - 2019-12-27: core bugfix: APP-NAME fields could become empty
RFC 5424 specifies that an empty APP-NAME needs to be indicated by
“-”. Instead, the field could become empty under certain conditions.
If so, outgoing 5424 messages were invalidly formatted.
This happened under quite unusual conditions, but could be seen
in practice.
closes https://github.com/rsyslog/rsyslog/issues/4043 - 2019-12-27: core bugfix: reopen /dev/urandom file descriptor after fork on Linux
This patch updates prepareBackground() in tools/rsyslogd.c to reopen any file
descriptors used for random number generation in the child process. This fixes
an issue on Linux systems where the file descriptor obtained for /dev/urandom
by seedRandomNumber() in runtime/srutils.c was left closed after the fork. This
could be observed in procfs, where /proc/fd/ would show no open descriptors to
/dev/urandom in the forked process. /dev/urandom is reopened as the child may be
be operating in a jail, and so should not continue to use file descriptors from
outside the jail (i.e. inherited from the parent process).
I found that this issue led to rsyslog intermittently hanging during seedIV()
in runtime/libgcry.c. After the fork, the closed file descriptor number tended
to get re-assigned. randomNumber() would then read from an incorrect (although
still valid) file descriptor, and could block (depending on the state of that
file descriptor). This gave rise to the intermittent hang that I observed.
Thanks to Simon Haggett for the patch. - 2019-12-20: imdocker bugfix: did not compile without atomic operations
- 2019-12-20: omclickhouse: new parameter “timeout”
Thanks to Pavlo Bashynskiy for the patch. - 2019-12-20: omhiredis: add ‘set’ mode plus some fixes
- new mode ‘set’ to send SET/SETEX commands
- new parameter ‘expiration’ to send SETEX instead of SET commands (only applicable to ‘set’ mode)
- fixes to missing frees
Thanks to github user frikilax for the patch.
- 2019-12-18: relp: Add support setting openssl configuration commands.
Add new configuration parameter tls.tlscfgcmd to omrelp and imrelp.
(Using relpSrvSetTlsConfigCmd and relpCltSetTlsConfigCmd)
OpenSSL Version 1.0.2 or higher is required for this feature.
A list of possible commands and their valid values can be found in the
documentation: https://www.openssl.org/docs/man1.0.2/man3/SSL_CONF_cmd.html
The setting can be single or multiline, each configuration command is
separated by linefeed (n). Command and value are separated by
equal sign (=). Here are a few samples:
tls.tlscfgcmd=“Protocol=ALL,-SSLv2,-SSLv3,-TLSv1,-TLSv1.2”
tls.tlscfgcmd=“Protocol=ALL,-SSLv2,-SSLv3,-TLSv1
MinProtocol=TLSv1.2”
Add to new testcases for librelp and tlscfgcmd.
closes https://github.com/rsyslog/rsyslog/issues/3959 - 2019-12-18: bugfix core: potential segfault in template engine
under some circumstances (not entirely clear right now), memory
was freed but later re-used as state-tracking structures were not
properly maintained. Github issue mentioned below has full details.
Thanks to github user snaix for analyzing this issue and providing
a patch. I am committing as myself as snaix did not disclose his or
her identity.
closes https://github.com/rsyslog/rsyslog/issues/3019
closes https://github.com/rsyslog/rsyslog/issues/4040 - 2019-12-18: fixed some minor issues detected by clang static analyzer 9
- 2019-12-10: core/config bugfix: false error msg when config.enabled=“on” is used
When the ‘config.enabled=“on”’ config parameter an invalid error message
was emitted that this parameter is not supported. However, it was still
applied properly. This commit removes the invalid error message.
closes https://github.com/rsyslog/rsyslog/issues/4011 - 2019-12-03: omsnmp bugfix: “traptype” parameter invalidly rejected value 6
“Traptype” needs to support values 0 to 6.
However, if value 6(ENTERPRISESPECIFIC) was set, an invalid error message
was emitted. Otherwise processing was correct.
This could lead to problems with automatic config deployment,
as valid configurations were invalidly reported as incorrect.
That in turn could make a deployment fail.
closes https://github.com/rsyslog/rsyslog/issues/3973 - 2019-12-03: omsnmp: add new parameter “snmpv1dynsource”
If set, the source field from SNMPv1 trap can be overwritten
with a template, default is “%fromhost-ip%”. The content should be a
valid IPv4 Address that can be passed to inet_addr(). If the content
is not a valid IPv4 Address, the source will not be set.
closes: https://github.com/rsyslog/rsyslog/issues/3991 - 2019-12-02: imfile bugfix: state file renaming sometimes did not work properly
Now checking if file-id changes and renaming - cleaning state file
accordingly and always checking and cleaning old inode-only style
state files.
Thanks to Jiri Vymazal for the patch. - 2019-12-02: ratelimit: increase rate limit interval parameter max value
The burst parameter in the ratelimit was increased to an unsigned int
but the interval remained an unsigned short. While it may be unusual,
there is possibly a chance to need to represent an interval longer than
about 3/4 of a day.
While here, go through and normalize all the various incarnations of
rate limiting to be explicitly unsigned int for the burst and interval.
Thanks to github user frikilax for the patch. - 2019-12-02: ommongodb: Add other supported formats for ‘time’ and ‘date’ fields
Thanks to github user frikilax for the patch. - 2019-12-02: imjournal bugfix: too many messages in error case
Under certain error conditions,ignorePreviousMessages="on"could be ignored
an existing messages be processed.
Thanks to github user 3chas3 for the patch. - 2019-11-27: core bugfix: action on retry mangles messages
When a failed action goes into retry, template content is rendered
invalid if the action uses more than 1 template.
closes https://github.com/rsyslog/rsyslog/issues/3898
Thanks to Mikko Kortelainen for the patch. - 2019-11-27: testbench: improve mysql testing support
tests can now run in parallel and are hardened against several glitches - 2019-11-22: omhttp: add basic support for Loki Rest
Loki is a new message indexer and querier from Grafana Labs. See
https://github.com/grafana/loki for details on Loki.
This change provides the initial message structure to send bulk message
payloads to the Loki Rest endpoint. omhttp, received a new bulk message
format called lokirest. Additionally, the plugin relies on the user to
provide the correct “stream” read message format.
A loki template must be json compatible and include a “stream” key of
key value tags, and a values key of an array of 2 element arrays, where
each 2 element array is the unix epoch in nanoseconds followed by an
unstructured message.
An example:
template(name=“array_loki” type=“string” string="{“stream”:{“host”:"%HOSTNAME%",“facility”:"%syslogfacility-text%",“priority”:"%syslogpriority-text%",“syslogtag”:"%syslogtag%"},“values”: [[ “%timegenerated:::date-unixtimestamp%000000000”, “%msg%” ]]}") - 2019-11-22: testbench: obtain python binary path via AM_PATH_PYTHON
see also https://github.com/rsyslog/rsyslog/issues/3853 - 2019-11-22: omprog: detect violation of interface protocol
The spec for the omprog interaction with the program it calls specifies
that the program receives one message via one line. In other words:
it must be a string terminated by LF.
However, omprog does currently rely on a proper template to fulfill this
requirement, If the template does not provide for the LF, it is never
written. For the called program, this looks like it does not receive any
input at all. Even if it finally reads data (e.g. due to full buffer),
it will not properly be able to discern the messages.
This handling is improved with this commit.
We cannot just check the template, because at the end of the template
may by a non-constant value. As such, we do not know at config load
time if there is this problem or not.
So the correct approach is to, during runtime, check if each message
is properly terminated. For those that are not:- we append a LF, because anything else makes matters worse
- log a warning message, at least for a sample of the messages
The warning is useful in the (expected most often) case that the template
is simply missing the LF. While appending works, it slows down processing.
As such the user should be given a chance to correct the config bug.
To avoid clutter, the warning is emitted at most once every 30 seconds.
This value is hardcoded as we do not envision a need to adjust it. Usually
users should quickly fix the template.
closes https://github.com/rsyslog/rsyslog/issues/3975
- 2019-11-19: core queue: emit warning if parameters are set for direct queue
Direct queues do not apply queue parameters because they are actually
no physical queue. As such, any parameter set is ignored. This can
lead to unintentional results.
The new code detects this case and warns the user.
closes https://github.com/rsyslog/rsyslog/issues/77 - 2019-11-19: imjournal bugfix: do not wait too long on recovery try
When trying to recover journal errors, imjournal waited a hardcoded
period of 10s between tries. This was pretty long and could lead to
loss of journal data.
This commit adjust it to 100ms, which should still be fully sufficient
to prevent the journal from “hammering” the CPU.
It may be worth considering to make this setting configurable - but
let’s first see if there is real demand to actually do that.
closes https://github.com/rsyslog/rsyslog/issues/3969 - 2019-11-19: mmutf8fix: enhance handling of incorrect UTF-8 sequences
- Invalid utf8 detection didn’t handle 3 and 4-byte overlong encodings (2
byte overlong encodings were handled explicitly by rejection E0 and E1
start bytes). Unified checks for overlong encodings. - Surrogates U+D800…U+DFFF are not valid codepoints (Unicode Standard, D92)
- Replacement of characters in invalid 3 or 4-bytes encodings was too
eager. It must not replace bytes which are valid UTF-8 sequences. For
example, in [0xE0 0xC2 0xA7] sequence the 0xC2 is invalid as a continuation
byte, but it starts a valid UTF8 symbol [0xC2 0xA7]. That is, with current
code processing the sequence will result in “???” but the correct result is “?§”
(provided that the replacement character is “?”). - Various tests for UTF-8 invalid/valid sequences.
Thanks to Sergei Turchanov for the patch.
- Invalid utf8 detection didn’t handle 3 and 4-byte overlong encodings (2
- 2019-11-14: imfile: add new input parameter escapeLF.replacement
The new parameter permits to specify a replacement to be configured
when “escapeLF” is set to “on”. Previously, a fixed replacement string
was used ("#012"/"\n") depending on circumstances. If the parameter is
set to an empty string, the LF is simply discarded.
closes https://github.com/rsyslog/rsyslog/issues/3889
Scheduled Release 8.1911.0 (aka 2019.11) 2019-11-12
- 2019-11-12: core queue: add config param “queue.takeFlowCtlFromMsg”
This is a fine-tuning option which permits to control whether or not
rsyslog shall alays take the flow control setting from the message. If
so, non-primary queues may also block when reaching high water mark.
This permits to add some synchronous processing to rsyslog core engine.
However, it is dangerous, as improper use may make the core engine
stall. As such, enabling this option requires very careful planning
of the rsyslog configuration and deep understanding of the consequences.
Note that the option is applied to individual queues, so a configuration
with a large number of queues can (and must if use) be fine-tuned to
the exact use case.
The rsyslog team strongly recommends to let the option turned off,
which is the default setting.
see also https://github.com/rsyslog/rsyslog/issues/3941 - 2019-11-12: imrelp: add new config parameter “flowcontrol”
This permits to fine-tune the flowControl parameter. Possible values are
“no”, “light”, and “full”. With light being the default and previously
only value.
Changing the flow control setting may be useful for some rare applications,
but be sure to know exactly what you are doing when changing this setting.
Most importantly, whole rsyslog may block and become unresponsive if you
change flowcontrol to “full”. While this may be a desired effect when
intentionally trying to make it most unlikely that rsyslog needs to
lose/discard messages, usually this is not what you want.
see also https://github.com/rsyslog/rsyslog/issues/3941 - 2019-11-11: imrelp: remove unsafe debug instrumentation
dbgprintf, which is not signal safe, was called from a signal handler
to get better understanding during debugging. While this usually works,
it can occasionally (5%) lead to a hang during shutdown. We have now
removed that debug info as it is no longer vital.
Note: this could only happen during debug runs. Production mode was
not affected. As such, this fix is only relevant to developers.
However, it caused some confusion in the following issue tracker.
see also https://github.com/rsyslog/rsyslog/issues/3941 - 2019-11-06: ossl driver bugfix: fix wrong OpenSSL Version check
Fix OpenSSL Version check in:- SetGnutlsPriorityString function in nsd_ossl.c
- initTLS() function tcpflood.c
See https://www.openssl.org/docs/man1.1.0/man3/OPENSSL_VERSION_NUMBER.html
for more.
This bug lead to not enabling some functionality correctly.
Removed “MinProtocol=TLSv1.1” from two testcases because MinProtocol
is only supported by OpenSSl 1.1.0 or higher and was not really
necessary for the testcases.
closes https://github.com/rsyslog/rsyslog/issues/3939
- 2019-11-05: mmdarwin: Optimizations, new parameters, update to protocol header
- use permanent worker-dependent buffers to avoid malloc/free for each entry
- move socket structures to worker data, remove global mutex
- add log lines for parameters and general workflow
- don’t send body if empty/incomplete (see new parameters)
- don’t close/reopen socket every time -> let session open or create new every X
entry (see new parameters) - clean up code
- added ‘send_partial’, to let mmdarwin send body if not all fields were
retrieved, or not; default false = only send complete bodies - added ‘socket_max_use’ to open new session every X packet, useful for
some versions of Darwin (prior to 1.1)
default is 0 = do not open new session/keep only one - added ‘evt_id’ to the darwin header (Darwin v1+ compatibility)
Note: mmdarwin is a contributed module
Thanks to github user frikilax for the patch.
- 2019-11-01: mmkubernetes bugfix: improper use of realloc()
could cause problems under extreme memory shortage - very unlikely
credits to LGTM.COM for detecting this - 2019-10-31: imjournal: set the journal data threshold to MaxMessageSize
When data is read from the journal using sd_journal_get_data it may be
truncated to a certain threshold (64K by default).
If the rsyslog MaxMessageSize is larger than the threshold, there is a
chance rsyslog will receive incomplete messages from the journal.
Empirically, this appears to happen reliably when XZ compression is
used by journald. Systems where journald uses LZ4 compression do not
appear to suffer this issue reliably–if at all.
This change sets the threshold to the MaxMessageSize when the
journal is opened.
Thanks to Robert Winslow Dalpe for the patch. - 2019-10-30: improg bugfix: allow improg to handle multi-line inputs
miscellaneous bug fixes in improg:- properly truncate string after an input event is submitted
- set msgoffset to 0.
- tests added to check above fixes
Thanks to Nelson Yen for the fix.
- 2019-10-30: mmdblookup bugfix: missing space in city name
This fixes the issue that spaces in city names are dropped. However, the
fix is more or less a work-around. As it turns out, the libmaxminddb API
is not correctly used. In the somewhat longer term, we should fix this.
see also https://github.com/maxmind/libmaxminddb/issues/218
closes https://github.com/rsyslog/rsyslog/issues/1650 - 2019-10-30: core/queue: provide ability to run diskqueue on multiple threads
Up until this release, disk queues could only use a single thread,
what limited their performance with outputs like ElasticSearch.
Now disk queues can utilize multiple threads just like any other
queue type. Most importantly, the disk queue part of a DA queue
now inherits the max number of threads from its memory queue
counterpart.
NOTE: the new multi-threaded DA disk queue is actually a change of
behavior. We have not guarded it by a new config switch as we
assume the new behavior is most often exactly within user
expectations. In any case, we cannot see any harm from running
the disk queue on multiple threads.
see also https://github.com/rsyslog/rsyslog/issues/3543
closes https://github.com/rsyslog/rsyslog/issues/3833 - 2019-10-25: omfile bugfix: file handle leak
The stream class does not close re-opened file descriptors.
This lead to leaking file handles and ultimately to the inability
to open any files/sockets/etc as rsyslog ran out of handles.
The bug was depending on timing. This involved different OS
thread scheduler timing as well as workload. The bug was more
common under the following conditions:- async writing of files
- dynafiles
- not committing file data at end of transaction
However it could be triggered under other conditions as well.
The refactoring done in 8.1908 increased the likelihood of
experiencing this bug. But it was not a real regression, the new
code was valid, but changed the timing so that the race was more
likely.
Thanks to Michael Biebl for reporting this bug and helping to
analyze it.
closes https://github.com/rsyslog/rsyslog/issues/3885
- 2019-10-22: imfile bugfix: improper use of calloc()
could cause problems under extreme memory shortage - very unlikely
credits to LGTM.COM for detecting this - 2019-10-22: TLS driver bugfix: improper use of calloc()
can cause problems under extreme memory shortage - very unlikely
credits to LGTM.COM for detecting this - 2019-10-22: imuxsock bugfix: improper use of calloc()
can cause problems under extreme memory shortage - very unlikely
credits to LGTM.COM for detecting this - 2019-10-17: build system bugfix: incorrect default in ./configure help text
closes https://github.com/rsyslog/rsyslog/issues/3904
Thanks to Michael Biebl for pointing this out. - 2019-10-17: mmkubernetes bugfix: improper use of calloc()
can cause problems under extreme memory shortage - very unlikely
credits to LGTM.COM for detecting this - 2019-10-16: core queue bugfix: propagate batch size to DA queue
This was a long-standing bug where the DA queue always had a fixed small batch
size because the setting was not propagated from the memory queue. This also
removes a needless and counter-productive “debug aid” which seemed to be in
the code for quite some while. It did not cause harm because of the batch
size issue. - 2019-10-16: testbench: fix unreliable gzipwrite test
The test was timing-sensitive as we did not properly check all data
was output to the output file - we just relied on sleep periods.
This has been changed. Also, we made some changes to the testing
framework to fully support sequence checking of multiple ZIP files. - 2019-10-16: core queue bugfix: handle multi-queue-file delete correctly
Rsyslog may leave some dangling disk queue files under the following
conditions:- batch sizes and/or messages are large
- queue files are comparatively small
- a batch spans more than two queue files (from n to n+m with m>1)
In this case, queue files n+1 to (n+m-1) are not deleted. This can
lead to problems when the queue is re-opened again. In extreme cases
this can also lead to stalled processing when the max disk space is
used up by such left-over queue files.
Using defaults this scenario is very unlikely, but it can happen,
especially when large messages are being processed.
- 2019-10-16: imjournal: fix regression from yesterday’s patch
commit 78976a9bc059 introduced a regression that caused writing
the journal state file to fail. This happens when the state file
is given as relative file name and the working directory is also
a relative path. This situation is very uncommon. So most deployments
will never experience it. We discovered the issue during CI runs
where the trigger condition is given. Note that it also takes
multiple times of loading the journal to actually see the bug.
see also https://github.com/rsyslog/rsyslog/pull/3878 - 2019-10-15: imjournal plugin code restructuring, added remote option
Decomposed ReadJournal() a bit, also now coupling journald
variables in one struct, added few warning messages and debug
prints to help with bug hunts in future, also got rid of two
needless journald calls. WorkAroundJournalBug now deprecated.
Added option to pull journald records from outside local machine.
Thanks to Jiri Vymazal for the patch. - 2019-10-11: core bugfix: potential abort on very long action name
The action name is stored in modified form for the debug header and
some messages. If it is extremely long, a buffer can be overrun,
resulting in misaddressing and potential segfault for rsyslog. This
can also happen if the action is NOT named, but a custom path to
the output module is given and that path is very long. This triggers
the same issue because by default the module load path is included
in the action name.
This patch corrects the problem and truncates overly long names
when being used for name generation.
The problem was detected during testbench work. We did never receive
a bug report from practice. - 2019-10-10: testbench: add test for mmpstrucdata with RFC5424 escape sequences
Scheduled Release 8.1910.0 (aka 2019.10) 2019-10-01
- 2019-10-01: core bugfix: incorrect error message on duplicate module load
A Null-pointer was passed to printf instead of the module name.
On some platforms this may lead to a segfault. On most platforms
printf check’s for NULL pointers and uses the string “(null)”
instead. In any case, the module name is missing from the error message. - 2019-10-01: imczmq nitfix: potential NULL ptr in printf on out-of-memory condition
very unlikely to happen but if it does without any real issue on most platforms. - 2019-10-01: work around some compiler warning messages induced by pthreads API
- 2019-10-01: core ratelimiting: more verbose message when rate-limiting happens
When messages are rate-limited, the error message now also contains the
rate limiter setting. This enables the user to more quickly understand what
the problem is (especially if default values apply).
Thanks to Jiri Vymazal for the patch. - 2019-10-01: openssl TLS driver: do not emit unnecessary error message
On older openssl versions, an API was missing to set user-defined parameters. If we
had such an older version, rsyslog emitted an error message even if the user did
not configure such parameters. This has been corrected, so that a message is only
emitted if there really is a problem. Based on user feedback the severity has also
been downgraded to “warning”. - 2019-10-01: pmcisconames (contributed module) bugfix: potential misaddressing
- 2019-09-30: pmaixforwardedfrom (contributed module) bugfix: potential misaddressing
- 2019-09-30: pmdb2diag (contributed module) bugfix: Out of bounds issue
Add a new sanity check after determining the level len.
Thanks to Philippe Duveau for the patch.
see also: https://nvd.nist.gov/vuln/detail/CVE-2019-17040 - 2019-09-02: ability to set stricter TLS operation modes
- checking of extendedKeyUsage certificate field
- stricter checking of certificate name/addresses
Thanks to Jiri Vymazal for the patch.
- 2019-08-21: testbench: add basic test for immark
- 2019-08-20: core: do not unnecessarily set hostname on each HUP
- 2019-08-20: build system: support cross-platform build for mysql/mariadb
rsyslog fails to cross build from source, because it uses mysql_config
and mysql_config is unfixably broken for cross compilation. It would be
better to use pkg-config. The attached patch makes rsyslog try
pkg-config first and fall back to mysql_config.
Thanks to Helmut Grohne for providing a base patch. - 2019-08-20: core/tcpsrv: potential race on startup/shutdown
if the tcpsrv component is started and quickly terminated, it may hang
for a short period of time. Also a very small amount of memory is leaked
immediately before shutdown. While this leak is irrelevant in practice
(the OS clean up the process anyways), it leads to CI failures. The hang,
however, can lead to longer than expected shutdown times for rsyslog.
The problem can be experienced via imtcp, imgssapi and imdiag (users
of affected core component).
Scheduled Release 8.1908.0 (aka 2019.08) 2019-08-20
- 2019-08-19: testbench: add test for $allowedSender functionality
- 2019-08-19: testbench: harden some tests against very slow CI machines
- 2019-08-16: testbench: make most tests use a port file and assign listen port 0
This makes the test much more robust against heavily loaded test systems. - 2019-08-16: core/action: guard action.externalstate.file content against whitespace
remove trailing whitespace before checking the status string. This is
most important as a line usually ends with \n, which is considered
trailing whitespace. Accepting this increases usability. - 2019-08-16: imtcp bugfix: multiple listenerPortFile parameter did not work
… because they were treated as module-global. If we had multiple imtcp
listeners with multiple port files, only the last filename was always used.
closes https://github.com/rsyslog/rsyslog/issues/3817 - 2019-08-16: testbench: improve testbench plumbing for gzip and fail cases
We have added new capabilities to the testbench plumbing to automatically
deal with gzip-compressed files. This also permits to use the wait_seq_check
function to work for gzip tests as well. The known-timing-sensitive
gzipwr_large test now makes use of the new capabilities. This enables us
to more reliably detect when we can savely shutdown the tested instance.
This commit also adds an ability to “abort” the full testbench run on
first test failure. This is especially useful during CI. - 2019-08-13: testbench: add test for imuxsock legacy format
This was never tested. Ensures we don’t accidentally break existing
configurations. - 2019-08-13: omelasticsearch bugfix: segfault on unknown retryRuleset
omelasticsearch does some “interesting tricks” for an output module.
This causes a segfault if the retryRuleset is now known.
The action module interface currently expects that all config errors
be detected during instance creation. Instead omelasticsearch defers
the retry ruleset check to a later state. The reason is that it wants
to support the use the same rulesetname it is defined in - and this
is not yet available at action parsing.
We fix this by ensuring that any deleted instance is properly unlinked
from the instance list. One may argue the module interface should get
upgrade for such cases, but this is a longer-term approach.
closes https://github.com/rsyslog/rsyslog/pull/3796 - 2019-08-12: imptcp bugfix: port=“0” parameter did not work as expected
when multiple interfaces and/or protocols could be bound, each of
them used a different listener ports were assigned. While this is
basically correct, it makes things unusable, especially as
listenPortFileName will only contain the port number used for
the latest listener.
This patch now follows the model of nsd_ptcp.c to assign only
the first port randomly and then use that port consistently. - 2019-08-10: omelasticsearch bugfix: potential resource leak with “rebindinterval”
If the “rebindInterval” parameter was used connections could be linked. This
was especially the case with small intervals (such as “2”). This is fixed by
forcing libcurl to close the connection on rebind.
Thanks to Noriko Hosoi for providing the patch. - 2019-08-10: imjournal bugfix: state file close with fsync() was incorrect
This lead to fsync() now always applied where expected.
Thanks to Jiri Vymazal for the patch. - 2019-08-10: testbench: add addtl test for multithreading and HUP
- 2019-08-10: imptcp bugfix: received bytes counter improperly maintained
imptcp counts the number of bytes received. However, receives
happen on different worker thread. The access to the counter
was not synchronized, which can cause loss of updates. Also,
thread debuggers validly flag this as an error, which creates
problems under CI.
This commit fixes the situation via atomic operations and
falls back to mutex calls if they are not available.
Detected by LLVM thread sanitizer.
closes https://github.com/rsyslog/rsyslog/issues/3798 - 2019-08-07: testbench: add basic tests for omusrmsg
- 2019-08-05: omhttp bugfix: enable checkpath configuration parameter
omhttp, ‘checkpath’ option, was not configurable in the past.- add ‘checkpath’ to the cnfparamdescr table.
- fix issue with checkpath passing extra garbage characters in string.
- add ‘checkpath’ into unit test - omhttp-retry.sh
Thanks to Nelson Yen for the fix.
- 2019-08-05: testbench bugfix: some tests were executed when req module was missing
In actual case if --enable-impstats was not given some other tests failed. - 2019-08-03: iminternal bugfix: race on termination
This could in theory lead to loss of shutdown messages, but was mostly a
cosmetic issues. We primarily fixed it to get TSAN-clean so that we can
utilize LLVM TSAN in CI. - 2019-08-02: testbench: new test for omfile outchannel functionality
- 2019-08-02: core/janitor bugfix: properly maintain dynafile cache
When the janitor cleans out timed-out files, it does not
properly indicate the entry is gone. Especially when running
in async mode this can lead to use-after-free and thus
memory corruption or segfault.
see also https://github.com/rsyslog/rsyslog/issues/3756 - 2019-08-01: omfile bugfix: race file when async writing is enabled
This seems to be a long-standing bug, introduced around 7 years ago.
It became more visible by properly closing files during HUP, which
was done in 8.1905.0 (and was another bugfix). Note that due to this
race a memory corruption can occur under bad circumstances. As such,
this may have also caused segfaults or system hangs (mutexes could
have been affected).
closes https://github.com/rsyslog/rsyslog/issues/3772 - 2019-08-01: testbench: additional tests for HUP
- 2019-07-31: imrelp bugfix: hang after HUP
termination condition was not properly checked; this lead to
premature termination after patch 1c8712415b9 was applied.
It is open to debate if patch 1c8712415b9 changed the module
interface. Actually it looks like this was previously not
well thought out.
closes https://github.com/rsyslog/rsyslog/issues/3760 - 2019-07-24: mmdarwin: add new module
This is a contributed module. For details see doc.
Thanks to the Advens team for contributing it. - 2019-07-23 iminternal bugfix: suppress mutex double-unlock
If there is a burst of log messages during a time when rsyslog is unable
to output (either during log rotation, an out-of-space condition, or
some other similar condition), rsyslog can SEGFAULT due to a mutex
double-unlock. - 2019-07-23 imtcp: enable listenPortFileName parameter
this parameter was added, but it had no effect as it was not
passed down to the driver layer. This has been fixed. That also
now enables us to use dynamically-assigned port, which are
very useful for further testbench stabilization. Quite some
false positives occurred because the pre-selected port was
already in use again when rsyslog started. - 2019-07-19 imtcp: enable listenPortFileName parameter
this parameter was added, but it had no effect as it was not
passed down to the driver layer. This has been fixed. That also
now enables us to use dynamically-assigned port, which are
very useful for further testbench stabilization. Quite some
false positives occurred because the pre-selected port was
already in use again when rsyslog started. - 2019-07-18 core/action: no error file written if act suspended on TX commit
when an action was already disabled while the action was tried to be
committed, no error file was written. Note that this state is highly
unlikely to happen. Most probably, it can only happen if parameter
action.externalstate.file is used.