Vector v0.34.0 release notes

The Vector team is pleased to announce version 0.34.0!

Be sure to check out the upgrade guide for breaking changes in this release.

In addition to the usual enhancements and bug fixes, this release also includes

  • A new protobuf encoder for sinks.
  • A fix to pass event metadata and secrets through disk buffers to have this data available in sinks. Note this change has potential security implications as things like Datadog API keys or Splunk HEC tokens could end up persisted in disk buffers. See the release highlight for details about this change and recommended practices to either secure the disk buffers or to avoid storing secrets in events altogether.

This release also marks the deprecation of the OS package repositories hosted at repositories.timber.io. Instead, packages have been moved to apt.vector.dev and yum.vector.dev. Please see the release highlight for details about this change and instructions on how to migrate. The repositories located at repositories.timber.io will be decommissioned on February 28th, 2024.

Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

Known issues

  • The Datadog Metrics sink fails to send a large number of requests due to incorrectly sized batches #19110. This is fixed in v0.34.1.
  • The Loki sink incorrectly sets the Content-Encoding header on requests to application/json when the default snappy compression is used. This results in Loki rejecting the requests with an HTTP 400 response. This is fixed in v0.34.1.
  • The protobuf encoder does not work in sinks #19230. Fixed in v0.34.2.

Changelog

7 enhancements

  • Sinks now have additional options for encoding.timestamp_format:

    • unix_float: Represents the timestamp as a Unix timestamp in floating point.
    • unix_ms: Represents the timestamp as a Unix timestamp in milliseconds.
    • unix_ns: Represents the timestamp as a Unix timestamp in nanoseconds.
    • unix_us: Represents the timestamp as a Unix timestamp in microseconds
    Thanks to srstrickland for contributing this change!
  • The prometheus_remote_write sink now has the ability to disable aggregation by setting batch.aggregate to false.
  • The nats source has a new subscriber_capacity configuration option to control how many messages the NATS subscriber buffers before incoming messages are dropped.

  • Sources and sinks that run a HTTP server now emit additional internal metrics:

    • http_server_requests_received_total
    • http_server_responses_sent_total
    • http_server_handler_duration_seconds

  • Sources that run a gRPC server now emit additional internal metrics:

    • grpc_server_messages_received_total
    • grpc_server_messages_sent_total
    • grpc_server_handler_duration_seconds

  • Event metadata, including secrets like Datadog API key or Splunk HEC token, are now persisted when writing events to a disk buffer so that sinks have access to it.

    As part of this change, disk buffers created by Vector now have more restrictive file permissions on Unix platforms. Previously, they were world-readable, but are now only writable by the Vector process user (typically vector) and readable by group.

    See the release highlight for details about this change and recommended practices to either secure the disk buffers or to avoid storing secrets in events altogether.

  • The journald source has a new emit_cursor option that, when enabled, adds the __CURSOR field to emitted log records. Thanks to sproberts92 for contributing this change!

3 new features

  • Sinks can now encode data as protobuf through support for a new protobuf encoder (configurable using encoding.codec). Thanks to goakley for contributing this change!

  • Vector’s version of VRL was updated to 0.8.1, with the following changes:

    • Added the contains_all function
    • from_unix_timestamp now accepts a new unit, microseconds
    • parse_nginx_log no longer fails if upstream_response_length, upstream_response_time, and upstream_status are missing
    • Added the parse_float function
    • Improved fallibility diagnostics
  • Vector now has the ability to start with an empty configuration when using --allow-empty-config. This is useful if you want to start Vector before loading a configuration using --watch-config or when reloading.

11 bug fixes

  • The docker_logs source no longer increments component_errors_total for out-of-order logs since this is not an error.
  • The native_json codec no longer errors when encoding 64-bit float values that represent infinity. Instead, these are encoded as the strings inf and -inf. This most commonly occurred when encoding histograms where the last bucket bound included infinity.
  • The http_server source no longer panics when handling metrics decoded using the native or native_json codecs. This means that it can now be used in conjunction with the http sink to send data between Vector instances using the native or native_json codecs; however, the vector source/sink pair is still the preferred route for Vector-to-Vector communication.
  • The kafka source now fully drains acknowledgements during consumer group rebalancing and when Vector is shutting down. This avoids situations where Vector would duplicate message processing. Thanks to jches for contributing this change!
  • The gcp_stackdriver_metrics sink now correctly handles configured batch sizes greater than the default of 1. Previously it would only send the last event in each batch.
  • Sources that receive incoming TLS traffic now correctly apply any configured tls.alpn_protocols options. Previously, these were only applied for sources creating outgoing TLS connections. Thanks to anil-db for contributing this change!
  • Sources now correctly emit a log and increment component_discarded_events_total when incoming requests are cancelled before the events are pushed to downstream components.
  • The amqp sink no longer panics when the channel is in an error state. Instead, Vector now emits an error event when this occurs.
  • The datadog_agent source now records the “interval” on any incoming metrics that have it set rather than just rate. This is useful as metrics can be interpreted as rates later when viewing the data in Datadog, where the interval field will be used.

  • The blackhole sink no longer reports events processed by default. Instead, this behavior can be opted into by setting print_interval_secs to a non-zero number.

    See the upgrade guide for more details.

  • A bug in the clickhouse sink heath check where it would add an extra / to the URI, resulting in failures, was fixed. This was a regression in v0.33.0.

Download Version 0.34.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...