Vector v0.13.0 release notes

The Vector team is pleased to announce version 0.13.0!

This release brings a new datadog_logs source to accept logs from dd-agents, a new gcp_stackdriver_metrics sink to send metrics to GCP Stackdriver, and a new subcommand, vector tap that allows for inspecting events flowing out of a component. It also brings a number of smaller enhancements, particularly around the Vector Remap Language used by our remap, filter, and route transforms.

Check out the highlights and changelog for more details.

Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

Introducing vector tap

Author photo for Jesse Szwedko
Jesse Szwedko
type: new feature domain: graphql domain: cli domain: observability

Changelog

14 enhancements

ci Add new VRL comparison benchmark 6387 observability Add events_in_total internal metric to sources 6758 releasing Correct the glibc requirements for packages 6774 cli Support VECTOR_LOG_FORMAT and VECTOR_COLOR environment variables 6930 remap Support newlines in if-statement 6893 kafka source Add defaults for metadata keys so that they are added by default 6928 route transform route transform can now process metrics 6927 remap introduce is_TYPE helper functions such as is_string 6775 graphql api Additions to GraphQL API to allow tapping the events flowing out of a component 6610 graphql api Add json to Log type to expose the log event as JSON 6915 kafka source Kafka message headers are now added as fields on the event 7030 remap parse_regex now suppresses the numeric capture groups by default but they can be re-added via numeric_groups: true in the function call 7069 splunk_hec source Support reading the channel from the channel query parameter rather than just the X-Splunk-Request-Channel header 7067 remap Support identifiers with leading numeric characters as fields in paths 7045

10 new features

new source New datadog_logs source 6744 remap Add abort statement to halt processing 6723 remap Add parse_query_string function 6796 remap New parse_linux_authorization function 6883 remap Support dropping events when remap fails or is aborted via new abort statement 6722 remap Add parse_csv function 6853 new sink Add gcp_stackdriver_metrics sink. 6666 observability Vector tap subcommand to view events flowing out of a component 6871 remap Add parse_nginx_log function for parsing common nginx log formats 6973 remap Add to_regex function to convert strings to regexes 7074

32 bug fixes

tests Use next_addr instead of fixed addresses for prometheus_exporter tests 6766 internal docs Use regular markdown image syntax in reference section 6835 ci Fix workflow for packaging for Debian 6824 tests Explicitly mount docker socket for AWS ECS metadata mock container 6833 external docs --no-topology has been removed from vector validate 6851 syslog source No longer hang indefinitely on invalid TCP data 6864 internal docs Use regular markdown image syntax in reference section 6835 remap Allow newlines in REPL 6894 remap Allow non-matching brackets in REPL; would previously hang 6896 exceptions Typo on the expectation for building regexps 6951 ci Fix release-homebrew workflow dependency 6945 releasing Fix package naming for latest artifacts 6947 tests Export default container tool setting 6954 deps Upgrade reqwest in k8s-e2e-tests 6990 tests Retry errors in Kafka integration tests 6992 host_metrics source Fix filtering of filesystem metrics by device 6921 releasing Ensure /etc/default/vector is not automatically overwritten 7007 tests Fix formatting of timestamps in tests 7036 tests Fix aws_ec2_metadata transform test 7062 remap Prevent lexer getting stuck on an unterminated literal which would cause VRL to hang 7037 ci Update homebrew when bootstrapping OSX 7095 datadog service Document existence of region parameter 7093 socket source Remove socket file on source shutdown which would otherwise prevent Vector from restarting 7047 clickhouse sink Clickhouse sink now correctly encodes fields containing an array 7081 docker_logs source Can now read logs from containers that have an attached TTY 7111 ci Ensure all benchmark artifacts are included 7113 remap Preserve type defs when assigning fields to avoid unnecessary type assertions and confusing error messages 7118 kubernetes_logs source Actually use the checkpoint files to avoid re-ingesting logs when Vector is restarted 7140 remap Defaults for case_sensitive parameter for string matching functions (like starts_with) were updated to default to case sensitive matching to match docs 7091 compression Fix compression handling for aws_s3, aws_kinesis_firehose, splunk_hec, and http sources to handle multi-part files 7138 docker_logs source Ensure docker labels are not injected as nested objects if they contain a . 7152 kubernetes_logs source Refactor stream and invocation errors to support recoverable error types 6816

What’s next

End-to-end acknowledgements
We’ve heard from a number of users that they’d like improved delivery guarantees for events flowing through Vector. We are working on a feature to allow, for components that are able to support it, to only acknowledging data flowing into source components after that data has been sent by any associated sinks. For example, this would avoid acknowledging messages in Kafka until the data in those messages has been sent via all associated sinks.
Mapping and iteration in VRL

Shortly you will be able to iterate over data in VRL to:

  • allow processing of fields containing arrays of unknown length
  • map over keys and values in objects to transform them
Kubernetes aggregator role

We are hard at work at expanding the ability to run Vector as an aggregator in Kubernetes. This will allow you to build end-to-end observability pipelines in Kubernetes with Vector. Distributing processing on the edge, centralizing it with an aggregator, or both. If you are interested in beta testing, please join our chat and let us know.

This was mentioned in the release notes for 0.11.0, but didn’t quite make it for 0.12 as anticipated. We do expect it to be released in the next few months.

Download Version 0.13.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...