Vector v0.15.0 release notes

The Vector team is pleased to announce version 0.15.0!

This release includes a number of new components for collecting and sending data using Vector:

It also contains a number of additional enhancements and bug fixes. Check out the highlights and changelog for more details.

Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

Mapping one log event into multiple

Author photo for Jesse Szwedko
Jesse Szwedko
type: new feature domain: vrl domain: remap transform

Changelog

52 enhancements

kubernetes_logs source Changed default for glob_minimum_cooldown_ms to 1s 7693 remap Add match_array function 7752 aws_s3 source Add concurrency support 7712 helm platform Kubernetes recommended labels and additional labels 7687 splunk_hec source Add x-forwarded-for header support 7575 vector source vector sink Batch events per request for v2 7706 dedupe transform Remove warn level internal_event 7812 tests Create affinity pod in separate namespace for k8s tests 7831 clickhouse sink Allow enabling input_format_skip_unknown_fields 7847 dev K8s e2e tests script minor improvements 7853 kafka sink Add acknowledgements support 7677 remap Add assert_eq! function 7866 config Allow wildcards anywhere in input identifiers 7873 helm platform Add metricRelabelings for Prometheus PodMonitor 7914 http source Add binary encoding support for http source 7859 file source Add support for acknowledgements 7816 observability Add flag to send tracing spans to Datadog APM 7929 helm platform Configure a HAProxy for load balancing the aggregator 7782 aws_s3 source Send content_md5 when writing objects to work with S3 object locking 7936 remap Add standalone key support for parse_key_value 7907 vrl Add decode_percent and encode_percent functions 7937 vrl Add encode_key_value function 7751 vrl Add support standalone for standalone key encoding in encode_key_value 7951 transforms Add new filter, datadog_search, condition type 7837 file source Add fingerprint.lines option to use multiple lines to calculate fingeprints 7940 vrl Accept standalone key in parse_logfmt + minor related fix 7977 kafka source Add support for acknowledgements 7787 helm platform Add option to deploy Topology-aware service 7950 aws_s3 source Batch SQS deletes to improve throughput 7992 docker_logs source Re-add named pipe support on Windows 7996 kafka sink Disable sending key when no key_field configured 7860 observability Emit component utilization as a metric 8028 security Harden SystemD service 6696 helm platform Option to add additional ports to the service in vector-agent helm chart 8021 helm platform Add maxUnavailable for updateStrategy in vector-agent helm chart 8035 vrl Add ip_aton and ip_ntoa functions 8042 releasing Refine Vector user for debian 8064 vrl Add parse_xml function 8046 vrl Add format_int and parse_int functions 8078 kubernetes_logs source Expose lines option 8091 gcp_stackdriver_logs sink Allow template syntax in labels of monitored resource descriptor 8131 loki sink Allow label keys to be templated 8118 gcp_stackdriver_logs sink Allow template syntax in log_id 8173 aws_ec2_metadata transform Add ec2 metadata to metric event instances 8157 aws_s3 source Improve performance of S3 source 8215 vrl parse_key_value fixes 8106 vrl Add unnest function 7404 search Add page title to Algolia search index 8241 external docs Add Vector config examples to main page of vector.dev 8243 elasticsearch sink Add support for acknowledgements 8037 helm platform Introduce customConfig key 8168 internal_metrics source Add host tag to internal_metrics 8164

21 new features

remap Add parse_ruby_hash function 7688 remap Allow strings to be compared 7704 new source Initial dnstap source 6681 new source New fluent source 7548 new sink New datadog_events sink 7678 cli Add support for --config-dir to read configuration from directories 7767 elasticsearch sink Add support for tempalting the bulk action 7805 elasticsearch sink Add first-class support for data streams 7774 new source Add logstash source 7857 elasticsearch sink Reintroduce data stream support 7903 new sink Add datadog_events sink 7678 new sink Add datadog_events sink 7678 new sink New azure_blob sink 6861 new transform Aggregate transform to reduce metric volume maintaining data 7846 elasticsearch sink Allow metrics to be sent 7932 new source Add eventstoredb_metrics source 6663 new sink Add datadog_events sink 7678 new sink New redis sink 7078 cli Add graph subcommand for generating graph in DOT format 8116 vrl Use timezone from configuration when parsing timestamps without a timezone 8089 new sink Add datadog_events sink 7678

30 bug fixes

codecs Ensure that flat dotted paths are removed by only_fields 7588 remap Fix type handling for and/or expressions that can error 7848 remap Parse nginx logs with and without referrer 7852 new_relic_logs sink Default encoding field to json 7870 sample transform Accept remap conditions 7864 remap handle starting message from nginx 7930 aws_s3 source Fix missing labels for s3 source metrics 7967 observability Add missing tracing context on HTTP client metrics 8007 processing Manage timeouts in multiline better 7944 prometheus_scrape source Don’t crash on out-of-order histograms 8013 remap Fix parse_apache_log to handle lack of thread id 8053 host_metrics source init_roots on build to only output warnings when host_metrics is used 8054 loki sink Avoid flattening objects 8044 external docs Remove job and instance tags from internal_metrics 8076 kafka source add tokio feature to rdkafka to improve performance 8092 dnstap source Add package ‘apt-utils’ in docker image for dnstap integration test 8077 aws service Allow zero size compressed messages 8069 vrl Ensure parse_syslog handles non structured messages 8102 vrl Replace referrer by referer on nginx parser 8086 unit tests check_fields deprecation notice is incorrect 8132 vrl to_int now truncates floats 8145 journald source Fix event metrics reporting 8163 external docs Fix broken README image 8213 metrics Deny adding aggregated summaries 8096 internal_metrics source Add histogram bucket for 64 8161 sinks Fix broken handling of adaptive_concurrency defaults 8189 file source Handle legacy fingerprint checksums from < v0.14.0 8225 file source Write checkpoints before shutdown 8252 observability Fix span labels for http-based sources 8266 elasticsearch sink Fix data stream timestamp handling 8271

What’s next

Enabling Adaptive Concurrency Control by default
We released Adaptive Concurrency Control in version 0.11.0 of Vector, but, up until now, this feature has been opt-in. We’ve been collecting user feedback, making enhancements, and expect to enable this feature as the default in 0.16.0. Users will still be able to configure static concurrency controls as they do now.
End to end acknowledgements

We’ve heard from a number of users that they’d like improved delivery guarantees for events flowing through Vector. We are working on a feature to allow, for components that are able to support it, to only acknowledging data flowing into source components after that data has been sent by any associated sinks. For example, this would avoid acknowledging messages in Kafka until the data in those messages has been sent via all associated sinks.

This release includes support in additional source and sink components that support acknowledgements, but it has not yet been fully documented and tested. We expect to officially release this with 0.16.0.

Kubernetes aggregator role

We are hard at work at expanding the ability to run Vector as an aggregator in Kubernetes. This will allow you to build end-to-end observability pipelines in Kubernetes with Vector. Distributing processing on the edge, centralizing it with an aggregator, or both. If you are interested in beta testing, please join our chat and let us know.

We do expect this to be released with 0.16.0.

Download Version 0.15.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...