Vector v0.10.0 release notes

Code name Lamport Clock

Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

Leveraging glibc when possible

Author photo for Ana Hobden
Ana Hobden
type: performance domain: operations
New Reduce transform

Author photo for Ana Hobden
Ana Hobden
type: new feature domain: transforms
Kafka components support SASL

Author photo for Ana Hobden
Ana Hobden
type: new feature domain: sinks sink: kafka
New compression options for some sinks

Author photo for Ana Hobden
Ana Hobden
type: new feature sink: aws_kinesis_firehose sink: aws_kinesis_streams sink: aws_s3 sink: humio_logs
Now supporting the bearer auth strategy

Author photo for Ana Hobden
Ana Hobden
type: enhancement domain: sinks sink: http
RegexSet support to regex transform

Author photo for Ana Hobden
Ana Hobden
type: enhancement domain: transforms transform: regex_parser

Changelog

26 enhancements

splunk_hec sink Add sourcetype 2318 topology Shutdown Vector if all sources finish 2533 filter transform swimlanes transform Add support for negating conditions in check_fields 2514 elasticsearch sink Add explicit AWS region 2532 journald source Add support for include/exclude units 2540 regex_parser transform Add RegexSet support to regex 2493 json_parser transform log parse errors as warnings 2593 http sink add bearer auth strategy 2607 security Add support for loading multiple CAs 2616 networking Add support for inline TLS keys 2724 humio sink new relic sink Add compression to humio and new_relic 2682 elasticsearch sink add pipeline config 2734 cli Consolidate and beautify validate 2622 gcp_stackdriver_logs sink Add severity key 2732 sinks Use to_raw_value instead of to_string in JsonArrayBuffer 2860 aws_s3 sink Add content options 2848 elasticsearch sink Show client errors as warning logs 2825 elasticsearch sink add assume_role to aws auth 2895 kafka source kafka sink Add SASL to kafka 2897 sinks Batch buffer rework 2866 journald source Allow remapping priority numbers to words 2751 file source Option to remove file after some time of reaching eof 2908 sinks add compression option for aws sinks 2953 kafka source Extract timestamp from event 2956 operations Add musl and glibc support to install.sh 2969 syslog source Accept octet-counting encoded messages 2955

12 new features

new source New generator source 2286 operations Optimize CI test workflow with slim builds 2440 kafka sink Add support for templates in the topic config 2598 internal_metrics source Instrument few more components with metrics 2620 new sink Initial influxdb_logs sink implementation 2474 platforms Add dev workflow simplification rfc 2685 new transform Introduce WASM Plugins 2006 operations Introduce Dev Workflow improvements 2723 filter transform Adding list of string as filter option 2745 operations More usable event RFC 2692 filter transform Adding a condition to filter using IP CIDR 2831 new transform add reduce transform 2870

44 bug fixes

kafka sink Run entire healthcheck lazily 2501 socket sink Properly detect shutdown 2429 loki sink replace OrigID with OrgID 2536 operations Iterate over target dirs to avoid missing mapall command on macOS 2539 datadog_metrics sink Use post requests 2547 splunk_hec sink Send time in . format 2570 humio_logs sink remove superfluous warning 2583 elasticsearch sink remove id key from body 2581 journald source Fix units/include_units switch 2567 socket sink Fix possible infinite loop decoding too-long lines 2587 aws_s3 sink Allow removed keys to be used for partitioning 2591 datadog_logs sink Enable tls by default 2527 tests Fixup loki tests 2613 tests Remove race between sink and source 2619 file source Fix the bug with channel closing at file server 2652 kafka sink kafka source Fix the Kafka TLS key parameter handling 2658 topology Check if transform/sink is present 2617 sources Shutdown starting from tcp and unix sockets 2618 influxdb_logs sink Fix buffer compression 2679 kafka sink Three KafkaCompression typos in tests 2694 splunk_hec sink Remove extra slash on Uri join 2711 pulsar sink Add new EncodingConfig 2671 sinks increment request id 2765 config Handle templates with invalid strftime elements 2737 aws_cloudwatch_sink Make throttling errors retryable 2752 aws_cloudwatch_logs sink Drop excessive large events 2770 sinks upgrade tower-limit to fix rate limiting 2779 kafka sink Fix typo integration tests 2832 sinks address potential deadlock 2838 influxdb_metrics sink Use i for unsigned ints on Influx v1 2868 splunk_hec source fix wrap path 2873 elasticsearch sink refresh aws creds when required 2859 aws_cloudwatch_logs sink Handle and enforce batch.max_bytes 2916 operations Fix the cmark-gfm installation at builder image 2996 operations Use perl rename at builder 2999 gcp provider sync token request 2993 loki sink Set a proper default for the batch buffer initialization 3016 deps update crates in lock file after merges 3030 cli Cleanup generate command 3046 docs adjust log event in coercer transform 3072 reduce transform improve array handling 3076 loki sink Drop the batch byte size limit 3025 file sink append instead of overwriting files 3084 elasticsearch sink stop logging every received event 3085

What’s next

Kubernetes! ⎈
We’re doing this integration right. We’ve been alpha testing with a number of large clusters and we’ve honed in on a very high-quality approach that reflects the UX Vector is going after. You can learn more in our Kubernetes Integration RFC. Do you want to beta test? Tell us!
WASM plugins 🔌
This release included an experimental implementation of a WASM transform behind a wasm build flag. We plan to stabilize WASM transforms, as well as add WASM sinks and sources, enabling you to extend Vector in the language of your choice with little performance penalty. ❤️ WASM. Check out the WASM Plugin RFC for more info.
Dynamic HTTP Rate-Limiting (AIMD)
Fiddling with rate-limits is a frustrating endeavor. If you set them too high you risk overwhelming the service; too low and you’re unnecessarily limiting throughput. What happens if you deploy a few more Vector instances? Vector is planning to solve this by automatically detecting the optimal rate limits, taking inspiration from TCP congestion control algorithms. Check out the Dynamic HTTP Rate Limiting RFC.

Download Version 0.10.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...