Vector v0.9.0 release notes

Code name The Homestretch

Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

New Apache Pulsar Sink

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sinks sink: pulsar
New Encoding Options

Author photo for Ben Johnson
Ben Johnson
type: enhancement domain: sinks
New Honeycomb Sink

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sinks sink: honeycomb
The elasticsearch sink compression option now defaults to none

Author photo for Ben Johnson
Ben Johnson
type: breaking change domain: sinks sink: elasticsearch
Improved Shutdown

Author photo for Ben Johnson
Ben Johnson
type: enhancement domain: topology
4 New Sinks!

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sinks
À la carte Custom Vector Builds

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sources source: vector
New Dedupe Transform

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sources source: vector
New Filter Transform

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: transforms transform: filter
New Tag Cardinality Limit Transform

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: transforms transform: tag_cardinality_limit
The Vector Source Now Accepts Metrics

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sources source: vector
Compression Now Available In The Kafka Sink

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sinks sink: kafka
More Condition Predicates

Author photo for Ben Johnson
Ben Johnson
type: enhancement domain: config domain: transforms transform: filter transform: swimlanes
New Datadog Logs Sink

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sinks sink: datadog_logs
New Papertrail Sink

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sinks sink: papertrail
NixOS Support

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: platforms
The Vector Source and Sink Support TLS

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sources source: vector
A New Guides Section

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: website
Lua Transform v2

Author photo for Ben Johnson
Ben Johnson
type: enhancement domain: sources source: vector

Changelog

48 enhancements

config Allow setting individual log_schema.* options 1923 socket source syslog source vector source Add TLS support to socket, syslog, and vector sources 1892 config Default to check_fields condition when specifying conditions 1947 config Add is_log and is_metric conditions 1950 operations Create feature flags for all components enabling custom Vector builds 1924 cli Add --fragment flag to generate subcmd 1956 cli Allow names in generate subcmd expression 1955 logplex source splunk_hec source Add TLS support 1968 sinks Add new encoding.only_fields and encoding.except_fields options 1915 networking Move all TLS support over to openssl 1986 config Add contains and prefix predicates 1997 vector sink Add support for TLS 2025 elasticsearch sink Add timeout query parameter 2038 sampler transform Add new key_field option 2039 lua transform Add version configuration option 2056 config Support escaping in the field path notation 2081 add_fields transform Allow native TOML tables 2068 remove_fields transform rename_fields transform Add new drop_empty option to drop empty objects automatically 2077 lua transform Support metric events in version 2 2095 config Make LogSchema more optional 2113 networking Add User-agent header to all outgoing HTTP requests 2130 splunk_hec sink Add new index option to specify custom Splunk indexes 2127 observability The add_fields, remove_fields, and rename_fields now produce a debug log when a field is replaced or nonexistent 2148 aws_s3 sink Increase default rate limit to allow for higher throughput 2161 socket sink Add support for UDP mode 2162 networking Add support for tls.verify_hostname in HTTP based sinks 2164 config Rename prefix condition predicate to starts_with 2181 config Add new ends_with condition predicate 2183 field_filter transform Deprecate in favor of the new filter transform 2195 config Add new regex condition predicate 2198 observability Warn that old field_filter transform is deprecated 2197 regex_parser transform Add new target_field option 2023 networking Detect TCP disconnects earlier 2209 lua transform Implement all hooks and timers in version 2 2126 vector source vector sink Support metric events 2245 ux Use external tagging for metrics serialization 2231 config Add source_key and source_type_key to… 2244 filter transform Accept metrics as input 2267 lua transform Set search_dirs to config dirs by default 2274 gcp_cloud_storage sink Make ACL optional 2283 docker source Add elegant shutdown behavior 2260 sources Add source_type field 2298 cli Show possible values for --log-format, --color, and --format 2300 topology Improve topology shutdown logs 2345 gcp provider Add support for instance accounts 2351 splunk_hec source Wire in new ShutdownSignal for improved shutdown 2373 journald source Wire in ShutdownSignal for improved shutdown 2364 sources Wire in new ShutdownSignal for improved shutdown 2365

14 new features

cli Add --log-format CLI option 1908 new source Initial http source implementation 1650 kafka sink Add support for compression 1969 new transform Initial dedupe transform implementation 1848 new transform Initial ’tag_cardinality_limit’ transform implementation 1959 new sink Initial papertrail sink implementation 1835 new sink Initial honeycomb sink implementation 1847 new sink Initial pulsar sink implementation 1665 new sink Initial datadog_logs sink implementation 1832 elasticsearch sink Check for and log partial ingestion failures 2185 new transform Initial filter transform implementation 2088 new transform Initial kubernetes_pod_metadata transform implementation 1888 cli Add -qqq command line flag to disable logs 2230 new source Initial internal metrics source implementation 1953

20 bug fixes

config Force symlink re-evaluation when detecting a changed Vector config 2034 transforms Fix insertion to arrays in field path notation 2062 merge transform Support nested fields and arrays 1936 journald source Fix handling of message array data 2053 gcp_stackdrive_logging sink Rename gcp_stackdriver_logging sink to gcp_stackdriver_logs 2121 tests Fix disk pressure 2217 elasticsearch sink Default compression to none 2219 loki sink Pass tls settings to healthcheck 2234 influxdb_metrics sink Pass tls settings to healthcheck 2238 gcp_cloud_storage sink Use kebab case for ACL encoding 2259 aws_kinesis_firehose sink Fix healthchecks 2276 aws_ec2_metadata transform Only run required requests based on the supplied fields 2265 website correct typo in testing post 2291 website small fixes for testing post 2292 kafka sink Run healthcheck in rt context 2288 aws_cloudwatch_logs sink Sort events by timestamp 2320 sinks Reduce credentials fetch timeout 2342 lua transform Don’t require isdst, wday, yday in timestamps 2335 aws_cloudwatch_logs sink Allow creation race of groups and streams 2355 lua transform support sub-second timestamp resolution 2333

What’s next

Kubernetes!
We’re doing this integration right. We’ve been alpha testing with a number of large clusters and we’ve honed in on a very high-quality approach that reflects the UX Vector is going after. You can learn more in our Kubernetes Integration RFC. Feedback is welcome!
WASM Driven Plugin Architecture
This release included an overhaul of our lua transform, which represents a big step towards making Vector more flexible. Lua is an excellent language when you need some quick scripting capabilities. But what if you need something more powerful? Like writing your own custom source, transform, or sink? Vector plans to solve this with WASM foreign module support, enabling you to extend Vector in the language of your choice with little performance penalty. ❤️ WASM. Check out the WASM Foreign Module Support RFC for more info.
Dynamic HTTP Rate-Limiting (AIMD)
Fiddling with rate-limits is a frustrating endeavor. If you set them too high you risk overwhelming the service; too low and you’re unnecessarily limiting throughput. And what happens if you deploy a few more Vector instances? Vector is planning to solve this by automatically detecting the optimal rate limits, taking inspiration from TCP congestion control algorithms. Check out the Dynamic HTTP Rate Limiting RFC.
Vector Observability

Did you know this release includes a new hidden internal source that emits metrics reflecting Vector’s own internal state? 👀 We didn’t highlight this source because we have not finished instrumenting Vector and we’ll be changing the internal schema used.

Fun fact, we took an event-driven approach to Vector’s internal observability, which we strongly believe in. We want Vector to be a good example of our own advice. Checkout the Event-Driven Observability RFC.

Download Version 0.9.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...