Vector v0.12.0 release notes

Code name Vector Remap Language

The Vector team is pleased to announce 0.12.0. This release introduces the Vector Remap Language, an expression-oriented language designed for transforming observability data (logs and metrics) in a safe and performant manner.

Check out the VRL announcement post for more details as well as the highlights and changelog for a complete list of changes in 0.12.0.

Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

Improved file source checkpointing

Author photo for Ben Johnson
Ben Johnson
type: enhancement domain: sources source: file
The kafka sink now supports metrics

Author photo for Ben Johnson
Ben Johnson
type: enhancement domain: metrics sink: kafka
Introducing vector top

Author photo for Luc Perkins
Luc Perkins
type: new feature domain: observability
The internal_logs source

Author photo for Luc Perkins
Luc Perkins
type: new feature source: internal_logs

Changelog

106 enhancements

socket source Add UNIX datagram mode 5298 remap support path queries for variables 5277 remap add support for literal arrays 5278 remap accept array of arguments for function parameters 5283 kubernetes platform Support exposing internal mertics out of the box in our Helm charts 4854 networking Implement TCP keepalive time configuration 5157 docker_logs source Enable excluding Docker containers via configuration 5301 codec Remove default encoding.codec where appropriate 5281 prometheus sink Export metadata to remote_write 5445 auth Add support for basic auth credendtials in URL for more sinks 5379 remap add integer division 5353 remap allow multiple statements in boolean conditional 5438 docker_logs source Add host and TLS configurations 5532 remap track inner container type definitions 5375 remap add support for literal maps 5279 kubernetes platform Expose host metrics in our Helm charts out of the box 5100 config Add global healthchecks option 5528 remap transform add parse_aws_alb_log function 5489 remap improve error messages, part 1 5477 remap transform add parse_aws_vpc_flow_log function 5504 elasticsearch sink add support for data streams 5126 generator source Enable format selection for generator source 5399 remap Add includes function for arrays 5541 remap Add to_syslog_level function for parsing Syslog levels into strings 5503 remap transform docs for remap transform metrics 5626 remap Add to_syslog_severity function for Syslog log level keywords 5505 codec observability Print warning on encoding error 5651 remap add encode_json function 5653 networking security humio sink Support TLS configuration for Humio sinks 5692 graphql api File source metrics 5645 docker_logs source Add hostname to Docker source 5690 generator source Add JSON format 5630 generator source Add RFC 3164 format 5631 http source Add gzip & deflate decompression 5638 graphql api componentByName query 5707 kafka sink Better tracing/tests on rdkafka options. 5652 kubernetes platform Adds script for generating README.md for Agent role helm-chart 5590 ci Build ARMv7 images for all supported platforms 5787 remap transform add parse_aws_cloudwatch_log_subscription_message function 5764 remap Enable casting timestamps into integers 5753 kafka sink allow send metrics to kafka 5712 remap non-boolean if conditional compile-time error 5835 graphql api Relay connections (components, initially) 5747 graphql api Relay-compatible connections for sources/transforms/sinks 5880 remap Add is_nullish function to VRL 5754 remap Move VRL CLI into the Vector CLI 5756 remap bare variable names 5856 remap Add to_syslog_facility function for parsing syslog facilities into strings 5770 remap Add to_unix_timestamp function 5767 remap Add push and append functions 5750 remap Add base64 encoding and decoding functions 5768 config remove the check_fields default for conditions 5978 kubernetes platform Allow passing component configs as YAML in Helm charts 5673 remap Open docs URL for help in VRL REPL 5926 remap Pretty print functions list in VRL REPL 5935 prometheus_remote_write sink Support X-Scope-OrgID header 5987 config deprecate check_fields conditions 6037 remap Add charset parameter to encode_base64 function 6065 prometheus_exporter sink add tls support 6078 internal_logs source Capture initialization logs 6014 remap New get_env_var Remap function 6017 graphql api Filtering in the API 6028 remap Update VRL CLI behavior 6035 remap Add length function for VRL 6088 graphql api API sorting, starting with components 6115 graphql api Sorting for sources/transforms/sinks 6136 file source Add encoding charset and line delimiter configuration 5436 postgresql_metrics source support regexp in filters 6118 auth aws platform Add auth.access_key_id and auth.secret_access_key options 6188 remap transform add get_hostname function 6141 kubernetes platform add the ability to configure Vector API at the Helm charts 6248 performance Add more benchmark comparisons 6233 file source split start_at_beginning into simpler options 6178 networking Make size of socket send/receive buffers configurable 6177 remap maintain type definitions for maps 6182 remap Add charset option to decode_base64 function 6296 host_metrics source Add uptime & boot_time metrics 6292 remap Add join function 6313 graphql api FileSourceMetricFile sort 6154 file source Log IO errors when globbing 6384 templating Add support for metrics in templates 6351 releasing change release profile 6202 prometheus_remote_write source Reintegrate complex metrics 6280 ci Restrict benchmark runs to specific directories 6421 remap improved type definitions for or with nullable fields 6462 dev Convert to indoc blocks 6386 deps Split fake data into separate module 6464 cli Enable opening docs for VRL error codes 6465 remap add merge and merge equals operations 6482 performance Adjust the strength of gauge_add atomics 6480 remap Add function name hinting to VRL REPL 6205 observability Add events_in_total & events_out_total metrics 6433 remap Extract version in parse_syslog function 6499 ci Add VRL test suite to CI 6537 ci Build Rustdoc site on PR merge 6551 remap Update VRL error messages 6506 ci rustdoc build changes 6561 data model Enhance vector protocol for new metric data 6554 transforms Allow specifying default timezone 6533 lua transform Add support for metric namespaces 6623 remap added rate limit parameter to log function. 6622 remap add parse_apache_log function 6608 remap added the parse_logfmt remap function 6659 kubernetes_logs source support for kubeconfigs as an alternative to in_cluster configuration 4837 datadog_logs sink Define events for the datadog_logs sink 6637 http source URL path configuration flexibility, path also exposed as a new field 6626

25 new features

remap add redact function 5297 administration Helm chart for deploying Vector with an aggregator role 4404 remap add initial “trl” CLI 5450 remap support multiline expressions 5309 new source New internal_logs source 5346 remap add initial CLI REPL support 5490 remap add remap functions to CLI 5531 remap add ok error handling function 5469 remap enable remap to work with metric events 5475 remap add parse_regex and parse_regex_all remap functions 5594 remap added parse_key_value remap function 5760 remap add error coalescing operator 5830 remap support “abort on error” bang-function-calls 5876 remap error-return-value error handling pattern 5911 file source allow user configuration of the ignore_not_found option 5616 remap diagnostic error messages 6023 remap improve test harness set-up 6105 config add wildcard expansion to inputs 6170 prometheus_remote_write sink Add HTTP auth support 6281 remap transform Add parse_common_log function 6230 remap transform Add parse_glog function 6293 kubernetes platform Expose node-port for vector-aggregator 6020 remap re-add progressive type-checking to the language 6535 observability emit basic component utilization statistics 6634 remap transform Add tag_types_externally function 6382

72 bug fixes

kubernetes platform Add exclusion label at vector-aggregator 5388 config fixing Compression parsing with tests 5429 tests require features for generate_configfile test 5427 remap Fix build issue caused by macro removal 5480 metrics Add missing methods to Subscriber forwarder 5529 kubernetes platform Correct the prometheus format parsing at K8s E2E tests for the metrics pipeline 5540 kubernetes platform Implement improvements to the Pod recreation on Helm release upgrades 5525 observability Emit processed_events_total in topology for transforms 5492 prometheus sink fix set metric reset on flush period 5258 documentation tenant_id is templatable on loki sink 5629 remap Fix type definition for includes() 5628 tests podman pod stop/rm don’t need a –name option 5669 networking Allow owning the same port with different protocols 5625 journald source handle incorrect int sequences as strings 5683 journald source fix tests for fake journal 5694 http source lower handle http req log to debug 5705 internal_metrics source Make internal metrics pass promtools check 5743 syslog source do not use static year in tests 5803 metrics Trivial fixes for metrics batch buffer 5699 internal docs Correct misspelling 5826 deps fix features for lib/shared 5829 remap correctly handle nested map insertions 5834 docker_logs source Fix HTTP and HTTPS URL 5882 tests Handle early return from influxdb create database 5928 prometheus_exporter sink Fix prometheus exposition format 5914 config Avoid alias in_flight_limit 5950 remap Rename tokenize to parse_tokens 5983 remap Make the redact function fallible 5933 tests Set core_threads for merge_and_fork 5968 loki sink Fix remove_label_fields 5974 influxdb_metrics sink Fix transmission of counter metrics 5922 kubernetes platform Adjusting pod security policy to allow vector-agent to read host logs 6019 kubernetes platform Adjusting to adhere to linting rules used by ct 6021 remap allow comments between multiline statements 6036 internal_metrics source Rework defaults and a minimal scraping interval value 5682 buffers Trigger leveldb compact after certain amount of deleted bytes 5821 kubernetes_logs source Support for collecting logs from static pods 6056 prometheus_exporter sink Fix Metrics not Updating 5917 remap ipv6_to_ipv4 function allows ipv4 addresses to pass through 6052 remap Update VRL docs URLs 6120 docker_logs source fix and document exclude_self logic 5998 aws_cloudwatch_logs sink Rename healthcheck error 6192 nats sink call ack on any result 6267 remap fix field sublocation_type in parse_aws_vpc_flow_log 6229 tests test again released version of loki 6279 remap parse_grok should error when it fails to parse 6271 config disallow non-unique component ids 6270 metrics Fix metrics::tests::test_cardinality_metric 6278 tests use poll backend for watcher 6286 loki sink Fix out-of-order error 5973 tests revert poll watcher change 6311 metrics Rework the metrics batch buffer 6251 deployment #5642 - build deb packages that install on Deb 8 6332 remap Re-flatten the remap-functions directory tree 6335 topology Fix handling of component errors 6309 file source Deprecate fingerprint.bytes option 6338 docker dependencies Update bollard to 0.9.1 to fix #5937 6369 observability Emit processed_events_total after transform has processed event 6294 mongodb_metrics source postgresql_metrics source remove connect in SourceConfig::build 6246 metrics sinks Move normalization out of MetricsBuffer 6340 wasm transform Fix parse_json Wasm function 6398 templating Fix merge error with test cases 6405 kubernetes_logs source Handle OK response from api_watcher with embedded desync elegantly 6053 vector source Close stream on bad data in vector source 6414 http sink Retry sending after no connection 6542 elasticsearch sink parse create bulk action error 6550 remap re-enable @ in identifiers. 6529 remap don’t mark / as a valid escape target 6577 config Preload log scheme 6581 elastic service networking Set correct Host header with AWS ES 6617 remap allow partially quoted fields to be parsed as undelimited. 6639 remap correct type defs for boolean operations 6662

What’s next

Vector's own observability
We will be working to improve Vector’s own observability with high-quality internal metrics, logs, CLI tools, and dashboards.
Improved Vector to Vector communication
The vector source and sink currently use the TCP protocol to transmit data. While this has served users well, it presents operational challenges for large-scale deployments of Vector. Because HTTP is widely used and easy to integrate into platforms like Kubernetes we will be moving these components to the HTTP protocol.
Schema support
With the inclusion of type-safety in VRL we will be working to officially support common and custom schemas within Vector. This will be an opt-in feature that will provide Vector with type information about your data, making Vector type safe from end to end and further raising confidence that Vector will operate properly post-deployment.

Download Version 0.12.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...