Global options reference

Page source Edit this page

Global configuration parameters

acknowledgements

common optional object
Controls how acknowledgements are handled by all sources. These settings may be overridden in individual sources.

acknowledgements.enabled

common optional bool
Controls if sources will wait for destination sinks to deliver the events, or persist them to a disk buffer, before acknowledging receipt. If set to true, all capable sources will have acknowledgements enabled.
default: false

data_dir

optional string literal
The directory used for persisting Vector state, such as on-disk buffers, file checkpoints, and more. Please make sure the Vector project has write permissions to this directory.
Examples 
"/var/lib/vector"
"/var/local/lib/vector/"
"/home/user/vector/"
default: /var/lib/vector/

enrichment_tables

optional object

Configuration options for an enrichment table to be used in a remap transform. Currently supported are:

For the lookup in the enrichment tables to be as performant as possible, the data is indexed according to the fields that are used in the search. Note that indices can only be created for fields for which an exact match is used in the condition. For range searches, an index isn’t used and the enrichment table drops back to a sequential scan of the data. A sequential scan shouldn’t impact performance significantly provided that there are only a few possible rows returned by the exact matches in the condition. We don’t recommend using a condition that uses only date range searches.

enrichment_tables.file

required object
Configuration options for the file that provides the enrichment table.
enrichment_tables.file.encoding
required object
Configuration options for the encoding of the enrichment table’s file.
enrichment_tables.file.encoding.delimiter
optional string literal
The delimiter used to separate fields in each row of the CSV file.
Examples 
":"
default: ,
enrichment_tables.file.encoding.include_headers
optional bool

Set include_headers to true if the first row of the CSV file contains the headers for each column. This is the default behavior.

If you set it to false, there are no headers and the columns are referred to by their numerical index.

default: true
enrichment_tables.file.encoding.type
common optional string literal
The encoding of the file. Currently, only CSV is supported.
default: csv
enrichment_tables.file.path
required string literal
The path of the enrichment table file. Currently, only CSV files are supported.
Examples 
"/data/info.csv"
"./info.csv"
enrichment_tables.file.schema
common optional object

Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.

Timestamp coercions need to be prefaced with timestamp|, for example "timestamp|%F". Timestamp specifiers can use either of the following:

  1. One of the built-in-formats listed in the Timestamp Formats table below.
  2. The time format specifiers from Rust’s chrono library.

Types

  • bool
  • string
  • float
  • integer
  • date
  • timestamp (see the table below for formats)

Timestamp Formats

FormatDescriptionExample
%F %TYYYY-MM-DD HH:MM:SS2020-12-01 02:37:54
%v %TDD-Mmm-YYYY HH:MM:SS01-Dec-2020 02:37:54
%FT%TISO 8601/RFC 3339 format without time zone2020-12-01T02:37:54
%a, %d %b %Y %TRFC 822/2822 without time zoneTue, 01 Dec 2020 02:37:54
%a %d %b %T %Ydate command output without time zoneTue 01 Dec 02:37:54 2020
%a %b %e %T %Yctime formatTue Dec 1 02:37:54 2020
%sUNIX timestamp1606790274
%FT%TZISO 8601/RFC 3339 UTC2020-12-01T09:37:54Z
%+ISO 8601/RFC 3339 UTC with time zone2020-12-01T02:37:54-07:00
%a %d %b %T %Z %Ydate command output with time zoneTue 01 Dec 02:37:54 PST 2020
%a %d %b %T %z %Ydate command output with numeric time zoneTue 01 Dec 02:37:54 -0700 2020
%a %d %b %T %#z %Ydate command output with numeric time zone (minutes can be missing or present)Tue 01 Dec 02:37:54 -07 2020

Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.

Examples 
{
  "duration": "float",
  "status": "int",
  "success": "bool",
  "timestamp_custom": "timestamp|%a %b %e %T %Y",
  "timestamp_iso8601": "timestamp|%F",
  "timestamp_unix": "timestamp|%F %T"
}

enrichment_tables.geoip

required object

Configuration options for MaxMind databases.

The following MaxMind databases are currently supported:

  • GeoLite2-ASN.mmdb (free) — Determine the autonomous system number and organization associated with an IP address.
  • GeoLite2-City.mmdb (free) — Determine the country, subdivisions, city, and postal code associated with IPv4 and IPv6 addresses worldwide.
  • GeoIP2-City.mmdb (paid) — Determine the country, subdivisions, city, and postal code associated with IPv4 and IPv6 addresses worldwide.
  • GeoIP2-ISP.mmdb (paid) — Determine the Internet Service Provider (ISP), organization name, and autonomous system organization and number associated with an IP address.
  • GeoIP2-Anonymous-IP.mmdb (paid) — Determine proxy, VPN, hosting, and other anonymous IP addresses.

The database file should be in the MaxMind DB file format.

This enrichment table only supports lookup with IP address.

enrichment_tables.geoip.locale
optional string literal
The locale to use to lookup the country name and region name for the city database. See Locations Files
Examples 
"de"
"en"
"es"
"fr"
"ja"
"pt-BR"
"ru"
"zh-CN"
default: en
enrichment_tables.geoip.path
required string literal
Path to the database file.
Examples 
"/path/to/GeoLite2-City.mmdb"
"/path/to/GeoLite2-ISP.mmdb"

enrichment_tables.mmdb

required object

Configuration options for generic MaxMind databases.

The database file should be in the MaxMind DB file format.

This enrichment table only supports lookup with IP address.

enrichment_tables.mmdb.path
required string literal
Path to the database file.
Examples 
"/path/to/GeoLite2-City.mmdb"
"/path/to/GeoLite2-ISP.mmdb"

enrichment_tables.type

required string literal enum
Determines the type of enrichment data that is to be loaded.
Enum options
OptionDescription
fileEnrich data from a CSV file.
geoipEnrich data from a GeoIp MaxMind database.
mmdbEnrich data from any MaxMind database.
Examples 
"file"
"geoip"
"mmdb"

expire_metrics

optional object

If set, Vector will configure the internal metrics system to automatically remove all metrics that have not been updated in the given time.

If set to a negative value expiration is disabled.

Warning

Deprecated, please use expire_metrics_secs instead.

expire_metrics.nsecs

common optional uint
The fractional number of seconds after which to expire metrics.
Examples 
0

expire_metrics.secs

common optional uint
The whole number of seconds after which to expire metrics.
Examples 
60

expire_metrics_secs

optional float

Vector will expire internal metrics that haven’t been emitted/updated in the configured interval (default 300 seconds).

Note that internal counters that are expired but are later updated will have their values reset to zero. Be careful to set this value high enough to avoid expiring critical but infrequently updated internal counters.

If set to a negative value expiration is disabled.

Examples 
60
default: 300 (seconds)

healthchecks

optional object
Configures health checks for all sinks.

healthchecks.enabled

common optional bool
Disables all health checks if false, otherwise sink specific option overrides it.
default: true

healthchecks.require_healthy

optional bool
Exit on startup if any sinks’ health check fails. Overridden by --require-healthy command line flag.
default: false

log_schema

optional object
Configures default log schema for all events. This is used by Vector components to assign the fields on incoming events. These values are ignored if log namespacing is enabled. (See Log Namespacing)

log_schema.host_key

common optional string literal
Sets the event key to use for the event host field.
Examples 
"host"
"@host"
default: host

log_schema.message_key

common optional string literal
Sets the event key to use for the event message field.
Examples 
"message"
"@message"
default: message

log_schema.metadata_key

common optional string literal
Sets the event key to use for event metadata field (e.g. error or abort annotations in the remap transform).
Examples 
"@metadata"
"meta"
default: metadata

log_schema.source_type_key

common optional string literal
Sets the event key to use for the event source type field that is set by some sources.
Examples 
"source_type"
"@source_type"
default: source_type

log_schema.timestamp_key

common optional string literal
Sets the event key to use for the event timestamp field.
Examples 
"timestamp"
"@timestamp"
default: timestamp

proxy

optional object
Configures an HTTP(S) proxy for Vector to use.

proxy.enabled

optional bool
If false the proxy will be disabled.
default: true

proxy.http

optional string literal
The URL to proxy HTTP requests through.
Examples 
"http://foo.bar:3128"

proxy.https

optional string literal
The URL to proxy HTTPS requests through.
Examples 
"http://foo.bar:3128"

proxy.no_proxy

optional [string]

A list of hosts to avoid proxying. Allowed patterns here include:

PatternExample match
Domain namesexample.com matches requests to example.com
Wildcard domains.example.com matches requests to example.com and its subdomains
IP addresses127.0.0.1 matches requests to 127.0.0.1
CIDR blocks192.168.0.0./16 matches requests to any IP addresses in this range
Splat* matches all hosts

schema

optional object
Configures options for how Vector handles event schema.

schema.log_namespace

optional bool

Globally enables / disables log namespacing. See Log Namespacing for more details. If you want to enable individual sources, there is a config option in the source configuration.

Known issues:

  • Enabling log namespacing doesn’t work when disk buffers are used (see #18574)
default: false

secret

optional object
Configuration options to retrieve secrets from external backend in order to avoid storing secrets in plaintext in Vector config. Multiple backends can be configured. Use SECRET[<backend_name>.<secret_key>] to tell Vector to retrieve the secret. This placeholder is replaced by the secret retrieved from the relevant backend.

secret.aws_secrets_manager

required object

Retrieve secrets from AWS Secrets Manager.

The secret must be a JSON text string with key/value pairs. For example:

{
	"username": "test",
	"password": "example-password"
}

If an error occurred retrieving the secrets, Vector logs the error and exits.

Secrets are loaded when Vector starts or if Vector receives a SIGHUP signal triggering its configuration reload process.

secret.aws_secrets_manager.secret_id
required string literal
The ID of the secret to be retrieved.
Examples 
"/secret/foo-bar"

secret.directory

required object

Retrieve secrets from file contents in a directory.

The directory must contain files with names corresponding to secret keys.

If an error occurs while reading the file, Vector will log the error and exit.

Secrets are loaded when Vector starts or if Vector receives a SIGHUP signal triggering its configuration reload process.

secret.directory.path
required string literal
The path of the directory with secrets.
Examples 
"${CREDENTIALS_DIRECTORY}"
"/path/to/secrets-directory"
secret.directory.remove_trailing_whitespace
optional bool
Remove trailing whitespace from file contents.
default: false

secret.exec

required object

Run a local command to retrieve secrets.

The provided command will be run and provided a list of secrets to fetch, determined from the configuration file, on stdin as JSON in the format:

{"version": "1.0", "secrets": ["secret1", "secret2"]}

The executable is expected to respond with the values of these secrets on stdout, also as JSON, in the format:

{
	"secret1": {"value": "secret_value", "error": null},
	"secret2": {"value": null, "error": "could not fetch the secret"}
}

If an error is returned for any secrets, or if the command exits with a non-zero status code, Vector will log the errors and exit.

Secrets are loaded when Vector starts or if Vector receives a SIGHUP signal triggering its configuration reload process.

secret.exec.command
required [string]
The command to be run, plus any arguments required.
Examples 
[
  "/path/to/get-secret",
  "-s"
]
[
  "/path/to/vault-wrapper"
]
secret.exec.timeout
optional uint
The amount of time Vector will wait for the command to complete.
default: 5 (seconds)

secret.file

required object

Retrieve secrets from a file path.

The secret must be a JSON text string with key/value pairs. For example:

{
	"username": "test",
	"password": "example-password"
}

If an error occurs while reading the file, Vector will log the error and exit.

Secrets are loaded when Vector starts or if Vector receives a SIGHUP signal triggering its configuration reload process.

secret.file.path
required string literal
The file path to read.
Examples 
"/path/to/secret.json"

telemetry

optional object
Configures options for how Vector emits telemetry.

telemetry.tags

optional object
Controls which tags should be included with the vector_component_sent_events_total and vector_component_sent_event_bytes_total metrics.
telemetry.tags.emit_service
optional bool

Adds a service tag with the service component the event was received from.

For logs this is the field that has been determined to mean service. Each source may define different fields for this. For example, with syslog events the appname field is used.

Metric events will use the tag named service.

If no service is available a - is emitted for this tag.

default: false
telemetry.tags.emit_source
common optional bool

Add a source tag with the source component the event was received from.

If there is no source component, for example if the event was generated by the lua transform a - is emitted for this tag.

default: false

timezone

optional string literal
The name of the time zone to apply to timestamp conversions that do not contain an explicit time zone. The time zone name may be any name in the TZ database, or local to indicate system local time.
Examples 
"local"
"America/NewYork"
"EST5EDT"
default: local
Sidebar