Monitoring and observing Vector

Although Vector is primarily used to handle observability data from a wide variety of sources, we also strive to make Vector highly observable itself. To that end, Vector provides two sources, internal_logs and internal_metrics, that you can use to handle logs and metrics produced by Vector just like you would logs and metrics from any other source.

Logs

Vector provides clear, informative, well-structured logs via the internal_logs source. This section shows you how to use them in your Vector topology.

Which logs Vector pipes through the internal_logs source is determined by the log level, which defaults to info.

In addition to the internal_logs source, Vector also writes its logs to stderr, which can be captured by Kubernetes, SystemD, or however you are running Vector.

Accessing logs

You can access Vector’s logs by adding an internal_logs source to your topology. Here’s an example configuration that takes Vector’s logs and pipes them to the console as plain text:

[sources.vector_logs]
type = "internal_logs"

[sinks.console]
type = "console"
inputs = ["vector_logs"]
encoding.codec = "text"

Using Vector logs

Once Vector logs enter your topology through the internal_logs source, you can treat them like logs from any other system, i.e. you can transform them and send them off to any number of sinks. The configuration below, for example, transforms Vector’s logs using the remap transform and Vector Remap Language and then stores those logs in ClickHouse:

[sources.vector_logs]
type = "internal_logs"

[transforms.modify]
type = "remap"
inputs = ["vector_logs"]

# Reformat the timestamp to Unix time
source = '''
  .timestamp = to_unix_timestamp!(to_timestamp!(.timestamp))
'''

[sinks.database]
type = "clickhouse"
inputs = ["modify"]
host = "http://localhost:8123"
table = "vector-log-data"

Configuring logs

Levels

Vector logs at the info level by default. You can set a different level when starting up your instance using either command-line flags or the VECTOR_LOG environment variable. The table below details these options:

Stack traces

You can enable full error backtraces by setting the RUST_BACKTRACE=full environment variable. More on this in the Troubleshooting guide.

Metrics

You can monitor metrics produced by Vector using the internal_metrics source. As with Vector’s internal logs, you can configure an internal_metrics source and use the piped-in metrics however you wish. Here’s an example configuration that delivers Vector’s metrics to a Prometheus remote write endpoint.

[sources.vector_metrics]
type = "internal_metrics"

[sinks.prometheus]
type = ["prometheus_remote_write"]
endpoint = ["https://localhost:8087/"]
inputs = ["vector_metrics"]

Metrics catalogue

The table below provides a list of internal metrics provided by Vector. See the docs for the internal_metrics source for more detailed information about the available metrics.

Troubleshooting

More information in our troubleshooting guide:

How it works

Event-driven observability

Vector employs an event-driven observability strategy that ensures consistent and correlated telemetry data. You can read more about our approach in RFC 2064.

Log rate limiting

Vector rate limits log events in the hot path. This enables you to get granular insight without the risk of saturating IO and disrupting the service. The trade-off is that repetitive logs aren’t logged.