VRL example reference
Here you’ll find a comprehensive list of all VRL program examples. These examples demonstrate the breadth of the language and its observability-focused facilities.
Try Using the VRL subcommand
You can run these examples using the vector vrl subcommand with --input (input is newline
delimited JSON file representing a list of events). and --program (VRL program) to pass in the
example input and program as well as --print-object to show the modified object. The below
examples show pretty-printed JSON for the input events, so collapse these to single lines when
passing in via --input.
For example: vector vrl --input input.json --program program.vrl --print-object. This closely
matches how VRL will receive the input in a running Vector instance.
VRL REPL
Additionally, if vector vrl is run without any arguments, it will spawn a REPL
(Read–eval–print loop).
Assuming you have Vector installed, you can run vector vrl to start the REPL.
From there, you can type help and press return to get further help.
The REPL behaves nearly identical to the programs you write for your Vector configuration, and can be used to test individual snippets of complex programs before you commit them to your production configuration.
The vector vrl command has many other capabilities, run the command using
vector vrl --help to see more information.
Real world examples
Parse Syslog logs
Given this Vector log event......and this VRL program......the following log event is output:
{
"message": "\u003c102\u003e1 2020-12-22T15:22:31.111Z vector-user.biz su 2666 ID389 - Something went wrong"
}. |= parse_syslog!(.message){
"log": {
"appname": "su",
"facility": "ntp",
"hostname": "vector-user.biz",
"message": "Something went wrong",
"msgid": "ID389",
"procid": 2666,
"severity": "info",
"timestamp": "2020-12-22T15:22:31.111Z",
"version": 1
}
}Parse key/value (logfmt) logs
Given this Vector log event......and this VRL program......the following log event is output:
{
"message": "@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
}. = parse_key_value!(.message){
"log": {
"@timestamp": "Sun Jan 10 16:47:39 EST 2021",
"id": "ConsumerFetcherManager-1382721708341",
"level": "info",
"module": "kafka.consumer.ConsumerFetcherManager",
"msg": "Stopping all fetchers",
"tag#production": "stopping_fetchers"
}
}Parse custom logs
Given this Vector log event......and this VRL program......the following log event is output:
{
"message": "2021/01/20 06:39:15 +0000 [error] 17755#17755: *3569904 open() \"/usr/share/nginx/html/test.php\" failed (2: No such file or directory), client: xxx.xxx.xxx.xxx, server: localhost, request: \"GET /test.php HTTP/1.1\", host: \"yyy.yyy.yyy.yyy\""
}. |= parse_regex!(.message, r'^(?P<timestamp>\d+/\d+/\d+ \d+:\d+:\d+ \+\d+) \[(?P<severity>\w+)\] (?P<pid>\d+)#(?P<tid>\d+):(?: \*(?P<connid>\d+))? (?P<message>.*)$')
# Coerce parsed fields
.timestamp = parse_timestamp(.timestamp, "%Y/%m/%d %H:%M:%S %z") ?? now()
.pid = to_int!(.pid)
.tid = to_int!(.tid)
# Extract structured data
message_parts = split(.message, ", ", limit: 2)
structured = parse_key_value(message_parts[1], key_value_delimiter: ":", field_delimiter: ",") ?? {}
.message = message_parts[0]
. = merge(., structured){
"log": {
"client": "xxx.xxx.xxx.xxx",
"connid": "3569904",
"host": "yyy.yyy.yyy.yyy",
"message": "open() \"/usr/share/nginx/html/test.php\" failed (2: No such file or directory)",
"pid": 17755,
"request": "GET /test.php HTTP/1.1",
"server": "localhost",
"severity": "error",
"tid": 17755,
"timestamp": "2021-01-20T06:39:15Z"
}
}Multiple parsing strategies
Given this Vector log event......and this VRL program......the following log event is output:
{
"message": "\u003c102\u003e1 2020-12-22T15:22:31.111Z vector-user.biz su 2666 ID389 - Something went wrong"
}structured =
parse_syslog(.message) ??
parse_common_log(.message) ??
parse_regex!(.message, r'^(?P<timestamp>\d+/\d+/\d+ \d+:\d+:\d+) \[(?P<severity>\w+)\] (?P<pid>\d+)#(?P<tid>\d+):(?: \*(?P<connid>\d+))? (?P<message>.*)$')
. = merge(., structured){
"log": {
"appname": "su",
"facility": "ntp",
"hostname": "vector-user.biz",
"message": "Something went wrong",
"msgid": "ID389",
"procid": 2666,
"severity": "info",
"timestamp": "2020-12-22T15:22:31.111Z",
"version": 1
}
}Modify metric tags
Given this Vector metric event......and this VRL program......the following metric event is output:
{
"counter": {
"value": 102
},
"kind": "incremental",
"name": "user_login_total",
"tags": {
"email": "vic@vector.dev",
"host": "my.host.com",
"instance_id": "abcd1234"
}
}.tags.environment = get_env_var!("ENV") # add
.tags.hostname = del(.tags.host) # rename
del(.tags.email){
"metric": {
"counter": {
"value": 102
},
"kind": "incremental",
"name": "user_login_total",
"tags": {
"environment": "production",
"hostname": "my.host.com",
"instance_id": "abcd1234"
}
}
}Emitting multiple logs from JSON
Given this Vector log event......and this VRL program......the following log event is output:
{
"message": "[{\"message\": \"first_log\"}, {\"message\": \"second_log\"}]"
}. = parse_json!(.message) # sets `.` to an array of objects[
{
"log": {
"message": "first_log"
}
},
{
"log": {
"message": "second_log"
}
}
]Emitting multiple non-object logs from JSON
Given this Vector log event......and this VRL program......the following log event is output:
{
"message": "[5, true, \"hello\"]"
}. = parse_json!(.message) # sets `.` to an array[
{
"log": {
"message": 5
}
},
{
"log": {
"message": true
}
},
{
"log": {
"message": "hello"
}
}
]Invalid argument type
Given this Vector log event......and this VRL program......you should see this error:
{
"not_a_string": 1
}upcase(42)error[E110]: invalid argument type
┌─ :1:8
│
1 │ upcase(42)
│ ^^
│ │
│ this expression resolves to the exact type integer
│ but the parameter "value" expects the exact type string
│
= try: ensuring an appropriate type at runtime
=
= 42 = string!(42)
= upcase(42)
=
= try: coercing to an appropriate type and specifying a default value as a fallback in case coercion fails
=
= 42 = to_string(42) ?? "default"
= upcase(42)
=
= see documentation about error handling at https://errors.vrl.dev/#handling
= learn more about error code 110 at https://errors.vrl.dev/110
= see language documentation at https://vrl.dev
= try your code in the VRL REPL, learn more at https://vrl.dev/examples
Unhandled fallible assignment
Given this Vector log event......and this VRL program......you should see this error:
{
"message": "key1=value1 key2=value2"
}structured = parse_key_value(.message)error[E103]: unhandled fallible assignment
┌─ :1:14
│
1 │ structured = parse_key_value(.message)
│ ------------ ^^^^^^^^^^^^^^^^^^^^^^^^^
│ │ │
│ │ this expression is fallible because at least one argument's type cannot be verified to be valid
│ │ update the expression to be infallible by adding a `!`: `parse_key_value!(.message)`
│ │ `.message` argument type is `any` and this function expected a parameter `value` of type `string`
│ or change this to an infallible assignment:
│ structured, err = parse_key_value(.message)
│
= see documentation about error handling at https://errors.vrl.dev/#handling
= see functions characteristics documentation at https://vrl.dev/expressions/#function-call-characteristics
= learn more about error code 103 at https://errors.vrl.dev/103
= see language documentation at https://vrl.dev
= try your code in the VRL REPL, learn more at https://vrl.dev/examples
Array examples
Codec examples
decode_base16
Decodes the
value (a Base16 string) into its original string.Decode Base16 data
Source
decode_base16!("796f752068617665207375636365737366756c6c79206465636f646564206d65")Return
you have successfully decoded meLearn more about the decode_base16 function
decode_base64
Decodes the
value (a Base64 string) into its original string.Decode Base64 data (default)
Source
decode_base64!("eW91IGhhdmUgc3VjY2Vzc2Z1bGx5IGRlY29kZWQgbWU=")Return
you have successfully decoded meLearn more about the decode_base64 function
Decode Base64 data (URL safe)
Source
decode_base64!("eW91IGNhbid0IG1ha2UgeW91ciBoZWFydCBmZWVsIHNvbWV0aGluZyBpdCB3b24ndA==", charset: "url_safe")Return
you can't make your heart feel something it won'tLearn more about the decode_base64 function
decode_gzip
Decodes the
value (a Gzip string) into its original string.Decode Gzip data
Source
encoded_text = decode_base64!("H4sIAHEAymMAA6vML1XISCxLVSguTU5OLS5OK83JqVRISU3OT0lNUchNBQD7BGDaIAAAAA==")
decode_gzip!(encoded_text)Return
you have successfully decoded meLearn more about the decode_gzip function
decode_mime_q
Replaces q-encoded or base64-encoded encoded-word substrings in the
value with their original string.Decode single encoded-word
Source
decode_mime_q!("=?utf-8?b?SGVsbG8sIFdvcmxkIQ==?=")Return
Hello, World!Learn more about the decode_mime_q function
Embedded
Source
decode_mime_q!("From: =?utf-8?b?SGVsbG8sIFdvcmxkIQ==?= <=?utf-8?q?hello=5Fworld=40example=2ecom?=>")Return
From: Hello, World! <hello_world@example.com>Learn more about the decode_mime_q function
Without charset
Source
decode_mime_q!("?b?SGVsbG8sIFdvcmxkIQ==")Return
Hello, World!Learn more about the decode_mime_q function
decode_percent
Decodes a percent-encoded
value like a URL.Percent decode a value
Source
decode_percent("foo%20bar%3F")Return
foo bar?Learn more about the decode_percent function
decode_punycode
Decodes a punycode encoded
value, such as an internationalized domain name (IDN). This function assumes that the value passed is meant to be used in IDN context and that it is either a domain name or a part of it.Decode a punycode encoded internationalized domain name
Source
decode_punycode!("www.xn--caf-dma.com")Return
www.café.comLearn more about the decode_punycode function
Decode an ASCII only string
Source
decode_punycode!("www.cafe.com")Return
www.cafe.comLearn more about the decode_punycode function
Ignore validation
Source
decode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false)Return
١٠.٦٦.٣٠.٥.Learn more about the decode_punycode function
decode_snappy
Decodes the
value (a Snappy string) into its original string.Decode Snappy data
Source
encoded_text = decode_base64!("LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg==")
decode_snappy!(encoded_text)Return
The quick brown fox jumps over 13 lazy dogs.Learn more about the decode_snappy function
decode_zlib
Decodes the
value (a Zlib string) into its original string.Decode Zlib data
Source
encoded_text = decode_base64!("eJwNy4ENwCAIBMCNXIlQ/KqplUSgCdvXAS41qPMHshCB2R1zJlWIVlR6UURX2+wx2YcuK3kAb9C1wd6dn7Fa+QH9gRxr")
decode_zlib!(encoded_text)Return
you_have_successfully_decoded_me.congratulations.you_are_breathtaking.Learn more about the decode_zlib function
decode_zstd
Decodes the
value (a Zstandard string) into its original string.Decode Zstd data
Source
encoded_text = decode_base64!("KLUv/QBY/QEAYsQOFKClbQBedqXsb96EWDax/f/F/z+gNU4ZTInaUeAj82KqPFjUzKqhcfDqAIsLvAsnY1bI/N2mHzDixRQA")
decode_zstd!(encoded_text)Return
you_have_successfully_decoded_me.congratulations.you_are_breathtaking.Learn more about the decode_zstd function
encode_base16
Encodes the
value to Base16.Encode to Base16
Source
encode_base16("please encode me")Return
706c6561736520656e636f6465206d65Learn more about the encode_base16 function
encode_base64
Encodes the
value to Base64.Encode to Base64 (default)
Source
encode_base64("please encode me")Return
cGxlYXNlIGVuY29kZSBtZQ==Learn more about the encode_base64 function
Encode to Base64 (without padding)
Source
encode_base64("please encode me, no padding though", padding: false)Return
cGxlYXNlIGVuY29kZSBtZSwgbm8gcGFkZGluZyB0aG91Z2gLearn more about the encode_base64 function
Encode to Base64 (URL safe)
Source
encode_base64("please encode me, but safe for URLs", charset: "url_safe")Return
cGxlYXNlIGVuY29kZSBtZSwgYnV0IHNhZmUgZm9yIFVSTHM=Learn more about the encode_base64 function
encode_gzip
Encodes the
value to Gzip.Encode to Gzip
Source
encoded_text = encode_gzip("please encode me")
encode_base64(encoded_text)Return
H4sIAAAAAAAA/yvISU0sTlVIzUvOT0lVyE0FAI4R4vcQAAAALearn more about the encode_gzip function
encode_json
Encodes the
value to JSON.Encode to JSON
Source
.payload = encode_json({"hello": "world"})Return
{"hello":"world"}Learn more about the encode_json function
encode_key_value
Encodes the
value into key-value format with customizable delimiters. Default delimiters match
the logfmt format.Encode with default delimiters (no ordering)
Source
encode_key_value({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})Return
lvl=info msg="This is a message" ts=2021-06-05T17:20:00ZLearn more about the encode_key_value function
Encode with default delimiters (fields ordering)
Source
encode_key_value!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])Return
ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345Learn more about the encode_key_value function
Encode with default delimiters (nested fields)
Source
encode_key_value({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})Return
agent.name=foo event=log log.file.path=my.logLearn more about the encode_key_value function
Encode with default delimiters (nested fields ordering)
Source
encode_key_value!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])Return
event=log log.file.path=my.log agent.name=fooLearn more about the encode_key_value function
Encode with custom delimiters (no ordering)
Source
encode_key_value(
{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"},
field_delimiter: ",",
key_value_delimiter: ":"
)Return
lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00ZLearn more about the encode_key_value function
Encode with custom delimiters and flatten boolean
Source
encode_key_value(
{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "beta": true, "dropped": false},
field_delimiter: ",",
key_value_delimiter: ":",
flatten_boolean: true
)Return
beta,lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00ZLearn more about the encode_key_value function
encode_logfmt
Encodes the
value to logfmt.Encode to logfmt (no ordering)
Source
encode_logfmt({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})Return
lvl=info msg="This is a message" ts=2021-06-05T17:20:00ZLearn more about the encode_logfmt function
Encode to logfmt (fields ordering)
Source
encode_logfmt!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])Return
ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345Learn more about the encode_logfmt function
Encode to logfmt (nested fields)
Source
encode_logfmt({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})Return
agent.name=foo event=log log.file.path=my.logLearn more about the encode_logfmt function
Encode to logfmt (nested fields ordering)
Source
encode_logfmt!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])Return
event=log log.file.path=my.log agent.name=fooLearn more about the encode_logfmt function
encode_percent
Encodes a
value with percent encoding to safely be used in URLs.Percent encode all non-alphanumeric characters (default)
Source
encode_percent("foo bar?")Return
foo%20bar%3FLearn more about the encode_percent function
Percent encode only control characters
Source
encode_percent("foo bar", ascii_set: "CONTROLS")Return
foo %09barLearn more about the encode_percent function
encode_proto
Encodes the
value into a protocol buffer payload.Encode to proto
Source
.payload = encode_base64(encode_proto!({"name": "someone", "phones": [{"number": "123456"}]}, "resources/protobuf_descriptor_set.desc", "test_protobuf.Person"))Return
Cgdzb21lb25lIggKBjEyMzQ1Ng==Learn more about the encode_proto function
encode_punycode
Encodes a
value to punycode. Useful for internationalized domain names (IDN). This function assumes that the value passed is meant to be used in IDN context and that it is either a domain name or a part of it.Encode an internationalized domain name
Source
encode_punycode!("www.café.com")Return
www.xn--caf-dma.comLearn more about the encode_punycode function
Encode an internationalized domain name with mixed case
Source
encode_punycode!("www.CAFé.com")Return
www.xn--caf-dma.comLearn more about the encode_punycode function
Encode an ASCII only string
Source
encode_punycode!("www.cafe.com")Return
www.cafe.comLearn more about the encode_punycode function
Ignore validation
Source
encode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false)Return
xn--8hbb.xn--fiba.xn--8hbf.xn--eib.Learn more about the encode_punycode function
encode_snappy
Encodes the
value to Snappy.Encode to Snappy
Source
encoded_text = encode_snappy!("The quick brown fox jumps over 13 lazy dogs.")
encode_base64(encoded_text)Return
LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg==Learn more about the encode_snappy function
encode_zlib
Encodes the
value to Zlib.Encode to Zlib
Source
encoded_text = encode_zlib("please encode me")
encode_base64(encoded_text)Return
eJwryElNLE5VSM1Lzk9JVchNBQA0RQX7Learn more about the encode_zlib function
encode_zstd
Encodes the
value to Zstandard.Encode to Zstd
Source
encoded_text = encode_zstd("please encode me")
encode_base64(encoded_text)Return
KLUv/QBYgQAAcGxlYXNlIGVuY29kZSBtZQ==Learn more about the encode_zstd function
Coerce examples
Convert examples
from_unix_timestamp
Converts the value integer from a Unix timestamp to a VRL timestamp.
Converts from the number of seconds since the Unix epoch by default. To convert from milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.
Convert from a Unix timestamp (seconds)
Source
from_unix_timestamp!(5)Return
1970-01-01T00:00:05ZLearn more about the from_unix_timestamp function
Convert from a Unix timestamp (milliseconds)
Source
from_unix_timestamp!(5000, unit: "milliseconds")Return
1970-01-01T00:00:05ZLearn more about the from_unix_timestamp function
Convert from a Unix timestamp (nanoseconds)
Source
from_unix_timestamp!(5000, unit: "nanoseconds")Return
1970-01-01T00:00:00.000005ZLearn more about the from_unix_timestamp function
to_syslog_facility
Converts the
value, a Syslog facility code, into its corresponding
Syslog keyword. For example, 0 into "kern", 1 into "user", etc.Coerce to a Syslog facility
Source
to_syslog_facility!(4)Return
authLearn more about the to_syslog_facility function
to_syslog_level
Converts the
value, a Syslog severity level, into its corresponding keyword,
i.e. 0 into "emerg", 1 into "alert", etc.Coerce to a Syslog level
Source
to_syslog_level!(5)Return
noticeLearn more about the to_syslog_level function
to_syslog_severity
Coerce to Syslog severity
Source
to_syslog_severity!("alert")Return
1Learn more about the to_syslog_severity function
to_unix_timestamp
Converts the value timestamp into a Unix timestamp.
Returns the number of seconds since the Unix epoch by default. To return the number in milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.
Convert to a Unix timestamp (seconds)
Source
to_unix_timestamp(t'2021-01-01T00:00:00+00:00')Return
1609459200Learn more about the to_unix_timestamp function
Convert to a Unix timestamp (milliseconds)
Source
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "milliseconds")Return
1609459200000Learn more about the to_unix_timestamp function
Convert to a Unix timestamp (nanoseconds)
Source
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "nanoseconds")Return
1609459200000000000Learn more about the to_unix_timestamp function
Debug examples
assert
Asserts the
condition, which must be a Boolean expression. The program is aborted with
message if the condition evaluates to false.assert_eq
Asserts that two expressions,
left and right, have the same value. The program is
aborted with message if they do not have the same value.Unsuccessful assertion with custom log message
Source
assert_eq!(1, 0, message: "Unequal integers")Learn more about the assert_eq function
Enrichment examples
find_enrichment_table_records
Searches an enrichment table for rows that match the provided condition.
For file enrichment tables, this condition needs to be a VRL object in which
the key-value pairs indicate a field to search mapped to a value to search in that field.
This function returns the rows that match the provided condition(s). All fields need to
match for rows to be returned; if any fields do not match, then no rows are returned.
There are currently two forms of search criteria:
Exact match search. The given field must match the value exactly. Case sensitivity can be specified using the
case_sensitiveargument. An exact match search can use an index directly into the dataset, which should make this search fairly “cheap” from a performance perspective.Date range search. The given field must be greater than or equal to the
fromdate and less than or equal to thetodate. A date range search involves sequentially scanning through the rows that have been located using any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. Therefore, use date ranges as the only criteria when the enrichment data set is very small.
For geoip and mmdb enrichment tables, this condition needs to be a VRL object with a single key-value pair
whose value needs to be a valid IP address. Example: {"ip": .ip }. If a return field is expected
and without a value, null is used. This table can return the following fields:
ISP databases:
autonomous_system_numberautonomous_system_organizationisporganization
City databases:
city_namecontinent_codecountry_codecountry_nameregion_coderegion_namemetro_codelatitudelongitudepostal_codetimezone
Connection-Type databases:
connection_type
To use this function, you need to update your configuration to
include an
enrichment_tables
parameter.
Exact match
Source
find_enrichment_table_records!("test",
{
"surname": "smith",
},
case_sensitive: false)Return
[{"firstname":"Bob","id":1,"surname":"Smith"},{"firstname":"Fred","id":2,"surname":"Smith"}]Learn more about the find_enrichment_table_records function
Date range search
Source
find_enrichment_table_records!("test",
{
"surname": "Smith",
"date_of_birth": {
"from": t'1985-01-01T00:00:00Z',
"to": t'1985-12-31T00:00:00Z'
}
})Return
[{"firstname":"Bob","id":1,"surname":"Smith"},{"firstname":"Fred","id":2,"surname":"Smith"}]Learn more about the find_enrichment_table_records function
get_enrichment_table_record
Searches an enrichment table for a row that matches the provided condition. A single row must be matched. If no rows are found or more than one row is found, an error is returned.
For file enrichment tables, this condition needs to be a VRL object in which
the key-value pairs indicate a field to search mapped to a value to search in that field.
This function returns the rows that match the provided condition(s). All fields need to
match for rows to be returned; if any fields do not match, then no rows are returned.
There are currently two forms of search criteria:
Exact match search. The given field must match the value exactly. Case sensitivity can be specified using the
case_sensitiveargument. An exact match search can use an index directly into the dataset, which should make this search fairly “cheap” from a performance perspective.Date range search. The given field must be greater than or equal to the
fromdate and less than or equal to thetodate. A date range search involves sequentially scanning through the rows that have been located using any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. Therefore, use date ranges as the only criteria when the enrichment data set is very small.
For geoip and mmdb enrichment tables, this condition needs to be a VRL object with a single key-value pair
whose value needs to be a valid IP address. Example: {"ip": .ip }. If a return field is expected
and without a value, null is used. This table can return the following fields:
ISP databases:
autonomous_system_numberautonomous_system_organizationisporganization
City databases:
city_namecontinent_codecountry_codecountry_nameregion_coderegion_namemetro_codelatitudelongitudepostal_codetimezone
Connection-Type databases:
connection_type
To use this function, you need to update your configuration to
include an
enrichment_tables
parameter.
Exact match
Source
get_enrichment_table_record!("test",
{
"surname": "bob",
"firstname": "John"
},
case_sensitive: false)Return
{
"firstname": "Bob",
"id": 1,
"surname": "Smith"
}Learn more about the get_enrichment_table_record function
Date range search
Source
get_enrichment_table_record!("test",
{
"surname": "Smith",
"date_of_birth": {
"from": t'1985-01-01T00:00:00Z',
"to": t'1985-12-31T00:00:00Z'
}
})Return
{
"firstname": "Bob",
"id": 1,
"surname": "Smith"
}Learn more about the get_enrichment_table_record function
Enumerate examples
compact
Compacts the
value by removing empty values, where empty values are defined using the
available parameters.Compact an array
Source
compact(["foo", "bar", "", null, [], "buzz"], string: true, array: true, null: true)Return
["foo","bar","buzz"]Learn more about the compact function
Compact an object
Source
compact({"field1": 1, "field2": "", "field3": [], "field4": null}, string: true, array: true, null: true)Return
{
"field1": 1
}Learn more about the compact function
filter
Filter elements from a collection.
This function currently does not support recursive iteration.
The function uses the function closure syntax to allow reading the key-value or index-value combination for each item in the collection.
The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.
See the examples below to learn about the closure syntax.
Filter elements
Source
filter(array!(.tags)) -> |_index, value| {
# keep any elements that aren't equal to "foo"
value != "foo"
}Return
["bar","baz"]Learn more about the filter function
flatten
Flattens the
value into a single-level representation.Flatten array
Source
flatten([1, [2, 3, 4], [5, [6, 7], 8], 9])Return
[1,2,3,4,5,6,7,8,9]Learn more about the flatten function
Flatten object
Source
flatten({
"parent1": {
"child1": 1,
"child2": 2
},
"parent2": {
"child3": 3
}
})Return
{
"parent1.child1": 1,
"parent1.child2": 2,
"parent2.child3": 3
}Learn more about the flatten function
for_each
Iterate over a collection.
This function currently does not support recursive iteration.
The function uses the “function closure syntax” to allow reading the key/value or index/value combination for each item in the collection.
The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.
See the examples below to learn about the closure syntax.
Tally elements
Source
tally = {}
for_each(array!(.tags)) -> |_index, value| {
# Get the current tally for the `value`, or
# set to `0`.
count = int(get!(tally, [value])) ?? 0
# Increment the tally for the value by `1`.
tally = set!(tally, [value], count + 1)
}
tallyReturn
{
"bar": 1,
"baz": 1,
"foo": 2
}Learn more about the for_each function
includes
Determines whether the
value array includes the specified item.Array includes
Source
includes(["apple", "orange", "banana"], "banana")Return
trueLearn more about the includes function
keys
Returns the keys from the object passed into the function.
Get keys from the object
Source
keys({"key1": "val1", "key2": "val2"})Return
["key1","key2"]Learn more about the keys function
length
Returns the length of the value.
- If
valueis an array, returns the number of elements. - If
valueis an object, returns the number of top-level keys. - If
valueis a string, returns the number of bytes in the string. If you want the number of characters, seestrlen.
Length (object)
Source
length({
"portland": "Trail Blazers",
"seattle": "Supersonics"
})Return
2Learn more about the length function
Length (nested object)
Source
length({
"home": {
"city": "Portland",
"state": "Oregon"
},
"name": "Trail Blazers",
"mascot": {
"name": "Blaze the Trail Cat"
}
})Return
3Learn more about the length function
Length (array)
Source
length(["Trail Blazers", "Supersonics", "Grizzlies"])Return
3Learn more about the length function
Length (string)
Source
length("The Planet of the Apes Musical")Return
30Learn more about the length function
map_keys
Map the keys within an object.
If recursive is enabled, the function iterates into nested
objects, using the following rules:
- Iteration starts at the root.
- For every nested object type:
- First return the key of the object type itself.
- Then recurse into the object, and loop back to item (1) in this list.
- Any mutation done on a nested object before recursing into it, are preserved.
- For every nested array type:
- First return the key of the array type itself.
- Then find all objects within the array, and apply item (2) to each individual object.
The above rules mean that map_keys with
recursive enabled finds all keys in the target,
regardless of whether nested objects are nested inside arrays.
The function uses the function closure syntax to allow reading the key for each item in the object.
The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.
See the examples below to learn about the closure syntax.
Upcase keys
Source
map_keys(.) -> |key| { upcase(key) }Return
{
"BAR": "bar",
"FOO": "foo"
}Learn more about the map_keys function
De-dot keys
Source
map_keys(., recursive: true) -> |key| { replace(key, ".", "_") }Return
{
"labels": {
"app_kubernetes_io/name": "mysql"
}
}Learn more about the map_keys function
map_values
Map the values within a collection.
If recursive is enabled, the function iterates into nested
collections, using the following rules:
- Iteration starts at the root.
- For every nested collection type:
- First return the collection type itself.
- Then recurse into the collection, and loop back to item (1) in the list
- Any mutation done on a collection before recursing into it, are preserved.
The function uses the function closure syntax to allow mutating the value for each item in the collection.
The same scoping rules apply to closure blocks as they do for regular blocks, meaning, any variable defined in parent scopes are accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.
Check out the examples below to learn about the closure syntax.
Upcase values
Source
map_values(.) -> |value| { upcase!(value) }Return
{
"bar": "BAR",
"foo": "FOO"
}Learn more about the map_values function
match_array
Determines whether the elements in the
value array matches the pattern. By default, it checks that at least one element matches, but can be set to determine if all the elements match.Match at least one element
Source
match_array(["foobar", "bazqux"], r'foo')Return
trueLearn more about the match_array function
Match all elements
Source
match_array(["foo", "foobar", "barfoo"], r'foo', all: true)Return
trueLearn more about the match_array function
Not all elements match
Source
match_array(["foo", "foobar", "baz"], r'foo', all: true)Learn more about the match_array function
strlen
Returns the number of UTF-8 characters in value. This differs from
length which counts the number of bytes of a string.
Note: This is the count of Unicode scalar values which can sometimes differ from Unicode code points.
unflatten
Unflattens the
value into a nested representation.Unflatten
Source
unflatten({
"foo.bar.baz": true,
"foo.bar.qux": false,
"foo.quux": 42
})Return
{
"foo": {
"bar": {
"baz": true,
"qux": false
},
"quux": 42
}
}Learn more about the unflatten function
Unflatten recursively
Source
unflatten({
"flattened.parent": {
"foo.bar": true,
"foo.baz": false
}
})Return
{
"flattened": {
"parent": {
"foo": {
"bar": true,
"baz": false
}
}
}
}Learn more about the unflatten function
Unflatten non-recursively
Source
unflatten({
"flattened.parent": {
"foo.bar": true,
"foo.baz": false
}
}, recursive: false)Return
{
"flattened": {
"parent": {
"foo.bar": true,
"foo.baz": false
}
}
}Learn more about the unflatten function
Ignore inconsistent keys values
Source
unflatten({
"a": 3,
"a.b": 2,
"a.c": 4
})Return
{
"a": {
"b": 2,
"c": 4
}
}Learn more about the unflatten function
unique
Returns the unique values for an array.
The first occurrence of each element is kept.
Unique
Source
unique(["foo", "bar", "foo", "baz"])Return
["foo","bar","baz"]Learn more about the unique function
values
Returns the values from the object passed into the function.
Get values from the object
Source
values({"key1": "val1", "key2": "val2"})Return
["val1","val2"]Learn more about the values function
Event examples
get_secret
Returns the value of the given secret from an event.
Get the Datadog API key from the event metadata
Source
get_secret("datadog_api_key")Return
secret valueLearn more about the get_secret function
remove_secret
Removes a secret from an event.
Removes the Datadog API key from the event
Source
remove_secret("datadog_api_key")Learn more about the remove_secret function
set_secret
Sets the given secret in the event.
Set the Datadog API key to the given value
Source
set_secret("datadog_api_key", "abc122")Learn more about the set_secret function
set_semantic_meaning
Sets a semantic meaning for an event. Note: This function assigns
meaning at startup, and has no runtime behavior. It is suggested
to put all calls to this function at the beginning of a VRL function. The function
cannot be conditionally called. For example, using an if statement cannot stop the meaning
from being assigned.
Sets custom field semantic meaning
Source
set_semantic_meaning(.foo, "bar")Learn more about the set_semantic_meaning function
Path examples
del
Removes the field specified by the static path from the target.
For dynamic path deletion, see the remove function.
exists
Checks whether the path exists for the target.
This function distinguishes between a missing path
and a path with a null value. A regular path lookup,
such as .foo, cannot distinguish between the two cases
since it always returns null if the path doesn’t exist.
get
Dynamically get the value of a given path.
If you know the path you want to look up, use
static paths such as .foo.bar[1] to get the value of that
path. However, if you do not know the path names,
use the dynamic get function to get the requested
value.
single-segment top-level field
Source
get!(value: { "foo": "bar" }, path: ["foo"])Return
barLearn more about the get function
multi-segment nested field
Source
get!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])Return
bazLearn more about the get function
array indexing
Source
get!(value: ["foo", "bar", "baz"], path: [-2])Return
barLearn more about the get function
remove
Dynamically remove the value for a given path.
If you know the path you want to remove, use
the del function and static paths such as del(.foo.bar[1])
to remove the value at that path. The del function returns the
deleted value, and is more performant than remove.
However, if you do not know the path names, use the dynamic
remove function to remove the value at the provided path.
single-segment top-level field
Source
remove!(value: { "foo": "bar" }, path: ["foo"])Learn more about the remove function
multi-segment nested field
Source
remove!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])Return
{
"foo": {}
}Learn more about the remove function
array indexing
Source
remove!(value: ["foo", "bar", "baz"], path: [-2])Return
["foo","baz"]Learn more about the remove function
compaction
Source
remove!(value: { "foo": { "bar": [42], "baz": true } }, path: ["foo", "bar", 0], compact: true)Return
{
"foo": {
"baz": true
}
}Learn more about the remove function
set
Dynamically insert data into the path of a given object or array.
If you know the path you want to assign a value to,
use static path assignments such as .foo.bar[1] = true for
improved performance and readability. However, if you do not
know the path names, use the dynamic set function to
insert the data into the object or array.
single-segment top-level field
Source
set!(value: { "foo": "bar" }, path: ["foo"], data: "baz")Return
{
"foo": "baz"
}Learn more about the set function
multi-segment nested field
Source
set!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"], data: "qux")Return
{
"foo": {
"bar": "qux"
}
}Learn more about the set function
array
Source
set!(value: ["foo", "bar", "baz"], path: [-2], data: 42)Return
["foo",42,"baz"]Learn more about the set function
Cryptography examples
decrypt
Decrypts a string with a symmetric encryption algorithm.
Supported Algorithms:
- AES-256-CFB (key = 32 bytes, iv = 16 bytes)
- AES-192-CFB (key = 24 bytes, iv = 16 bytes)
- AES-128-CFB (key = 16 bytes, iv = 16 bytes)
- AES-256-OFB (key = 32 bytes, iv = 16 bytes)
- AES-192-OFB (key = 24 bytes, iv = 16 bytes)
- AES-128-OFB (key = 16 bytes, iv = 16 bytes)
- AES-128-SIV (key = 32 bytes, iv = 16 bytes)
- AES-256-SIV (key = 64 bytes, iv = 16 bytes)
- Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes)
- Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes)
- Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
- CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes)
- XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
- XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
Decrypt value
Source
ciphertext = decode_base64!("5fLGcu1VHdzsPcGNDio7asLqE1P43QrVfPfmP4i4zOU=")
iv = decode_base64!("fVEIRkIiczCRWNxaarsyxA==")
key = "16_byte_keyxxxxx"
decrypt!(ciphertext, "AES-128-CBC-PKCS7", key, iv: iv)Return
super_secret_messageLearn more about the decrypt function
encrypt
Encrypts a string with a symmetric encryption algorithm.
Supported Algorithms:
- AES-256-CFB (key = 32 bytes, iv = 16 bytes)
- AES-192-CFB (key = 24 bytes, iv = 16 bytes)
- AES-128-CFB (key = 16 bytes, iv = 16 bytes)
- AES-256-OFB (key = 32 bytes, iv = 16 bytes)
- AES-192-OFB (key = 24 bytes, iv = 16 bytes)
- AES-128-OFB (key = 16 bytes, iv = 16 bytes)
- AES-128-SIV (key = 32 bytes, iv = 16 bytes)
- AES-256-SIV (key = 64 bytes, iv = 16 bytes)
- Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes)
- Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes)
- Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
- CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes)
- XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
- XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
Encrypt value
Source
plaintext = "super secret message"
iv = "1234567890123456" # typically you would call random_bytes(16)
key = "16_byte_keyxxxxx"
encrypted_message = encrypt!(plaintext, "AES-128-CBC-PKCS7", key, iv: iv)
encode_base64(encrypted_message)Return
GBw8Mu00v0Kc38+/PvsVtGgWuUJ+ZNLgF8Opy8ohIYE=Learn more about the encrypt function
hmac
Calculates a HMAC of the value using the given key.
The hashing algorithm used can be optionally specified.
For most use cases, the resulting bytestream should be encoded into a hex or base64 string using either encode_base16 or encode_base64.
This function is infallible if either the default algorithm value or a recognized-valid compile-time
algorithm string literal is used. Otherwise, it is fallible.
Calculate message HMAC (defaults: SHA-256), encoding to a base64 string
Source
encode_base64(hmac("Hello there", "super-secret-key"))Return
eLGE8YMviv85NPXgISRUZxstBNSU47JQdcXkUWcClmI=Learn more about the hmac function
Calculate message HMAC using SHA-224, encoding to a hex-encoded string
Source
encode_base16(hmac("Hello there", "super-secret-key", algorithm: "SHA-224"))Return
42fccbc2b7d22a143b92f265a8046187558a94d11ddbb30622207e90Learn more about the hmac function
Calculate message HMAC using a variable hash algorithm
Source
.hash_algo = "SHA-256"
hmac_bytes, err = hmac("Hello there", "super-secret-key", algorithm: .hash_algo)
if err == null {
.hmac = encode_base16(hmac_bytes)
}Return
78b184f1832f8aff3934f5e0212454671b2d04d494e3b25075c5e45167029662Learn more about the hmac function
md5
Calculates an md5 hash of the
value.Create md5 hash
Source
md5("foo")Return
acbd18db4cc2f85cedef654fccc4a4d8Learn more about the md5 function
seahash
Calculates a Seahash hash of the
value.
Note: Due to limitations in the underlying VRL data types, this function converts the unsigned 64-bit integer SeaHash result to a signed 64-bit integer. Results higher than the signed 64-bit integer maximum value wrap around to negative values.sha1
Calculates a SHA-1 hash of the
value.Calculate sha1 hash
Source
sha1("foo")Return
0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33Learn more about the sha1 function
IP examples
ip_aton
Converts IPv4 address in numbers-and-dots notation into network-order bytes represented as an integer.
This behavior mimics inet_aton.
ip_cidr_contains
Determines whether the
ip is contained in the block referenced by the cidr.IPv4 contains CIDR
Source
ip_cidr_contains!("192.168.0.0/16", "192.168.10.32")Return
trueLearn more about the ip_cidr_contains function
IPv6 contains CIDR
Source
ip_cidr_contains!("2001:4f8:4:ba::/64", "2001:4f8:4:ba:2e0:81ff:fe22:d1f1")Return
trueLearn more about the ip_cidr_contains function
ip_ntoa
Converts numeric representation of IPv4 address in network-order bytes to numbers-and-dots notation.
This behavior mimics inet_ntoa.
ip_ntop
Converts IPv4 and IPv6 addresses from binary to text form.
This behavior mimics inet_ntop.
Convert IPv4 address from bytes after decoding from Base64
Source
ip_ntop!(decode_base64!("wKgAAQ=="))Return
192.168.0.1Learn more about the ip_ntop function
Convert IPv6 address from bytes after decoding from Base64
Source
ip_ntop!(decode_base64!("IAENuIWjAAAAAIouA3BzNA=="))Return
2001:db8:85a3::8a2e:370:7334Learn more about the ip_ntop function
ip_pton
Converts IPv4 and IPv6 addresses from text to binary form.
- The binary form of IPv4 addresses is 4 bytes (32 bits) long.
- The binary form of IPv6 addresses is 16 bytes (128 bits) long.
This behavior mimics inet_pton.
Convert IPv4 address to bytes and encode to Base64
Source
encode_base64(ip_pton!("192.168.0.1"))Return
wKgAAQ==Learn more about the ip_pton function
Convert IPv6 address to bytes and encode to Base64
Source
encode_base64(ip_pton!("2001:db8:85a3::8a2e:370:7334"))Return
IAENuIWjAAAAAIouA3BzNA==Learn more about the ip_pton function
ip_to_ipv6
Converts the
ip to an IPv6 address.IPv4 to IPv6
Source
ip_to_ipv6!("192.168.10.32")Return
::ffff:192.168.10.32Learn more about the ip_to_ipv6 function
ipv6_to_ipv4
Converts the
ip to an IPv4 address. ip is returned unchanged if it’s already an IPv4 address. If ip is
currently an IPv6 address then it needs to be IPv4 compatible, otherwise an error is thrown.IPv6 to IPv4
Source
ipv6_to_ipv4!("::ffff:192.168.0.1")Return
192.168.0.1Learn more about the ipv6_to_ipv4 function
is_ipv4
Check if the string is a valid IPv4 address or not.
An [IPv4-mapped][https://datatracker.ietf.org/doc/html/rfc6890] or [IPv4-compatible][https://datatracker.ietf.org/doc/html/rfc6890] IPv6 address is not considered valid for the purpose of this function.
Number examples
format_int
Formats the integer
value into a string representation using the given base/radix.Format as a hexadecimal integer
Source
format_int!(42, 16)Return
2aLearn more about the format_int function
Format as a negative hexadecimal integer
Source
format_int!(-42, 16)Return
-2aLearn more about the format_int function
format_number
Formats the
value into a string representation of the number.Format a number (3 decimals)
Source
format_number(1234567.89, 3, decimal_separator: ".", grouping_separator: ",")Return
1,234,567.890Learn more about the format_number function
Object examples
match_datadog_query
Matches an object against a Datadog Search Syntax query.
OR query
Source
match_datadog_query({"message": "contains this and that"}, "this OR that")Return
trueLearn more about the match_datadog_query function
AND query
Source
match_datadog_query({"message": "contains only this"}, "this AND that")Learn more about the match_datadog_query function
Attribute wildcard
Source
match_datadog_query({"name": "foobar"}, "@name:foo*")Return
trueLearn more about the match_datadog_query function
Tag range
Source
match_datadog_query({"tags": ["a:x", "b:y", "c:z"]}, s'b:["x" TO "z"]')Return
trueLearn more about the match_datadog_query function
merge
Merges the
from object into the to object.Object merge (shallow)
Source
merge(
{
"parent1": {
"child1": 1,
"child2": 2
},
"parent2": {
"child3": 3
}
},
{
"parent1": {
"child2": 4,
"child5": 5
}
}
)Return
{
"parent1": {
"child2": 4,
"child5": 5
},
"parent2": {
"child3": 3
}
}Learn more about the merge function
Object merge (deep)
Source
merge(
{
"parent1": {
"child1": 1,
"child2": 2
},
"parent2": {
"child3": 3
}
},
{
"parent1": {
"child2": 4,
"child5": 5
}
},
deep: true
)Return
{
"parent1": {
"child1": 1,
"child2": 4,
"child5": 5
},
"parent2": {
"child3": 3
}
}Learn more about the merge function
unnest
Unnest an array field from an object to create an array of objects using that field; keeping all other fields.
Assigning the array result of this to . results in multiple events being emitted from remap. See the
remap transform docs for more details.
This is also referred to as explode in some languages.
Parse examples
parse_apache_log
Parses Apache access and error log lines. Lines can be in
common,
combined, or the default error format.Parse using Apache log format (common)
Source
parse_apache_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326", format: "common")Return
{
"host": "127.0.0.1",
"identity": "bob",
"message": "GET /apache_pb.gif HTTP/1.0",
"method": "GET",
"path": "/apache_pb.gif",
"protocol": "HTTP/1.0",
"size": 2326,
"status": 200,
"timestamp": "2000-10-10T20:55:36Z",
"user": "frank"
}Learn more about the parse_apache_log function
Parse using Apache log format (combined)
Source
parse_apache_log!(
s'127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth" "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"',
"combined",
)Return
{
"agent": "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0",
"host": "127.0.0.1",
"identity": "bob",
"message": "GET /apache_pb.gif HTTP/1.0",
"method": "GET",
"path": "/apache_pb.gif",
"protocol": "HTTP/1.0",
"referrer": "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth",
"size": 2326,
"status": 200,
"timestamp": "2000-10-10T20:55:36Z",
"user": "frank"
}Learn more about the parse_apache_log function
Parse using Apache log format (error)
Source
parse_apache_log!(
s'[01/Mar/2021:12:00:19 +0000] [ab:alert] [pid 4803:tid 3814] [client 147.159.108.175:24259] I will bypass the haptic COM bandwidth, that should matrix the CSS driver!',
"error"
)Return
{
"client": "147.159.108.175",
"message": "I will bypass the haptic COM bandwidth, that should matrix the CSS driver!",
"module": "ab",
"pid": 4803,
"port": 24259,
"severity": "alert",
"thread": "3814",
"timestamp": "2021-03-01T12:00:19Z"
}Learn more about the parse_apache_log function
parse_aws_alb_log
Parses
value in the Elastic Load Balancer Access format.Parse AWS ALB log
Source
parse_aws_alb_log!(
"http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\""
)Return
{
"actions_executed": "forward",
"chosen_cert_arn": null,
"classification": null,
"classification_reason": null,
"client_host": "192.168.131.39:2817",
"domain_name": null,
"elb": "app/my-loadbalancer/50dc6c495c0c9188",
"elb_status_code": "200",
"error_reason": null,
"matched_rule_priority": "0",
"received_bytes": 34,
"redirect_url": null,
"request_creation_time": "2018-11-30T22:22:48.364000Z",
"request_method": "GET",
"request_processing_time": 0,
"request_protocol": "HTTP/1.1",
"request_url": "http://www.example.com:80/",
"response_processing_time": 0,
"sent_bytes": 366,
"ssl_cipher": null,
"ssl_protocol": null,
"target_group_arn": "arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067",
"target_host": null,
"target_port_list": [],
"target_processing_time": 0.001,
"target_status_code": "200",
"target_status_code_list": [],
"timestamp": "2018-11-30T22:23:00.186641Z",
"trace_id": "Root=1-58337364-23a8c76965a2ef7629b185e3",
"traceability_id": null,
"type": "http",
"user_agent": "curl/7.46.0"
}Learn more about the parse_aws_alb_log function
parse_aws_cloudwatch_log_subscription_message
Parses AWS CloudWatch Logs events (configured through AWS Cloudwatch subscriptions) from the
aws_kinesis_firehose source.Parse AWS Cloudwatch Log subscription message
Source
parse_aws_cloudwatch_log_subscription_message!(.message)Return
{
"log_events": [
{
"id": "35683658089614582423604394983260738922885519999578275840",
"message": "{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}",
"timestamp": "2020-09-14T19:09:29.039Z"
}
],
"log_group": "test",
"log_stream": "test",
"message_type": "DATA_MESSAGE",
"owner": "111111111111",
"subscription_filters": [
"Destination"
]
}Learn more about the parse_aws_cloudwatch_log_subscription_message function
parse_aws_vpc_flow_log
Parses
value in the VPC Flow Logs format.Parse AWS VPC Flow log (default format)
Source
parse_aws_vpc_flow_log!("2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA")Return
{
"account_id": "123456789010",
"action": null,
"bytes": null,
"dstaddr": null,
"dstport": null,
"end": 1431280934,
"interface_id": "eni-1235b8ca123456789",
"log_status": "NODATA",
"packets": null,
"protocol": null,
"srcaddr": null,
"srcport": null,
"start": 1431280876,
"version": 2
}Learn more about the parse_aws_vpc_flow_log function
Parse AWS VPC Flow log (custom format)
Source
parse_aws_vpc_flow_log!(
"- eni-1235b8ca123456789 10.0.1.5 10.0.0.220 10.0.1.5 203.0.113.5",
"instance_id interface_id srcaddr dstaddr pkt_srcaddr pkt_dstaddr"
)Return
{
"dstaddr": "10.0.0.220",
"instance_id": null,
"interface_id": "eni-1235b8ca123456789",
"pkt_dstaddr": "203.0.113.5",
"pkt_srcaddr": "10.0.1.5",
"srcaddr": "10.0.1.5"
}Learn more about the parse_aws_vpc_flow_log function
Parse AWS VPC Flow log including v5 fields
Source
parse_aws_vpc_flow_log!("5 52.95.128.179 10.0.0.71 80 34210 6 1616729292 1616729349 IPv4 14 15044 123456789012 vpc-abcdefab012345678 subnet-aaaaaaaa012345678 i-0c50d5961bcb2d47b eni-1235b8ca123456789 ap-southeast-2 apse2-az3 - - ACCEPT 19 52.95.128.179 10.0.0.71 S3 - - ingress OK",
format: "version srcaddr dstaddr srcport dstport protocol start end type packets bytes account_id vpc_id subnet_id instance_id interface_id region az_id sublocation_type sublocation_id action tcp_flags pkt_srcaddr pkt_dstaddr pkt_src_aws_service pkt_dst_aws_service traffic_path flow_direction log_status")Return
{
"account_id": "123456789012",
"action": "ACCEPT",
"az_id": "apse2-az3",
"bytes": 15044,
"dstaddr": "10.0.0.71",
"dstport": 34210,
"end": 1616729349,
"flow_direction": "ingress",
"instance_id": "i-0c50d5961bcb2d47b",
"interface_id": "eni-1235b8ca123456789",
"log_status": "OK",
"packets": 14,
"pkt_dst_aws_service": null,
"pkt_dstaddr": "10.0.0.71",
"pkt_src_aws_service": "S3",
"pkt_srcaddr": "52.95.128.179",
"protocol": 6,
"region": "ap-southeast-2",
"srcaddr": "52.95.128.179",
"srcport": 80,
"start": 1616729292,
"sublocation_id": null,
"sublocation_type": null,
"subnet_id": "subnet-aaaaaaaa012345678",
"tcp_flags": 19,
"traffic_path": null,
"type": "IPv4",
"version": 5,
"vpc_id": "vpc-abcdefab012345678"
}Learn more about the parse_aws_vpc_flow_log function
parse_cef
Parses the
value in CEF (Common Event Format) format. Ignores everything up to CEF header. Empty values are returned as empty strings. Surrounding quotes are removed from values.Parse output generated by PTA
Source
parse_cef!(
"CEF:0|CyberArk|PTA|12.6|1|Suspected credentials theft|8|suser=mike2@prod1.domain.com shost=prod1.domain.com src=1.1.1.1 duser=andy@dev1.domain.com dhost=dev1.domain.com dst=2.2.2.2 cs1Label=ExtraData cs1=None cs2Label=EventID cs2=52b06812ec3500ed864c461e deviceCustomDate1Label=detectionDate deviceCustomDate1=1388577900000 cs3Label=PTAlink cs3=https://1.1.1.1/incidents/52b06812ec3500ed864c461e cs4Label=ExternalLink cs4=None"
)Return
{
"cefVersion": "0",
"cs1": "None",
"cs1Label": "ExtraData",
"cs2": "52b06812ec3500ed864c461e",
"cs2Label": "EventID",
"cs3": "https://1.1.1.1/incidents/52b06812ec3500ed864c461e",
"cs3Label": "PTAlink",
"cs4": "None",
"cs4Label": "ExternalLink",
"deviceCustomDate1": "1388577900000",
"deviceCustomDate1Label": "detectionDate",
"deviceEventClassId": "1",
"deviceProduct": "PTA",
"deviceVendor": "CyberArk",
"deviceVersion": "12.6",
"dhost": "dev1.domain.com",
"dst": "2.2.2.2",
"duser": "andy@dev1.domain.com",
"name": "Suspected credentials theft",
"severity": "8",
"shost": "prod1.domain.com",
"src": "1.1.1.1",
"suser": "mike2@prod1.domain.com"
}Learn more about the parse_cef function
Ignore syslog header
Source
parse_cef!(
"Sep 29 08:26:10 host CEF:1|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232"
)Return
{
"cefVersion": "1",
"deviceEventClassId": "100",
"deviceProduct": "threatmanager",
"deviceVendor": "Security",
"deviceVersion": "1.0",
"dst": "2.1.2.2",
"name": "worm successfully stopped",
"severity": "10",
"spt": "1232",
"src": "10.0.0.1"
}Learn more about the parse_cef function
Translate custom fields
Source
parse_cef!(
"CEF:0|Dev|firewall|2.2|1|Connection denied|5|c6a1=2345:0425:2CA1:0000:0000:0567:5673:23b5 c6a1Label=Device IPv6 Address",
translate_custom_fields: true
)Return
{
"Device IPv6 Address": "2345:0425:2CA1:0000:0000:0567:5673:23b5",
"cefVersion": "0",
"deviceEventClassId": "1",
"deviceProduct": "firewall",
"deviceVendor": "Dev",
"deviceVersion": "2.2",
"name": "Connection denied",
"severity": "5"
}Learn more about the parse_cef function
parse_common_log
Parses the
value using the Common Log Format (CLF).Parse using Common Log Format (with default timestamp format)
Source
parse_common_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326")Return
{
"host": "127.0.0.1",
"identity": "bob",
"message": "GET /apache_pb.gif HTTP/1.0",
"method": "GET",
"path": "/apache_pb.gif",
"protocol": "HTTP/1.0",
"size": 2326,
"status": 200,
"timestamp": "2000-10-10T20:55:36Z",
"user": "frank"
}Learn more about the parse_common_log function
Parse using Common Log Format (with custom timestamp format)
Source
parse_common_log!(
"127.0.0.1 bob frank [2000-10-10T20:55:36Z] \"GET /apache_pb.gif HTTP/1.0\" 200 2326",
"%+"
)Return
{
"host": "127.0.0.1",
"identity": "bob",
"message": "GET /apache_pb.gif HTTP/1.0",
"method": "GET",
"path": "/apache_pb.gif",
"protocol": "HTTP/1.0",
"size": 2326,
"status": 200,
"timestamp": "2000-10-10T20:55:36Z",
"user": "frank"
}Learn more about the parse_common_log function
parse_csv
Parses a single CSV formatted row. Only the first row is parsed in case of multiline input value.
Parse a single CSV formatted row
Source
parse_csv!("foo,bar,\"foo \"\", bar\"")Return
["foo","bar","foo \", bar"]Learn more about the parse_csv function
Parse a single CSV formatted row with custom delimiter
Source
parse_csv!("foo bar", delimiter: " ")Return
["foo","bar"]Learn more about the parse_csv function
parse_duration
Parses the
value into a human-readable duration format specified by unit.Parse duration (milliseconds)
Source
parse_duration!("1005ms", unit: "s")Return
1.005Learn more about the parse_duration function
parse_etld
Parses the eTLD from
value representing domain name.Parse eTLD
Source
parse_etld!("sub.sussex.ac.uk")Return
{
"etld": "ac.uk",
"etld_plus": "ac.uk",
"known_suffix": true
}Learn more about the parse_etld function
Parse eTLD+1
Source
parse_etld!("sub.sussex.ac.uk", plus_parts: 1)Return
{
"etld": "ac.uk",
"etld_plus": "sussex.ac.uk",
"known_suffix": true
}Learn more about the parse_etld function
Parse eTLD with unknown suffix
Source
parse_etld!("vector.acmecorp")Return
{
"etld": "acmecorp",
"etld_plus": "acmecorp",
"known_suffix": false
}Learn more about the parse_etld function
Parse eTLD with custom PSL
Source
parse_etld!("vector.acmecorp", psl: "resources/public_suffix_list.dat")Return
{
"etld": "acmecorp",
"etld_plus": "acmecorp",
"known_suffix": false
}Learn more about the parse_etld function
parse_glog
Parses the
value using the glog (Google Logging Library) format.Parse using glog
Source
parse_glog!("I20210131 14:48:54.411655 15520 main.c++:9] Hello world!")Return
{
"file": "main.c++",
"id": 15520,
"level": "info",
"line": 9,
"message": "Hello world!",
"timestamp": "2021-01-31T14:48:54.411655Z"
}Learn more about the parse_glog function
parse_grok
Parse using Grok
Source
parse_grok!(
"2020-10-02T23:22:12.223222Z info Hello world",
"%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"
)Return
{
"level": "info",
"message": "Hello world",
"timestamp": "2020-10-02T23:22:12.223222Z"
}Learn more about the parse_grok function
parse_groks
Parse using multiple Grok patterns
Source
parse_groks!(
"2020-10-02T23:22:12.223222Z info Hello world",
patterns: [
"%{common_prefix} %{_status} %{_message}",
"%{common_prefix} %{_message}",
],
aliases: {
"common_prefix": "%{_timestamp} %{_loglevel}",
"_timestamp": "%{TIMESTAMP_ISO8601:timestamp}",
"_loglevel": "%{LOGLEVEL:level}",
"_status": "%{POSINT:status}",
"_message": "%{GREEDYDATA:message}"
}
)Return
{
"level": "info",
"message": "Hello world",
"timestamp": "2020-10-02T23:22:12.223222Z"
}Learn more about the parse_groks function
parse_influxdb
Parses the
value as an InfluxDB line protocol
string, producing a list of Vector-compatible metrics.Parse InfluxDB line protocol
Source
parse_influxdb!("cpu,host=A,region=us-west usage_system=64i,usage_user=10u,temperature=50.5,on=true,sleep=false 1590488773254420000")Return
[{"gauge":{"value":64},"kind":"absolute","name":"cpu_usage_system","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":10},"kind":"absolute","name":"cpu_usage_user","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":50.5},"kind":"absolute","name":"cpu_temperature","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":1},"kind":"absolute","name":"cpu_on","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":0},"kind":"absolute","name":"cpu_sleep","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"}]Learn more about the parse_influxdb function
parse_json
Parses the
value as JSON.Parse JSON
Source
parse_json!("{\"key\": \"val\"}")Return
{
"key": "val"
}Learn more about the parse_json function
Parse JSON with max_depth
Source
parse_json!("{\"top_level\":{\"key\": \"val\"}}", max_depth: 1)Return
{
"top_level": "{\"key\": \"val\"}"
}Learn more about the parse_json function
parse_key_value
Parses the value in key-value format. Also known as logfmt.
- Keys and values can be wrapped with
". "characters can be escaped using\.
Parse logfmt log
Source
parse_key_value!(
"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)Return
{
"@timestamp": "Sun Jan 10 16:47:39 EST 2021",
"id": "ConsumerFetcherManager-1382721708341",
"level": "info",
"module": "kafka.consumer.ConsumerFetcherManager",
"msg": "Stopping all fetchers",
"tag#production": "stopping_fetchers"
}Learn more about the parse_key_value function
Parse comma delimited log
Source
parse_key_value!(
"path:\"/cart_link\", host:store.app.com, fwd: \"102.30.171.16\", dyno: web.1, connect:0ms, service:87ms, status:304, bytes:632, protocol:https",
field_delimiter: ",",
key_value_delimiter: ":"
)Return
{
"bytes": "632",
"connect": "0ms",
"dyno": "web.1",
"fwd": "102.30.171.16",
"host": "store.app.com",
"path": "/cart_link",
"protocol": "https",
"service": "87ms",
"status": "304"
}Learn more about the parse_key_value function
Parse comma delimited log with standalone keys
Source
parse_key_value!(
"env:prod,service:backend,region:eu-east1,beta",
field_delimiter: ",",
key_value_delimiter: ":",
)Return
{
"beta": true,
"env": "prod",
"region": "eu-east1",
"service": "backend"
}Learn more about the parse_key_value function
Parse duplicate keys
Source
parse_key_value!(
"at=info,method=GET,path=\"/index\",status=200,tags=dev,tags=dummy",
field_delimiter: ",",
key_value_delimiter: "=",
)Return
{
"at": "info",
"method": "GET",
"path": "/index",
"status": "200",
"tags": [
"dev",
"dummy"
]
}Learn more about the parse_key_value function
parse_klog
Parses the
value using the klog format used by Kubernetes components.Parse using klog
Source
parse_klog!("I0505 17:59:40.692994 28133 klog.go:70] hello from klog")Return
{
"file": "klog.go",
"id": 28133,
"level": "info",
"line": 70,
"message": "hello from klog",
"timestamp": "2024-05-05T17:59:40.692994Z"
}Learn more about the parse_klog function
parse_linux_authorization
Parses Linux authorization logs usually found under either
/var/log/auth.log (for Debian-based systems) or
/var/log/secure (for RedHat-based systems) according to Syslog format.Parse Linux authorization event
Source
parse_linux_authorization!(
s'Mar 23 2023 01:49:58 localhost sshd[1111]: Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar'
)Return
{
"appname": "sshd",
"hostname": "localhost",
"message": "Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar",
"procid": 1111,
"timestamp": "2023-03-23T01:49:58Z"
}Learn more about the parse_linux_authorization function
parse_logfmt
Parses the value in logfmt.
- Keys and values can be wrapped using the
"character. "characters can be escaped by the\character.- As per this logfmt specification, the
parse_logfmtfunction accepts standalone keys and assigns them a Boolean value oftrue.
Parse logfmt log
Source
parse_logfmt!(
"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)Return
{
"@timestamp": "Sun Jan 10 16:47:39 EST 2021",
"id": "ConsumerFetcherManager-1382721708341",
"level": "info",
"module": "kafka.consumer.ConsumerFetcherManager",
"msg": "Stopping all fetchers",
"tag#production": "stopping_fetchers"
}Learn more about the parse_logfmt function
parse_nginx_log
Parses Nginx access and error log lines. Lines can be in
combined,
ingress_upstreaminfo, or error format.Parse via Nginx log format (combined)
Source
parse_nginx_log!(
s'172.17.0.1 - alice [01/Apr/2021:12:02:31 +0000] "POST /not-found HTTP/1.1" 404 153 "http://localhost/somewhere" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"',
"combined",
)Return
{
"agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36",
"client": "172.17.0.1",
"compression": "2.75",
"referer": "http://localhost/somewhere",
"request": "POST /not-found HTTP/1.1",
"size": 153,
"status": 404,
"timestamp": "2021-04-01T12:02:31Z",
"user": "alice"
}Learn more about the parse_nginx_log function
Parse via Nginx log format (error)
Source
parse_nginx_log!(
s'2021/04/01 13:02:31 [error] 31#31: *1 open() "/usr/share/nginx/html/not-found" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "POST /not-found HTTP/1.1", host: "localhost:8081"',
"error"
)Return
{
"cid": 1,
"client": "172.17.0.1",
"host": "localhost:8081",
"message": "open() \"/usr/share/nginx/html/not-found\" failed (2: No such file or directory)",
"pid": 31,
"request": "POST /not-found HTTP/1.1",
"server": "localhost",
"severity": "error",
"tid": 31,
"timestamp": "2021-04-01T13:02:31Z"
}Learn more about the parse_nginx_log function
Parse via Nginx log format (ingress_upstreaminfo)
Source
parse_nginx_log!(
s'0.0.0.0 - bob [18/Mar/2023:15:00:00 +0000] "GET /some/path HTTP/2.0" 200 12312 "https://10.0.0.1/some/referer" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 462 0.050 [some-upstream-service-9000] [some-other-upstream-5000] 10.0.50.80:9000 19437 0.049 200 752178adb17130b291aefd8c386279e7',
"ingress_upstreaminfo"
)Return
{
"body_bytes_size": 12312,
"http_referer": "https://10.0.0.1/some/referer",
"http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36",
"proxy_alternative_upstream_name": "some-other-upstream-5000",
"proxy_upstream_name": "some-upstream-service-9000",
"remote_addr": "0.0.0.0",
"remote_user": "bob",
"req_id": "752178adb17130b291aefd8c386279e7",
"request": "GET /some/path HTTP/2.0",
"request_length": 462,
"request_time": 0.05,
"status": 200,
"timestamp": "2023-03-18T15:00:00Z",
"upstream_addr": "10.0.50.80:9000",
"upstream_response_length": 19437,
"upstream_response_time": 0.049,
"upstream_status": 200
}Learn more about the parse_nginx_log function
parse_proto
Parses the
value as a protocol buffer payload.Parse proto
Source
parse_proto!(decode_base64!("Cgdzb21lb25lIggKBjEyMzQ1Ng=="), "resources/protobuf_descriptor_set.desc", "test_protobuf.Person")Return
{
"name": "someone",
"phones": [
{
"number": "123456"
}
]
}Learn more about the parse_proto function
parse_query_string
Parses the
value as a query string.Parse query string
Source
parse_query_string("foo=%2B1&bar=2&bar=3&xyz")Return
{
"bar": [
"2",
"3"
],
"foo": "+1",
"xyz": ""
}Learn more about the parse_query_string function
Parse Ruby on Rails’ query string
Source
parse_query_string("?foo%5b%5d=1&foo%5b%5d=2")Return
{
"foo[]": [
"1",
"2"
]
}Learn more about the parse_query_string function
parse_regex
Parses the value using the provided Regex pattern.
This function differs from the parse_regex_all function in that it returns only the first match.
Parse using Regex (with capture groups)
Source
parse_regex!("first group and second group.", r'(?P<number>.*?) group')Return
{
"number": "first"
}Learn more about the parse_regex function
Parse using Regex (without capture groups)
Source
parse_regex!("first group and second group.", r'(\w+) group', numeric_groups: true)Return
{
"0": "first group",
"1": "first"
}Learn more about the parse_regex function
parse_regex_all
Parses the value using the provided Regex pattern.
This function differs from the parse_regex function in that it returns all matches, not just the first.
Parse using Regex (all matches)
Source
parse_regex_all!("first group and second group.", r'(?P<number>\w+) group', numeric_groups: true)Return
[{"0":"first group","1":"first","number":"first"},{"0":"second group","1":"second","number":"second"}]Learn more about the parse_regex_all function
parse_ruby_hash
Parses the
value as ruby hash.Parse ruby hash
Source
parse_ruby_hash!(s'{ "test" => "value", "testNum" => 0.2, "testObj" => { "testBool" => true, "testNull" => nil } }')Return
{
"test": "value",
"testNum": 0.2,
"testObj": {
"testBool": true,
"testNull": null
}
}Learn more about the parse_ruby_hash function
parse_syslog
Parses the
value in Syslog format.Parse Syslog log (5424)
Source
parse_syslog!(
s'<13>1 2020-03-13T20:45:38.119Z dynamicwireless.name non 2426 ID931 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"] Try to override the THX port, maybe it will reboot the neural interface!'
)Return
{
"appname": "non",
"exampleSDID@32473": {
"eventID": "1011",
"eventSource": "Application",
"iut": "3"
},
"facility": "user",
"hostname": "dynamicwireless.name",
"message": "Try to override the THX port, maybe it will reboot the neural interface!",
"msgid": "ID931",
"procid": 2426,
"severity": "notice",
"timestamp": "2020-03-13T20:45:38.119Z",
"version": 1
}Learn more about the parse_syslog function
parse_timestamp
Parse timestamp
Source
parse_timestamp!("10-Oct-2020 16:00+00:00", format: "%v %R %:z")Return
2020-10-10T16:00:00ZLearn more about the parse_timestamp function
parse_tokens
Parses the value in token format. A token is considered to be one of the following:
- A word surrounded by whitespace.
- Text delimited by double quotes:
"..". Quotes can be included in the token if they are escaped by a backslash (\). - Text delimited by square brackets:
[..]. Closing square brackets can be included in the token if they are escaped by a backslash (\).
Parse tokens
Source
parse_tokens(
"A sentence \"with \\\"a\\\" sentence inside\" and [some brackets]"
)Return
["A","sentence","with \\\"a\\\" sentence inside","and","some brackets"]Learn more about the parse_tokens function
parse_url
Parses the
value in URL format.Parse URL
Source
parse_url!("ftp://foo:bar@example.com:4343/foobar?hello=world#123")Return
{
"fragment": "123",
"host": "example.com",
"password": "bar",
"path": "/foobar",
"port": 4343,
"query": {
"hello": "world"
},
"scheme": "ftp",
"username": "foo"
}Learn more about the parse_url function
Parse URL with default port
Source
parse_url!("https://example.com", default_known_ports: true)Return
{
"fragment": null,
"host": "example.com",
"password": "",
"path": "/",
"port": 443,
"query": {},
"scheme": "https",
"username": ""
}Learn more about the parse_url function
Parse URL with internationalized domain name
Source
parse_url!("https://www.café.com")Return
{
"fragment": null,
"host": "www.xn--caf-dma.com",
"password": "",
"path": "/",
"port": null,
"query": {},
"scheme": "https",
"username": ""
}Learn more about the parse_url function
Parse URL with mixed case internationalized domain name
Source
parse_url!("https://www.CAFé.com")Return
{
"fragment": null,
"host": "www.xn--caf-dma.com",
"password": "",
"path": "/",
"port": null,
"query": {},
"scheme": "https",
"username": ""
}Learn more about the parse_url function
parse_user_agent
Parses the
value as a user agent string, which has a loosely defined format
so this parser only provides best effort guarantee.Fast mode
Source
parse_user_agent(
"Mozilla Firefox 1.0.1 Mozilla/5.0 (X11; U; Linux i686; de-DE; rv:1.7.6) Gecko/20050223 Firefox/1.0.1"
)Return
{
"browser": {
"family": "Firefox",
"version": "1.0.1"
},
"device": {
"category": "pc"
},
"os": {
"family": "Linux",
"version": null
}
}Learn more about the parse_user_agent function
Reliable mode
Source
parse_user_agent(
"Mozilla/4.0 (compatible; MSIE 7.66; Windows NT 5.1; SV1; .NET CLR 1.1.4322)",
mode: "reliable"
)Return
{
"browser": {
"family": "Internet Explorer",
"version": "7.66"
},
"device": {
"category": "pc"
},
"os": {
"family": "Windows XP",
"version": "NT 5.1"
}
}Learn more about the parse_user_agent function
Enriched mode
Source
parse_user_agent(
"Opera/9.80 (J2ME/MIDP; Opera Mini/4.3.24214; iPhone; CPU iPhone OS 4_2_1 like Mac OS X; AppleWebKit/24.783; U; en) Presto/2.5.25 Version/10.54",
mode: "enriched"
)Return
{
"browser": {
"family": "Opera Mini",
"major": "4",
"minor": "3",
"patch": "24214",
"version": "10.54"
},
"device": {
"brand": "Apple",
"category": "smartphone",
"family": "iPhone",
"model": "iPhone"
},
"os": {
"family": "iOS",
"major": "4",
"minor": "2",
"patch": "1",
"patch_minor": null,
"version": "4.2.1"
}
}Learn more about the parse_user_agent function
parse_xml
Parses the
value as XML.Parse XML
Source
value = s'<book category="CHILDREN"><title lang="en">Harry Potter</title><author>J K. Rowling</author><year>2005</year></book>';
parse_xml!(value, text_key: "value", parse_number: false)Return
{
"book": {
"@category": "CHILDREN",
"author": "J K. Rowling",
"title": {
"@lang": "en",
"value": "Harry Potter"
},
"year": "2005"
}
}Learn more about the parse_xml function
Random examples
random_bytes
A cryptographically secure random number generator. Returns a string value containing the number of
random bytes requested.
Generate random base 64 encoded bytes
Source
encode_base64(random_bytes(16))Return
LNu0BBgUbh7XAlXbjSOomQ==Learn more about the random_bytes function
random_float
Returns a random float between [min, max).
Random float from 0.0 to 10.0, not including 10.0
Source
f = random_float(0.0, 10.0)
f >= 0 && f < 10Return
trueLearn more about the random_float function
random_int
Returns a random integer between [min, max).
Random integer from 0 to 10, not including 10
Source
i = random_int(0, 10)
i >= 0 && i < 10Return
trueLearn more about the random_int function
uuid_from_friendly_id
Convert a Friendly ID (base62 encoding a 128-bit word) to a UUID.
Convert a Friendly ID to a UUID
Source
uuid_from_friendly_id!("3s87yEvnmkiPBMHsj8bwwc")Return
7f41deed-d5e2-8b5e-7a13-ab4ff93cfad2Learn more about the uuid_from_friendly_id function
uuid_v4
Generates a random UUIDv4 string.
Create a UUIDv4
Source
uuid_v4()Return
1d262f4f-199b-458d-879f-05fd0a5f0683Learn more about the uuid_v4 function
uuid_v7
Generates a random UUIDv7 string.
Create a UUIDv7 with implicit now()
Source
uuid_v7()Return
06338364-8305-7b74-8000-de4963503139Learn more about the uuid_v7 function
Create a UUIDv7 with explicit now()
Source
uuid_v7(now())Return
018e29b3-0bea-7f78-8af3-d32ccb1b93c1Learn more about the uuid_v7 function
Create a UUIDv7 with custom timestamp
Source
uuid_v7(t'2020-12-30T22:20:53.824727Z')Return
0176b5bd-5d19-7394-bb60-c21028c6152bLearn more about the uuid_v7 function
String examples
camelcase
Takes the
value string, and turns it into camelCase. Optionally, you can
pass in the existing case of the function, or else an attempt is made to determine the case automatically.community_id
Generates an ID based on the Community ID Spec.
TCP
Source
community_id!(source_ip: "1.2.3.4", destination_ip: "5.6.7.8", source_port: 1122, destination_port: 3344, protocol: 6)Return
1:wCb3OG7yAFWelaUydu0D+125CLM=Learn more about the community_id function
contains_all
Determines whether the
value string contains all the specified substrings.String contains all
Source
contains_all("The Needle In The Haystack", ["Needle", "Haystack"])Return
trueLearn more about the contains_all function
String contains all (case sensitive)
Source
contains_all("the NEEDLE in the haystack", ["needle", "haystack"])Learn more about the contains_all function
downcase
Downcases the
value string, where downcase is defined according to the
Unicode Derived Core Property Lowercase.Downcase a string
Source
downcase("Hello, World!")Return
hello, world!Learn more about the downcase function
ends_with
Determines whether the
value string ends with the specified substring.String ends with (case sensitive)
Source
ends_with("The Needle In The Haystack", "The Haystack")Return
trueLearn more about the ends_with function
String ends with (case insensitive)
Source
ends_with("The Needle In The Haystack", "the haystack", case_sensitive: false)Return
trueLearn more about the ends_with function
find
Determines from left to right the start position of the first found element in
value
that matches pattern. Returns -1 if not found.join
Joins each string in the
value array into a single string, with items optionally separated from one another
by a separator.kebabcase
Takes the
value string, and turns it into kebab-case. Optionally, you can
pass in the existing case of the function, or else we will try to figure out the case automatically.match_any
Determines whether
value matches any of the given patterns. All
patterns are checked in a single pass over the target string, giving this
function a potential performance advantage over the multiple calls
in the match function.Regex match on a string
Source
match_any("I'm a little teapot", [r'frying pan', r'teapot'])Return
trueLearn more about the match_any function
parse_float
Parses the string
value representing a floating point number in base 10 to a float.Parse negative integer
Source
parse_float!("42.38")Return
42.38Learn more about the parse_float function
pascalcase
Takes the
value string, and turns it into PascalCase. Optionally, you can
pass in the existing case of the function, or else we will try to figure out the case automatically.PascalCase a string
Source
pascalcase("input-string")Return
InputStringLearn more about the pascalcase function
PascalCase a string
Source
pascalcase("input-string", "kebab-case")Return
InputStringLearn more about the pascalcase function
redact
Redact sensitive data in value such as:
- US social security card numbers
- Other forms of personally identifiable information with custom patterns
This can help achieve compliance by ensuring sensitive data does not leave your network.
Replace text using a regex
Source
redact("my id is 123456", filters: [r'\d+'])Return
my id is [REDACTED]Learn more about the redact function
Replace us social security numbers in any field
Source
redact({ "name": "John Doe", "ssn": "123-12-1234"}, filters: ["us_social_security_number"])Return
{
"name": "John Doe",
"ssn": "[REDACTED]"
}Learn more about the redact function
Replace with custom text
Source
redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "text", "replacement": "***"})Return
my id is ***Learn more about the redact function
Replace with SHA-2 hash
Source
redact("my id is 123456", filters: [r'\d+'], redactor: "sha2")Return
my id is GEtTedW1p6tC094dDKH+3B8P+xSnZz69AmpjaXRd63I=Learn more about the redact function
Replace with SHA-3 hash
Source
redact("my id is 123456", filters: [r'\d+'], redactor: "sha3")Return
my id is ZNCdmTDI7PeeUTFnpYjLdUObdizo+bIupZdl8yqnTKGdLx6X3JIqPUlUWUoFBikX+yTR+OcvLtAqWO11NPlNJw==Learn more about the redact function
Replace with SHA-256 hash using hex encoding
Source
redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "sha2", "variant": "SHA-256", "encoding": "base16"})Return
my id is 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92Learn more about the redact function
replace
Replaces all matching instances of pattern in value.
The pattern argument accepts regular expression capture groups.
Note when using capture groups:
- You will need to escape the
$by using$$to avoid Vector interpreting it as an environment variable when loading configuration - If you want a literal
$in the replacement pattern, you will also need to escape this with$$. When combined with environment variable interpolation in config files this means you will need to use$$$$to have a literal$in the replacement pattern.
Replace literal text
Source
replace("Apples and Bananas", "and", "not")Return
Apples not BananasLearn more about the replace function
Replace using regular expression
Source
replace("Apples and Bananas", r'(?i)bananas', "Pineapples")Return
Apples and PineapplesLearn more about the replace function
Replace first instance
Source
replace("Bananas and Bananas", "Bananas", "Pineapples", count: 1)Return
Pineapples and BananasLearn more about the replace function
Replace with capture groups (Note: Use $$num in config files)
Source
replace("foo123bar", r'foo(?P<num>\d+)bar', "$num")Return
123Learn more about the replace function
replace_with
Replaces all matching instances of pattern using a closure.
The pattern argument accepts a regular expression that can use capture groups.
The function uses the function closure syntax to compute the replacement values.
The closure takes a single parameter, which is an array, where the first item is always
present and contains the entire string that matched pattern. The items from index one on
contain the capture groups of the corresponding index. If a capture group is optional, the
value may be null if it didn’t match.
The value returned by the closure must be a string and will replace the section of the input that was matched.
This returns a new string with the replacements, the original string is not mutated.
Capitalize words
Source
replace_with("apples and bananas", r'\b(\w)(\w*)') -> |match| {
upcase!(match.captures[0]) + string!(match.captures[1])
}Return
Apples And BananasLearn more about the replace_with function
Replace with hash
Source
replace_with("email from test@example.com", r'\w+@example.com') -> |match| {
sha2(match.string, variant: "SHA-512/224")
}Return
email from adf6e1bc4415d24912bd93072ad34ef825a7b6eb3bf53f68def1fc17Learn more about the replace_with function
Replace first instance
Source
replace_with("Apples and Apples", r'(?i)apples|cones', count: 1) -> |match| {
"Pine" + downcase(match.string)
}Return
Pineapples and ApplesLearn more about the replace_with function
Named capture group
Source
replace_with("level=error A message", r'level=(?P<level>\w+)') -> |match| {
lvl = upcase!(match.level)
"[{{lvl}}]"
}Return
[ERROR] A messageLearn more about the replace_with function
screamingsnakecase
Takes the
value string, and turns it into SCREAMING_SNAKE case. Optionally, you can
pass in the existing case of the function, or else we will try to figure out the case automatically.SCREAMING_SNAKE a string
Source
screamingsnakecase("input-string")Return
INPUT_STRINGLearn more about the screamingsnakecase function
SCREAMING_SNAKE a string
Source
screamingsnakecase("input-string", "kebab-case")Return
INPUT_STRINGLearn more about the screamingsnakecase function
sieve
Keeps only matches of pattern in value.
This can be used to define patterns that are allowed in the string and remove everything else.
Sieve with regex
Source
sieve("test123%456.فوائد.net.", r'[a-z0-9.]')Return
test123456..net.Learn more about the sieve function
Custom replacements
Source
sieve("test123%456.فوائد.net.", r'[a-z.0-9]', replace_single: "X", replace_repeated: "<REMOVED>")Return
test123X456.<REMOVED>.net.Learn more about the sieve function
slice
Returns a slice of value between the start and end positions.
If the start and end parameters are negative, they refer to positions counting from the right of the
string or array. If end refers to a position that is greater than the length of the string or array,
a slice up to the end of the string or array is returned.
Slice a string (positive index)
Source
slice!("Supercalifragilisticexpialidocious", start: 5, end: 13)Return
califragLearn more about the slice function
Slice a string (negative index)
Source
slice!("Supercalifragilisticexpialidocious", start: 5, end: -14)Return
califragilisticLearn more about the slice function
snakecase
Takes the
value string, and turns it into snake-case. Optionally, you can
pass in the existing case of the function, or else we will try to figure out the case automatically.split
Splits the
value string using pattern.Split a string (no limit)
Source
split("apples and pears and bananas", " and ")Return
["apples","pears","bananas"]Learn more about the split function
Split a string (with a limit)
Source
split("apples and pears and bananas", " and ", limit: 2)Return
["apples","pears and bananas"]Learn more about the split function
starts_with
Determines whether
value begins with substring.String starts with (case sensitive)
Source
starts_with("The Needle In The Haystack", "The Needle")Return
trueLearn more about the starts_with function
String starts with (case insensitive)
Source
starts_with("The Needle In The Haystack", "the needle", case_sensitive: false)Return
trueLearn more about the starts_with function
strip_ansi_escape_codes
Strips ANSI escape codes from
value.Strip ANSI escape codes
Source
strip_ansi_escape_codes("\e[46mfoo\e[0m bar")Return
foo barLearn more about the strip_ansi_escape_codes function
strip_whitespace
Strips whitespace from the start and end of
value, where whitespace is defined by the Unicode
White_Space property.Strip whitespace
Source
strip_whitespace(" A sentence. ")Return
A sentence.Learn more about the strip_whitespace function
truncate
Truncates the
value string up to the limit number of characters.Truncate a string
Source
truncate("A rather long sentence.", limit: 11, suffix: "...")Return
A rather lo...Learn more about the truncate function
Truncate a string
Source
truncate("A rather long sentence.", limit: 11, suffix: "[TRUNCATED]")Return
A rather lo[TRUNCATED]Learn more about the truncate function
upcase
Upcases
value, where upcase is defined according to the Unicode Derived Core Property
Uppercase.Upcase a string
Source
upcase("Hello, World!")Return
HELLO, WORLD!Learn more about the upcase function
System examples
get_env_var
Returns the value of the environment variable specified by
name.Get an environment variable
Source
get_env_var!("HOME")Return
/rootLearn more about the get_env_var function
get_timezone_name
Returns the name of the timezone in the Vector configuration (see
global configuration options).
If the configuration is set to
local, then it attempts to
determine the name of the timezone from the host OS. If this
is not possible, then it returns the fixed offset of the
local timezone for the current time in the format "[+-]HH:MM",
for example, "+02:00".Get the IANA name of Vector’s timezone
Source
.vector_timezone = get_timezone_name!()Learn more about the get_timezone_name function
Timestamp examples
format_timestamp
Formats
value into a string representation of the timestamp.Format a timestamp (ISO8601/RFC 3339)
Source
format_timestamp!(t'2020-10-21T16:00:00Z', format: "%+")Return
2020-10-21T16:00:00+00:00Learn more about the format_timestamp function
Format a timestamp (custom)
Source
format_timestamp!(t'2020-10-21T16:00:00Z', format: "%v %R")Return
21-Oct-2020 16:00Learn more about the format_timestamp function
now
Returns the current timestamp in the UTC timezone with nanosecond precision.
Generate a current timestamp
Source
now()Return
2021-03-04T10:51:15.928937ZLearn more about the now function
Type examples
array
Returns
value if it is an array, otherwise returns an error. This enables the type checker to guarantee that the
returned value is an array and can be used in any function that expects an array.bool
Returns
value if it is a Boolean, otherwise returns an error. This enables the type checker to guarantee that the
returned value is a Boolean and can be used in any function that expects a Boolean.float
Returns
value if it is a float, otherwise returns an error. This enables the type checker to guarantee that the
returned value is a float and can be used in any function that expects a float.int
Returns
value if it is an integer, otherwise returns an error. This enables the type checker to guarantee that the
returned value is an integer and can be used in any function that expects an integer.is_nullish
Determines whether
value is nullish, where nullish denotes the absence of a
meaningful value.Null detection (blank string)
Source
is_nullish("")Return
trueLearn more about the is_nullish function
Null detection (dash string)
Source
is_nullish("-")Return
trueLearn more about the is_nullish function
is_timestamp
Check if
value’s type is a timestamp.Valid timestamp
Source
is_timestamp(t'2021-03-26T16:00:00Z')Return
trueLearn more about the is_timestamp function
object
Returns
value if it is an object, otherwise returns an error. This enables the type checker to guarantee that the
returned value is an object and can be used in any function that expects an object.Declare an object type
Source
object!(.value)Return
{
"field1": "value1",
"field2": "value2"
}Learn more about the object function
string
Returns
value if it is a string, otherwise returns an error. This enables the type checker to guarantee that the
returned value is a string and can be used in any function that expects a string.Declare a string type
Source
string!(.message)Return
{"field": "value"}Learn more about the string function
tag_types_externally
Adds type information to all (nested) scalar values in the provided value.
The type information is added externally, meaning that value has the form of "type": value after this
transformation.
Tag types externally (scalar)
Source
tag_types_externally(123)Return
{
"integer": 123
}Learn more about the tag_types_externally function
Tag types externally (object)
Source
tag_types_externally({
"message": "Hello world",
"request": {
"duration_ms": 67.9
}
})Return
{
"message": {
"string": "Hello world"
},
"request": {
"duration_ms": {
"float": 67.9
}
}
}Learn more about the tag_types_externally function
Tag types externally (array)
Source
tag_types_externally(["foo", "bar"])Return
[{"string":"foo"},{"string":"bar"}]Learn more about the tag_types_externally function
Tag types externally (null)
Source
tag_types_externally(null)Learn more about the tag_types_externally function
timestamp
Returns
value if it is a timestamp, otherwise returns an error. This enables the type checker to guarantee that
the returned value is a timestamp and can be used in any function that expects a timestamp.Declare a timestamp type
Source
timestamp(t'2020-10-10T16:00:00Z')Return
2020-10-10T16:00:00ZLearn more about the timestamp function