Lua
Modify event data using the Lua programming language
Warnings
lua transform is ~60% slower than the remap transform; we
recommend that you use the remap transform whenever possible. The lua transform is
designed solely for edge cases not covered by the remap transform and not as a go-to option. If the
remap transform doesn’t cover your use case, please open an issue and let
us know.Configuration
Example configurations
{
  "transforms": {
    "my_transform_id": {
      "type": "lua",
      "inputs": [
        "my-source-or-transform-id"
      ],
      "version": "1"
    }
  }
}[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "1"
transforms:
  my_transform_id:
    type: lua
    inputs:
      - my-source-or-transform-id
    version: "1"
{
  "transforms": {
    "my_transform_id": {
      "type": "lua",
      "inputs": [
        "my-source-or-transform-id"
      ],
      "metric_tag_values": "single",
      "search_dirs": [
        "/etc/vector/lua"
      ],
      "source": "function init()\n\tcount = 0\nend\n\nfunction process()\n\tcount = count + 1\nend\n\nfunction timer_handler(emit)\n\temit(make_counter(counter))\n\tcounter = 0\nend\n\nfunction shutdown(emit)\n\temit(make_counter(counter))\nend\n\nfunction make_counter(value)\n\treturn metric = {\n\t\tname = \"event_counter\",\n\t\tkind = \"incremental\",\n\t\ttimestamp = os.date(\"!*t\"),\n\t\tcounter = {\n\t\t\tvalue = value\n\t\t}\n \t}\nend",
      "timers": {
        "handler": "timer_handler",
        "interval_seconds": null
      },
      "version": "1"
    }
  }
}[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
metric_tag_values = "single"
search_dirs = [ "/etc/vector/lua" ]
source = """
function init()
\tcount = 0
end
function process()
\tcount = count + 1
end
function timer_handler(emit)
\temit(make_counter(counter))
\tcounter = 0
end
function shutdown(emit)
\temit(make_counter(counter))
end
function make_counter(value)
\treturn metric = {
\t\tname = "event_counter",
\t\tkind = "incremental",
\t\ttimestamp = os.date("!*t"),
\t\tcounter = {
\t\t\tvalue = value
\t\t}
 \t}
end"""
version = "1"
  [transforms.my_transform_id.timers]
  handler = "timer_handler"
transforms:
  my_transform_id:
    type: lua
    inputs:
      - my-source-or-transform-id
    metric_tag_values: single
    search_dirs:
      - /etc/vector/lua
    source: |-
      function init()
      	count = 0
      end
      function process()
      	count = count + 1
      end
      function timer_handler(emit)
      	emit(make_counter(counter))
      	counter = 0
      end
      function shutdown(emit)
      	emit(make_counter(counter))
      end
      function make_counter(value)
      	return metric = {
      		name = "event_counter",
      		kind = "incremental",
      		timestamp = os.date("!*t"),
      		counter = {
      			value = value
      		}
       	}
      end      
    timers:
      handler: timer_handler
      interval_seconds: null
    version: "1"
graph
optional objectExtra graph configuration
Configure output for component when generated with graph command
graph.node_attributes
optional objectNode attributes to add to this component’s node in resulting graph
They are added to the node as provided
graph.node_attributes.*
required string literalhooks
required objectLifecycle hooks.
These hooks can be set to perform additional processing during the lifecycle of the transform.
hooks.init
optional string literalThe function called when the first event comes in, before hooks.process is called.
It can produce new events using the emit function.
This can either be inline Lua that defines a closure to use, or the name of the Lua function to call. In both
cases, the closure/function takes a single parameter, emit, which is a reference to a function for emitting events.
hooks.process
required string literalThe function called for each incoming event.
It can produce new events using the emit function.
This can either be inline Lua that defines a closure to use, or the name of the Lua function to call. In both
cases, the closure/function takes two parameters. The first parameter, event, is the event being processed,
while the second parameter, emit, is a reference to a function for emitting events.
hooks.shutdown
optional string literalThe function called when the transform is stopped.
It can produce new events using the emit function.
This can either be inline Lua that defines a closure to use, or the name of the Lua function to call. In both
cases, the closure/function takes a single parameter, emit, which is a reference to a function for emitting events.
inputs
required [string]A list of upstream source or transform IDs.
Wildcards (*) are supported.
See configuration for more info.
metric_tag_values
optional string literal enumWhen set to single, metric tag values are exposed as single strings, the
same as they were before this config option. Tags with multiple values show the last assigned value, and null values
are ignored.
When set to full, all metric tags are exposed as arrays of either string or null
values.
| Option | Description | 
|---|---|
| full | All tags are exposed as arrays of either string or null values. | 
| single | Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored. | 
singlesearch_dirs
optional [string]A list of directories to search when loading a Lua file via the require function.
If not specified, the modules are looked up in the configuration directories.
source
optional string literalThe Lua program to initialize the transform with.
The program can be used to import external dependencies, as well as define the functions used for the various lifecycle hooks. However, it’s not strictly required, as the lifecycle hooks can be configured directly with inline Lua source for each respective hook.
"function init()\n\tcount = 0\nend\n\nfunction process()\n\tcount = count + 1\nend\n\nfunction timer_handler(emit)\n\temit(make_counter(counter))\n\tcounter = 0\nend\n\nfunction shutdown(emit)\n\temit(make_counter(counter))\nend\n\nfunction make_counter(value)\n\treturn metric = {\n\t\tname = \"event_counter\",\n\t\tkind = \"incremental\",\n\t\ttimestamp = os.date(\"!*t\"),\n\t\tcounter = {\n\t\t\tvalue = value\n\t\t}\n \t}\nend""-- external file with hooks and timers defined\nrequire('custom_module')"timers
optional [object]version
required string literal enumTransform API version.
Specifying this version ensures that backward compatibility is not broken.
| Option | Description | 
|---|---|
| 1 | Lua transform API version 1. This version is deprecated and will be removed in a future version. | 
| 2 | Lua transform API version 2. | 
Outputs
<component_id>
Telemetry
Metrics
linkcomponent_discarded_events_total
counterfilter transform, or false if due to an error.component_errors_total
countercomponent_received_event_bytes_total
countercomponent_received_events_count
histogramA histogram of the number of events passed in each internal batch in Vector’s internal topology.
Note that this is separate than sink-level batching. It is mostly useful for low level debugging performance issues in Vector due to small internal batches.
component_received_events_total
countercomponent_sent_event_bytes_total
countercomponent_sent_events_total
counterlua_memory_used_bytes
gaugeutilization
gaugeExamples
Add, rename, and remove log fields
Given this event...{
  "log": {
    "field_to_remove": "remove me",
    "field_to_rename": "old value"
  }
}transforms:
  my_transform_id:
    type: lua
    inputs:
      - my-source-or-transform-id
    version: "2"
    hooks:
      process: |-
        function (event, emit)
        	-- Add root level field
        	event.log.field = "new value"
        	-- Add nested field
        	event.log.nested = {}
        	event.log.nested.field = "nested value"
        	-- Rename field
        	event.log.renamed_field = event.log.field_to_rename
        	event.log.field_to_rename = nil
        	-- Remove fields
        	event.log.field_to_remove = nil
        	emit(event)
        end        
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
  [transforms.my_transform_id.hooks]
  process = """
function (event, emit)
\t-- Add root level field
\tevent.log.field = "new value"
\t-- Add nested field
\tevent.log.nested = {}
\tevent.log.nested.field = "nested value"
\t-- Rename field
\tevent.log.renamed_field = event.log.field_to_rename
\tevent.log.field_to_rename = nil
\t-- Remove fields
\tevent.log.field_to_remove = nil
\temit(event)
end"""
{
  "transforms": {
    "my_transform_id": {
      "type": "lua",
      "inputs": [
        "my-source-or-transform-id"
      ],
      "version": "2",
      "hooks": {
        "process": "function (event, emit)\n\t-- Add root level field\n\tevent.log.field = \"new value\"\n\t-- Add nested field\n\tevent.log.nested = {}\n\tevent.log.nested.field = \"nested value\"\n\t-- Rename field\n\tevent.log.renamed_field = event.log.field_to_rename\n\tevent.log.field_to_rename = nil\n\t-- Remove fields\n\tevent.log.field_to_remove = nil\n\temit(event)\nend"
      }
    }
  }
}{
  "field": "new value",
  "nested": {
    "field": "nested value"
  },
  "renamed_field": "old value"
}Add, rename, remove metric tags
Given this event...{
  "metric": {
    "counter": {
      "value": 2
    },
    "kind": "incremental",
    "name": "logins",
    "tags": {
      "tag_to_remove": "remove me",
      "tag_to_rename": "old value"
    }
  }
}transforms:
  my_transform_id:
    type: lua
    inputs:
      - my-source-or-transform-id
    version: "2"
    hooks:
      process: |-
        function (event, emit)
        	-- Add tag
        	event.metric.tags.tag = "new value"
        	-- Rename tag
        	event.metric.tags.renamed_tag = event.log.tag_to_rename
        	event.metric.tags.tag_to_rename = nil
        	-- Remove tag
        	event.metric.tags.tag_to_remove = nil
        	emit(event)
        end        
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
  [transforms.my_transform_id.hooks]
  process = """
function (event, emit)
\t-- Add tag
\tevent.metric.tags.tag = "new value"
\t-- Rename tag
\tevent.metric.tags.renamed_tag = event.log.tag_to_rename
\tevent.metric.tags.tag_to_rename = nil
\t-- Remove tag
\tevent.metric.tags.tag_to_remove = nil
\temit(event)
end"""
{
  "transforms": {
    "my_transform_id": {
      "type": "lua",
      "inputs": [
        "my-source-or-transform-id"
      ],
      "version": "2",
      "hooks": {
        "process": "function (event, emit)\n\t-- Add tag\n\tevent.metric.tags.tag = \"new value\"\n\t-- Rename tag\n\tevent.metric.tags.renamed_tag = event.log.tag_to_rename\n\tevent.metric.tags.tag_to_rename = nil\n\t-- Remove tag\n\tevent.metric.tags.tag_to_remove = nil\n\temit(event)\nend"
      }
    }
  }
}{
  "counter": {
    "value": 2
  },
  "kind": "incremental",
  "name": "logins",
  "tags": {
    "renamed_tag": "old value",
    "tag": "new value"
  }
}Drop an event
Given this event...{
  "log": {
    "field_to_remove": "remove me",
    "field_to_rename": "old value"
  }
}transforms:
  my_transform_id:
    type: lua
    inputs:
      - my-source-or-transform-id
    version: "2"
    hooks:
      process: |-
        function (event, emit)
        	-- Drop event entirely by not calling the `emit` function
        end        
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
  [transforms.my_transform_id.hooks]
  process = """
function (event, emit)
\t-- Drop event entirely by not calling the `emit` function
end"""
{
  "transforms": {
    "my_transform_id": {
      "type": "lua",
      "inputs": [
        "my-source-or-transform-id"
      ],
      "version": "2",
      "hooks": {
        "process": "function (event, emit)\n\t-- Drop event entirely by not calling the `emit` function\nend"
      }
    }
  }
}Iterate over log fields
Given this event...{
  "log": {
    "value_to_keep": "keep",
    "value_to_remove": "-"
  }
}transforms:
  my_transform_id:
    type: lua
    inputs:
      - my-source-or-transform-id
    version: "2"
    hooks:
      process: |-
        function (event, emit)
        	-- Remove all fields where the value is "-"
        	for f, v in pairs(event) do
        		if v == "-" then
        			event[f] = nil
        		end
        	end
        	emit(event)
        end        
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
  [transforms.my_transform_id.hooks]
  process = """
function (event, emit)
\t-- Remove all fields where the value is "-"
\tfor f, v in pairs(event) do
\t\tif v == "-" then
\t\t\tevent[f] = nil
\t\tend
\tend
\temit(event)
end"""
{
  "transforms": {
    "my_transform_id": {
      "type": "lua",
      "inputs": [
        "my-source-or-transform-id"
      ],
      "version": "2",
      "hooks": {
        "process": "function (event, emit)\n\t-- Remove all fields where the value is \"-\"\n\tfor f, v in pairs(event) do\n\t\tif v == \"-\" then\n\t\t\tevent[f] = nil\n\t\tend\n\tend\n\temit(event)\nend"
      }
    }
  }
}{
  "value_to_keep": "keep"
}Parse timestamps
Given this event...{
  "log": {
    "timestamp_string": "2020-04-07 06:26:02.643"
  }
}transforms:
  my_transform_id:
    type: lua
    inputs:
      - my-source-or-transform-id
    version: "2"
    hooks:
      process: process
    source: >-2
        timestamp_pattern = "(%d%d%d%d)[-](%d%d)[-](%d%d) (%d%d):(%d%d):(%d%d).?(%d*)"
        function parse_timestamp(str)
      	local year, month, day, hour, min, sec, millis = string.match(str, timestamp_pattern)
      	local ms = 0
      	if millis and millis ~= "" then
      		ms = tonumber(millis)
      	end
      	return {
      		year    = tonumber(year),
      		month   = tonumber(month),
      		day     = tonumber(day),
      		hour    = tonumber(hour),
      		min     = tonumber(min),
      		sec     = tonumber(sec),
      		nanosec = ms * 1000000
      	}
        end
        function process(event, emit)
      	event.log.timestamp = parse_timestamp(event.log.timestamp_string)
      	emit(event)
        end
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
source = """
  timestamp_pattern = "(%d%d%d%d)[-](%d%d)[-](%d%d) (%d%d):(%d%d):(%d%d).?(%d*)"
  function parse_timestamp(str)
\tlocal year, month, day, hour, min, sec, millis = string.match(str, timestamp_pattern)
\tlocal ms = 0
\tif millis and millis ~= "" then
\t\tms = tonumber(millis)
\tend
\treturn {
\t\tyear    = tonumber(year),
\t\tmonth   = tonumber(month),
\t\tday     = tonumber(day),
\t\thour    = tonumber(hour),
\t\tmin     = tonumber(min),
\t\tsec     = tonumber(sec),
\t\tnanosec = ms * 1000000
\t}
  end
  function process(event, emit)
\tevent.log.timestamp = parse_timestamp(event.log.timestamp_string)
\temit(event)
  end"""
  [transforms.my_transform_id.hooks]
  process = "process"
{
  "transforms": {
    "my_transform_id": {
      "type": "lua",
      "inputs": [
        "my-source-or-transform-id"
      ],
      "version": "2",
      "hooks": {
        "process": "process"
      },
      "source": "  timestamp_pattern = \"(%d%d%d%d)[-](%d%d)[-](%d%d) (%d%d):(%d%d):(%d%d).?(%d*)\"\n  function parse_timestamp(str)\n\tlocal year, month, day, hour, min, sec, millis = string.match(str, timestamp_pattern)\n\tlocal ms = 0\n\tif millis and millis ~= \"\" then\n\t\tms = tonumber(millis)\n\tend\n\treturn {\n\t\tyear    = tonumber(year),\n\t\tmonth   = tonumber(month),\n\t\tday     = tonumber(day),\n\t\thour    = tonumber(hour),\n\t\tmin     = tonumber(min),\n\t\tsec     = tonumber(sec),\n\t\tnanosec = ms * 1000000\n\t}\n  end\n  function process(event, emit)\n\tevent.log.timestamp = parse_timestamp(event.log.timestamp_string)\n\temit(event)\n  end"
    }
  }
}{
  "timestamp": "2020-04-07 06:26:02.643",
  "timestamp_string": "2020-04-07 06:26:02.643"
}Count the number of logs
Given this event...{
  "log": {}
}transforms:
  my_transform_id:
    type: lua
    inputs:
      - my-source-or-transform-id
    version: "2"
    hooks:
      init: init
      process: process
      shutdown: shutdown
    timers:
      - interval_seconds: 5
        handler: timer_handler
    source: |-
      function init()
      	count = 0
      end
      function process()
      	count = count + 1
      end
      function timer_handler(emit)
      	emit(make_counter(count))
      	count = 0
      end
      function shutdown(emit)
      	emit(make_counter(count))
      end
      function make_counter(value)
      	return metric = {
      		name = "event_counter",
      		kind = "incremental",
      		timestamp = os.date("!*t"),
      		counter = {
      			value = value
      		}
      	}
      end      
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
source = """
function init()
\tcount = 0
end
function process()
\tcount = count + 1
end
function timer_handler(emit)
\temit(make_counter(count))
\tcount = 0
end
function shutdown(emit)
\temit(make_counter(count))
end
function make_counter(value)
\treturn metric = {
\t\tname = "event_counter",
\t\tkind = "incremental",
\t\ttimestamp = os.date("!*t"),
\t\tcounter = {
\t\t\tvalue = value
\t\t}
\t}
end"""
  [transforms.my_transform_id.hooks]
  init = "init"
  process = "process"
  shutdown = "shutdown"
  [[transforms.my_transform_id.timers]]
  interval_seconds = 5
  handler = "timer_handler"
{
  "transforms": {
    "my_transform_id": {
      "type": "lua",
      "inputs": [
        "my-source-or-transform-id"
      ],
      "version": "2",
      "hooks": {
        "init": "init",
        "process": "process",
        "shutdown": "shutdown"
      },
      "timers": [
        {
          "interval_seconds": 5,
          "handler": "timer_handler"
        }
      ],
      "source": "function init()\n\tcount = 0\nend\nfunction process()\n\tcount = count + 1\nend\nfunction timer_handler(emit)\n\temit(make_counter(count))\n\tcount = 0\nend\nfunction shutdown(emit)\n\temit(make_counter(count))\nend\nfunction make_counter(value)\n\treturn metric = {\n\t\tname = \"event_counter\",\n\t\tkind = \"incremental\",\n\t\ttimestamp = os.date(\"!*t\"),\n\t\tcounter = {\n\t\t\tvalue = value\n\t\t}\n\t}\nend"
    }
  }
}{
  "counter": {
    "value": 1
  },
  "kind": "incremental",
  "name": "event_counter",
  "tags": {
    "renamed_tag": "old value",
    "tag": "new value"
  }
}How it works
Event Data Model
process hook takes an event as its first argument.
Events are represented as tables in Lua
and follow Vector’s data model exactly. Please refer to
Vector’s data model reference for the event
schema. How Vector’s types map to Lua’s type are covered below.Type Mappings
The correspondence between Vector’s data types and Lua data type is summarized by the following table:
| Vector Type | Lua Type | Comment | 
|---|---|---|
| String | string | |
| Integer | integer | |
| Float | number | |
| Boolean | boolean | |
| Timestamp | table | There is no dedicated timestamp type in Lua. Timestamps are represented as tables using the convention defined by os.dateandos.time. The table representation of a timestamp contains the fieldsyear,month,day,hour,min,sec,nanosec,yday,wday, andisdst. If such a table is passed from Lua to Vector, the fieldsyday,wday, andisdstcan be omitted. In addition to theos.timerepresentation, Vector supports sub-second resolution with ananosecfield in the table. | 
| Null | empty string | In Lua setting the value of a table field to nilmeans deletion of this field. In addition, the length operator#does not work in the expected way with sequences containing nulls. Because of thatNullvalues are encoded as empty strings. | 
| Map | table | |
| Array | sequence | Sequences are a special case of tables. Indexes start from 1, following the Lua convention. | 
Learning Lua
Search Directories
search_dirs option that allows you to specify
absolute paths that will be searched when using the
Lua require function. If this option is not
set, the directories of the configuration files will be used instead.