New decoding and framing options for sources

Page source Edit this page

Often, when consuming data from a source, the first operation you have to do on it is decode the data from its source representation. To make this easier, we’ve added new decoding options to most sources.

For example, if you have a kafka source that has JSON-encoded messages, now you can simply add decoding.codec = "json" to your source configuration like:

[sources.kafka]
type = "kafka"
bootstrap_servers = "localhost:9200"
topics = ["my_topic"]
decoding.codec = "json"

This will decode your messages from JSON, thus saving you from an additional remap transform.

In addition, we’ve added a new framing option to allow configuration for sources that have non-standard framing (for example a custom-delimiter separating messages).

For example, if you have an http source where the messages are delimited by commas instead of newlines, you can configure this like:

[sources.http]
type = "http"
address = "0.0.0.0:8080"
framing.method = "character_delimited"
framing.character_delimited.delimiter = ","

To have Vector parse each comma-delimited element as a new message. This can be used with decoding option specified above. See the docs for other framing options.