Driplane on Dripline Documentation

Basics

Wed, 16 Sep 2020 22:38:02 +0200

JS

This filter allows to extend the basic driplane’s filter, defining Javascript scripts. It is based on islazy/plugin which in turn is based on robertkrimen/otto.
Defining a JS file with our custom logic, it is possible create a complex filter.

Parameters

Parameter	Type	Default	Description
path	STRING	empty	path of the Javascript file (it can contains multiple functions)
function	STRING	empty	name of the function in the JS file to call when a `Message` is received

... | js(path="script.js", function="MyFunction") | ...

Output

The output Message of this filter depends on the return value of the JS function itself.

Build and install

Wed, 16 Sep 2020 22:38:02 +0200

Build and install

Launching the follow command go will take care of the downloading, the dependency resolving, the building and finally of the installation of the binary in the $GOBIN directory.

go get -u github.com/Matrix86/driplane/…

First look

Wed, 16 Sep 2020 22:38:02 +0200

First look

Driplane include a simple language to define where to get the stream and what operation to exec, or how to filter the data.

In the rule could be defined 3 type of nodes:

FEEDER : it is the node responsible for creating of a stream of data (it reads every changes on a file, gets tweets from Twitter, etc..)
FILTER : it receives data from a feeder or another filter and checks some conditions on them or makes some changes on them. It sends the data to the next filter ONLY if the condition is verified.

Docker

Wed, 16 Sep 2020 22:38:02 +0200

Docker

driplane is containerized using a really lightweight Linux distribution called Alpine Linux.

To pull the latest image version:

docker pull matrix86/driplane

To run it:

docker run –rm -v config:/app/config -it matrix86/driplane:latest -config config/config.yaml

where the config directory contains the config.yaml file, the rule directory, the js directory and the templates directory.

Link to the Docker repository

Entrypoint

Wed, 16 Sep 2020 22:38:02 +0200

Entrypoint

The JS file consists of one or more functions, and they can be used in different filters or in combination between them.
The function must respect some constraints:

the function’s name contained in the function parameter of the filter has to start with a capital letter;
the function prototype must 3 variables;
the function must return a JS object with at least the filtered field;

Function’s prototype

The function’s name specified in the function parameter of the filter, receives 3 input parameters:

Plugin Packages

Wed, 16 Sep 2020 22:38:02 +0200

Packages

Since the JS doesn’t have all the useful functions to perform operations like on file manipulation or http request (see otto docs for more info), driplane maps and export to the JS environment some Go functions as packages.

These mapped functions are not definitive and they could change over time.

File

This package contains functions for manipulating files.

Return

The return value will be a JS object containing 2 fields:

Syntax

Wed, 16 Sep 2020 22:38:02 +0200

Syntax

The syntax of the driplane’s rules is very simple. As for the BASH every filter’s output is sent to the next filter’s input concatenating the two of them with the | character.
All the rules have to start with a name and end with a ; char.

Import

Using the directive import it is possible to include one or more files and their rules in another one. All the rules defined in an included file
are available for the file that imports them.

Templates

Fri, 17 Apr 2026 00:00:00 +0000

Templates

Many filters in driplane support Go templates in their parameters. Templates allow you to dynamically compose strings using the fields of the current Message (both the main field and any extra fields).

Templates are based on the standard Go text/template package and are extended with all the functions from the Sprig library, giving you access to over 100 useful functions for string manipulation, math, date formatting, encoding, and more.

Basic syntax

Inside a template, you can reference any field of the Message using the {{ .field_name }} syntax:

LLM

Fri, 06 Mar 2026 00:00:00 +0000

LLM

This filter sends the received Message to a Large Language Model (LLM) and propagates the response. It uses any-llm-go to support multiple LLM providers through a unified interface.

Both the prompt and system_prompt parameters support templates, allowing you to dynamically compose prompts using the main field and any extra fields of the incoming Message.

Supported Providers

OpenAI, Anthropic, Ollama, DeepSeek, Groq, Mistral, Gemini, llama.cpp, Llamafile.

Parameters

Parameter	Type	Default	Description
model	STRING		(required) The model name to use (e.g. `"gpt-4"`, `"claude-3-opus-20240229"`, `"llama3"`)
prompt	STRING		(required) The user prompt sent to the LLM. Supports templates with `Message` fields
system_prompt	STRING	empty	An optional system prompt. Supports templates with `Message` fields
provider	STRING	“openai”	The LLM provider to use: `openai`, `anthropic`, `ollama`, `deepseek`, `groq`, `mistral`, `gemini`, `llamacpp`, `llamafile`
api_key	STRING	empty	API key for the provider (required for cloud providers like OpenAI, Anthropic, etc.)
api_url	STRING	empty	Custom base URL for the API endpoint (useful for proxies or self-hosted instances)
target	STRING	“main”	The field of the `Message` where the LLM response will be stored. Use `"main"` to replace the message content, or any other name to set an extra field
temperature	FLOAT	0.7	Sampling temperature for the model (higher values produce more random output)
max_tokens	INT	1024	Maximum number of tokens to generate in the response

... | llm(model="gpt-4", prompt="Summarize: {{ .main }}", api_key="sk-...", provider="openai") | ...

Output

The LLM response text is placed in the field specified by target (default: main). The following extra fields are set on the output Message:

WebRSS

Thu, 05 Mar 2026 00:00:00 +0000

WebRSS

This feeder creates a stream by scraping a website and extracting articles using CSS selectors.
It is useful for websites that do not expose an RSS/Atom feed. It is based on Colly so you can refer to it for more info on how selectors work.

Parameters

Parameter	Type	Default	Description
url	STRING	empty	URL of the page to scrape
item_selector	STRING	empty	CSS selector matching each article/item block on the page
title_selector	STRING	empty	CSS selector for the article title (relative to the item block)
link_selector	STRING	empty	CSS selector for the article link (relative to the item block)
desc_selector	STRING	empty	CSS selector for the article description (relative to the item block)
date_selector	STRING	empty	CSS selector for the article date (relative to the item block)
link_attr	STRING	“href”	HTML attribute to read the URL from on the matched link element
freq	DURATION	60m	how often the page should be scraped

... | <webrss: url="https://website.io/blog", item_selector="a[href^='/blog/']", title_selector="h4, h3", link_selector="self", desc_selector="p", freq="1h"> | ...

Output

Text

The main field of the Message will contain the title of the article extracted via title_selector.

Sat, 10 Feb 2024 23:02:24 +0200

This filter allows you to download files received from the Telegram feeder or send messages.
Note: it can be used only in a rule with the Telegram Feeder.

Parameters

The following parameters are required from this filter:

Parameter	Type	Default	Description
action	STRING	“send_message”	action to perform: “send_message”, “download_file”
to	STRING	""	used with action `send_message`, it has to contain the recipient of the message: username, @username, phone number with the country code (supports templates)
to_chatid	STRING	empty	the recipient of the message will be a chat
filename	STRING	""	specified the path of where to store the downloaded file: `msg_filename` in the extra will contain the name of the file (supports templates)
text	STRING	""	the text of the file (supports templates)

Each action can use different parameters:

Sat, 10 Feb 2024 22:55:47 +0200

This feeder creates a stream using the Telegram API.
In order to use this feeder, you need to create a Telegram Application and get app_id and app_hash.
If you set the session folder the session will be stored and only the first run it will ask for the code received from the phone number specified in the config.

Based on gotd/td

Parameters

Parameter	Type	Default	Description
app_id	STRING	empty	app ID (see this)
app_hash	STRING	empty	App hash (see this)
phone_number	STRING	empty	Phone number of the account to use (it should contain the country code)
session_folder	STRING	empty	Path of the folder where storing the sessions

*events = “channel_message”,“chat_message”

Imap

Thu, 16 Nov 2023 20:08:45 +0200

Imap

This feeder creates a stream starting from emails received on the account read by an IMAP client. It is possible to define how often the email account should be checked.
Every time the email inbox is parsed a Message is sent down the lane.

Parameters

Parameter	Type	Default	Description
host	STRING	empty	Host of the IMAP server
port	STRING	empty	Port of the IMAP server
username	STRING	empty	Username of the account
password	STRING	empty	Password of the account
mailbox	STRING	“INBOX”	Name of the mailbox to read
freq	DURATION	“1m”	how often the email account should be checked
start_from_beginning	BOOL	“true”	if “true” it reads all the emails in the mailbox from the beginning
get_attachments	BOOL	“false”	if “true” it reads also the attachments

... | <imap: host="imap.gmail.com", port="993", username="test@gmail.com", password="xxxxx", get_attachments="true", freq="30m"> ...

Output

Text

The main field of the Message will contain the email’s subject.

XLS

Thu, 16 Nov 2023 20:08:45 +0200

XLS

This filter allows you to extract all the rows from a Excel file.
Based on the qax-os/excelize library.

Parameters

Parameter	Type	Default	Description
target	STRING	“main”	the field of the Message that should be used for the filter (it could be the `main` or and extra field)
filename	STRING	empty	the filename of the XLS file to parse (supports templates)

... | xls(target="{{ .extra_field }}") | ...

The filename field override the target. They are mutually exclusive, so you can specify only one of them.

Output

The filter produces one Message for each row of the XLS file.

RateLimit

Tue, 31 Jan 2023 18:50:23 +0200

RateLimit

This filter allows you to set a rate limit on the messages that can go through it. So for example if we don’t want to limit the number of messages in a pipe to 5 messages per second we just to set the parameter rate to 5.

Parameters

Parameter	Type	Default	Description
rate	STRING	“0”	how many event per second you want to have as rate limiter

... | ratelimit(rate="5") | ...

Output

The filter will slow down the rate of the messages as specified on the parameter rate.

Json

Sun, 29 Jan 2023 18:57:01 +0200

Json

This filter is used to extract information from a JSON doc received through the Message.
It is possible to use XPath query for JSON specifying a selector to search in the doc and extract data (it uses jsonquery).

Parameters

Parameter	Type	Default	Description
target	STRING	“main”	the field of the Message that should be used for the filter (it could be main or an extra field)
selector	STRING	""	the selector to find the data in the JSON

... | json(selector="id", target="doc") | ...

Output

The filter will generate one or more Messages. It is possible to use more than 1 time this filter.
The field fulltext will contain the original target string.

Html

Tue, 19 Jul 2022 23:38:02 +0200

Html

This filter is used to extract information from an HTML page received through the Message.
Like jQuery for JS, we can set a selector to find in the page, extract text from the tags and the html content (we are using goquery library under the hood).

Parameters

Parameter	Type	Default	Description
target	STRING	“main”	the field of the Message that should be used for the filter (it could be main or an extra field)
selector	STRING	""	the selector to find in the HTML page
get	STRING	“html”	what do we want to retrieve on the tags found in the selected one: `html`, `text`, `attr`
attr	STRING	""	if get is `attr` you can define what attr name it should extract

... | html(selector=".link", get="attr", attr="href") | ...

Output

The filter will generate one or more Messages. It is possible to use more than 1 time this filter.
The field fulltext will contain the original target string.

Apt

Tue, 15 Mar 2022 11:38:24 +0100

Apt feeder

This feeder can create the stream starting from an apt repository. It supports also flat repositories.
It is possible to specify the frequency of the quering and receive a message every time a new package is published.

Parameters

Parameter	Type	Default	Description
url	STRING	empty	URL of the apt repo
freq	DURATION	60s	how often the feed should be parsed
suite	STRING	“stable”	suite of the repo to keep under control
arch	STRING	empty"	architecture of the repo, if empty the first arch returned by the Release file will be used
index	STRING	empty	URL of the Packages file (it overrides the url parameter)
insecure	BOOL	false	allow repository with insecure certificates

... | <apt: url="http://apt.modmyi.com/dists/stable/Release", insecure="true", freq="3h"> | ...

Output

Text

The main field of the Message will contain the filename of the package and all the other field will be present in extra fields.

Elasticsearch

Sat, 12 Mar 2022 15:40:23 +0100

ElasticSearch

This filter writes on ElasticSearch all the message it receives as input and return as output the document ID.
You can also specify what message’s field it should use as input.

Parameters

Parameter	Type	Default	Description
target	STRING	“main”	the field of the Message that should be used for filter’s output (it could be main or an extra field)

... | elasticsearch(target="input_field") | ...

Output

The main field will contain the docID returned by ElasticSearch.

File

Sat, 12 Mar 2022 15:33:42 +0100

File

This filter get as input the path of a local file, read it and return the content back to the pipeline.

Parameters

Parameter	Type	Default	Description
target	STRING	“main”	the field of the Message that should be used for filter’s output (it could be main or an extra field)

... | file(target="file_content") | ...

Output

The output will contain the content of the read file. Using the target parameter you can specify the output field’s name.

Striptag

Thu, 01 Jul 2021 15:35:53 +0200

StripTag

This filter is used to remove all the HTML tags from a string.

Parameters

Parameter	Type	Default	Description
target	STRING	“main”	the field of the Message that should be used for the filter (it could be main or an extra field)

... | striptag(target="main") | ...

Output

The main of the output Message will have the text extracted from the target config, stripped by all the HTML tags.

Timer

Sat, 20 Feb 2021 18:06:02 +0200

Timer feeder

This feeder trigger a pipeline every time the timer is fired.

Parameters

Parameter	Type	Default	Description
freq	DURATION	60s	The intervals (in duration) on how often to execute the pipeline

<timer: freq="30s"> | ...

Output

Text

The main field of the Message will contain time in rfc3339 format.

Extra

Name	Description
rfc3339	time in rfc3339 format
timestamp	time in Unix timestamp

Examples

Soon…

Folder

Mon, 01 Feb 2021 13:38:02 +0200

Folder feeder

This feeder can create the stream of fsnotify events for a given folder or for cloud platform storage like Amazon S3,
Google Drive and Dropbox.

The feeder use cloudwatcher to keep track of changes on the chosen directory.

Parameters

Parameter	Type	Default	Description
name	STRING	empty	the path of the folder that it has to keep track
type	STRING	“local”	the type of service to use `local`, `dropbox`, `gdrive`, `s3` or `git`
freq	DURATION	2s	how often the directory should be checked for updates

Some services like Gdrive, S3 and Dropbox require additional configurations (you can check them from here).
You can pass them using the config file (more here) OR
define them in the rule itself: <folder: name="/", type="gdrive", client_id="xxx", client_secret="yyy", token="zzz">

<folder: name="/tmp", type="local"> | ...

Output

Text

The main field of the Message will contain the filename while the op extra the type of event.

Slack

Mon, 12 Oct 2020 19:38:02 +0200

Slack

This filter allows you to send files, messages and download files from Slack.
It can be used alone or with the Slack Feeder to create a simple Slack bot for events API.

Parameters

The following parameters are required from this filter:

Parameter	Type	Default	Description
action	STRING	“send_message”	action to perform: “send_message”, “send_file”, “download_file”, “user_info”
token	STRING	""	Slack bot Token

Each action can use different parameters:

action = send_message

Parameter	Type	Default	Description
to	STRING	""	channel ID or User ID that should receive the message (supports templates)
target	STRING	“main”	if `text` is not used you can choose which field of `Message` to use as text
text	STRING	""	you can define the text of the message to send (supports templates)
blocks	BOOL	false	if true you can use the Slack template blocks (block builder)

action = send_file

Parameter	Type	Default	Description
to	STRING	""	channel ID or User ID that should receive the message (supports templates)
target	STRING	“main”	if `filename` is not specified you can choose which field of `Message` has to be used as file content to send
filename	STRING	""	path of the file to send (supports templates)

action = download_file

Parameter	Type	Default	Description
url	STRING	""	it should contain the Slack private url for the file download (supports templates)
target	STRING	“urlprivate”	if `url` is not specified you can choose which field of `Message` contains the Slack private url
filename	STRING	""	path of where to save the downloaded file. If not specified the file content will be inserted in the `main` field (supports templates)

action = user_info

Parameter	Type	Default	Description
target	STRING	“user”	this field has to contain the USERID

In the output Message you can find all the user information returned by Slack:

Slack

Tue, 06 Oct 2020 22:05:07 +0200

Slack

This feeder creates a stream using the Slack Events API.
In order to use this feeder, you need to create a Slack App, define its Bot Token scopes and finally enable the Event Subscriptions.
The feeder will start a webserver to receive the events, so it should be reachable by slack. If you are behind a NAT or Firewall, you can enable localtunnel.
Based on slack-go/slack

Parameters

Parameter	Type	Default	Description
bot_token	STRING	empty	token Bot (starting with xoxb-*)
app_token	STRING	empty	App token (starting with xapp-*)
verification_token	STRING	empty	token to verify the requests (you can find it in the Basic information section)
addr	STRING	“:3000”	address and port in the form IP:PORT
socket_mode	BOOL	false	enable the Socket Mode
lt_enable	BOOL	false	enable localtunnel for the server
lt_baseurl	STRING	“https://localtunnel.me”	URL to the Proxy server
lt_subdomain	STRING	empty	specify a subdomain to use with localtunnel
events	STRING	*events	comma separated event list to handle
ignore_bot	BOOL	true	if true the bot will ignore mentions and messages created by another bot

*events = “app_mention”,“app_home_opened”,“app_uninstalled”,“grid_migration_finished”,“grid_migration_started”,“link_shared”,“message”,“member_joined_channel”,“member_left_channel”,“pin_added”,“pin_removed”,“reaction_added”,“reaction_removed”,“tokens_revoked”,“file_shared”

Cache

Wed, 16 Sep 2020 22:38:02 +0200

Cache

This filter introduces a simple cache mechanism in the rule. It is a TTL based cache and it can have a local visibility (cache is only visible to the current filter) or a global visibility (cache shared across ALL the rules).
If the target of the Message as input has been cached before, and his TTL is not expired, it will be dropped and not propagated to the next filter.
Otherwise if the target of the Message is new to the cache, it is inserted in it and propagated to the next filter.

Changed

Wed, 16 Sep 2020 22:38:02 +0200

Changed

This filter is similar to the cache. It can only stop the propagation of the Message across the lane, but only if the target of the received Message is different from the previous one.

Parameters

Parameter	Type	Default	Description
target	STRING	“main”	the field of the Message that should be used for the filter (it could be main or and extra field)

... | changed(target="original_author") | ...

Output

The output is not being changed. This filter can only stop or not the propagation of the Message.

Echo

Wed, 16 Sep 2020 22:38:02 +0200

Echo

This filter prints the Message on the logs. It is mostly used to debug the rules.

Parameters

Parameter	Type	Default	Description
extra	BOOL	“false”	print also all the extra fields

... | echo(extra="false") | ...

Output

The output is not being changed. This filter prints the received Message and send it to the next filter in the rule.

Examples

Soon…

File

Wed, 16 Sep 2020 22:38:02 +0200

File feeder

This feeder can create the stream starting from a file. Like the tail -f command it opens the specified file and propagates a data message if a line is being added to the file.

Parameters

Parameter	Type	Default	Description
filename	STRING	empty	the path of the file that it has to keep track
toend	BOOL	“false”	the feeder will start to create data messages only for new added lines

... | <file: filename="path/of/the/file.txt", toend="false"> | ...

Output

Text

The main field of the Message will contain the new read line.

Format

Wed, 16 Sep 2020 22:38:02 +0200

Format

This filter is used to format the received Message. It is based on
Go templates and it can load templates from the template_path
directory specified in the configuration file. All Sprig functions are available.

Parameters

Parameter	Type	Default	Description
type	STRING	“text”	specify the type of template to use : `"text"` or `"html"`
template	STRING	empty	a template could be specified directly here, instead of load it from file (supports templates)
file	STRING	empty	load the template from file (supports templates)
target	STRING	“main”	the field of the Message that should be used for the filter (it could be main or an extra field)

In the template is allowed to use all the fields of the received Message: main or extra.

Hash

Wed, 16 Sep 2020 22:38:02 +0200

Hash

This filter is used to search or extract hashes from a Message.
Supported types of hashes are:

MD5
SHA1
SHA256
SHA512

Parameters

Parameter	Type	Default	Description
target	STRING	“main”	the field of the Message that should be used for the filter (it could be main or an extra field)
extract	BOOL	“false”	if `"true"` the main field of the output Message will be the extracted hash
md5	BOOL	“true”	if `"false"` md5 hashes will be ignored
sha1	BOOL	“true”	if `"false"` sha1 hashes will be ignored
sha256	BOOL	“true”	if `"false"` sha256 hashes will be ignored
sha512	BOOL	“true”	if `"false"` sha512 hashes will be ignored

... | hash(target="description", extract="true") | ...

Output

If the extract parameter is “false”, the received Message will be propagated only if at least one hash is found in it.
Otherwise, if extract is “true” and the Message contains one or more hashes, the main field of the propagated Message will contain only the extracted hash.

Http

Wed, 16 Sep 2020 22:38:02 +0200

HTTP

This filter allows you to send HTTP requests. When a Message arrives to the filter, we can decide if use the main field of the Message as URL on the request, or its content for the HTTP data.
This behaviour can be handled with the parameters.

Parameters

Parameter	Type	Default	Description
url	STRING	empty	URL of the web page. It is possible use templates to use fields of the `Message`
download_to	STRING	empty	path of where to download the file. It is possible use templates to use fields of the `Message`
text_only	BOOL	“false”	if “true” it removes all the tags from the body response
method	STRING	“GET”	HTTP method to use on the request
headers	JSON	empty	Headers to use in the request
data	JSON	empty	POST fields to send with the request (it’s not possible to use in combination with `rawData`)
rawData	STRING	empty	raw body of the request (it’s not possible to use in combination with `data`)
status	STRING	empty	the filter will propagate the Message only if the returned status has the specified value
cookies	STRING	empty	Path of the JSON file containing the cookies to use

... | http(url="{{ .main }}", cookies="exported.json", headers="{\"Content-type\": \"application/json\"}") | ...

Output

If the request was successful, the output Message will have the main field set to the HTTP body response. If the status is set, and the response http status is different from it, the Message will be dropped.

Mail

Wed, 16 Sep 2020 22:38:02 +0200

Mail

This filter allows you to send an e-mail. When a Message arrives to the filter, we can decide if use the main field of the Message as URL on the request, or its content for the HTTP data.
This behaviour can be handled with the parameters.

Parameters

Parameter	Type	Default	Description
body	STRING	empty	the body of the e-mail (supports templates)
username	STRING	empty	username for the host authentication
password	STRING	empty	password for the host authentication
host	STRING	empty	host server used to send the e-mail
port	STRING	empty	port of the host server
fromAddr	STRING	empty	source e-mail address
fromName	STRING	empty	source name address
to	STRING	empty	destination e-mail address (supports multi-destination, comma separated)
subject	STRING	empty	subject field of the e-mail to send
use_auth	BOOL	“false”	if “true” the sendmail server will receive the credentials specified in `username` and `password` fields

... | http(url="{{ .main }}", cookies="exported.json", headers="{\"Content-type\": \"application/json\"}") | ...

Every default’s value can be set in the configuration, creating a section with the name of theFilter/Feeder.
more info

Output

The input Message is always propagated to the next filter without changes.

MIME type

Wed, 16 Sep 2020 22:38:02 +0200

MIME type

This filter allows you to detect the MIME type of a file and its extension.
Based on the gabriel-vasile/mimetype library.

Parameters

Parameter	Type	Default	Description
target	STRING	“main”	the field of the Message that should be used for the filter (it could be the `main` or and extra field)
filename	STRING	empty	the filename of the file to detect (supports templates)

... | mime(target="{{ .extra_field }}") | ...

The filename field override the target. They are mutually exclusive, so you can specify only one of them.

Output

The propagated Message will contain the mimetype’s string in the main field and the extension in the extra field mimetype_ext.

Number

Wed, 16 Sep 2020 22:38:02 +0200

Number

This filter allows you to treat a string from the input Message as numeric value and apply some operator on it (for example, check if this field contains a number greater than X).

Parameters

Parameter	Type	Default	Description
target	STRING	“main”	the field of the Message that should be used for the filter (it could be main or and extra field)
op	STRING	""	Compare operator to use for the numeric value (">", “>=”, “<”, “<=”, “!=”, “==”)
value	STRING	""	It has to be a numeric value and it is the number to use for the comparison

... | number(target="num_field", op=">=", value="44") | ...

Output

If the comparison is verified the received Message will be propagated.

Override

Wed, 16 Sep 2020 22:38:02 +0200

Override

This filter allows you to change a field of a Message, before sending it to the next filter. Templates with Sprig functions can be used.

Parameters

Parameter	Type	Default	Description
name	STRING	empty	name of the field to change (supports templates)
value	STRING	empty	new value to assign to the Message’s field specified (supports templates)

... | override(name="description", value="{{ .title }}") | ...

Output

The propagated Message will be identical to the original, with only the specified field changed.

Pdf

Wed, 16 Sep 2020 22:38:02 +0200

Pdf

This filter allows you to extract plain text from a PDF file.
Based on the ledongthuc/pdf library.

Parameters

Parameter	Type	Default	Description
target	STRING	“main”	the field of the Message that should be used for the filter (it could be the `main` or and extra field)
filename	STRING	empty	the filename of the PDF file to parse (supports templates)

... | pdf(target="{{ .extra_field }}") | ...

The filename field override the target. They are mutually exclusive, so you can specify only one of them.

Output

The propagated Message contains the plain text of the input PDF file (fulltext will be set to the file name received as input).

Random

Wed, 16 Sep 2020 22:38:02 +0200

Random

This filter is used to inject an extra field with a random number in the propagated Message.

Parameters

Parameter	Type	Default	Description
output	STRING	“main”	the field of the propagated `Message` that will contain the random number
min	STRING	“0”	the min value of the extracted number is `min`
max	STRING	“999999”	the max value of the extracted number is `max`

... | random(output="random_field", min="9", max="90") | ...

Output

The output Message will be equal to the input, but it will also include the new random field.

RSS

Wed, 16 Sep 2020 22:38:02 +0200

RSS

This feeder creates a stream starting from a feed RSS, ATOM or JSON.
It is based on gofeed so you can refer to it for more info and supported formats.

Parameters

Parameter	Type	Default	Description
url	STRING	empty	URL of the feed
freq	DURATION	60s	how often the feed should be parsed
start_from_beginning	BOOL	“false”	if “true” it starts to parse the feed from the beginning (the first time it will ignore the pubdate field of the feed)
ignore_pubdate	BOOL	“false”	if “true” it ignores the pubdate and it returns all the feed content every time

... | <rss: url="https://example.rss", freq="5s", start_from_beginning="false"> | ...

Output

Text

The main field of the Message will contain the item.Title string of the gofeed.Item struct.

System

Wed, 16 Sep 2020 22:38:02 +0200

System

This filter allows you to exec a command on the host machine. The received Message can be used to create the command to launch.
It supports templates with Sprig functions.

Parameters

Parameter	Type	Default	Description
cmd	STRING	empty	command line to exec for each received Message (supports templates)

... | system(cmd="echo '{{ .author }} wrote {{ .main }}' >> logs.txt") | ...

Output

The propagated Message will contain the output of the command if it is provided, and it is not failed.

Text

Wed, 16 Sep 2020 22:38:02 +0200

Text

This filter searches or extracts strings from the received Message. It can be used with a regular expression or a simple string.
If the string is found, the condition is matched and the Message is propagated to the next filter.

Parameters

Parameter	Type	Default	Description
target	STRING	“main”	the field of the Message that should be used for the filter (it could be main or and extra field)
regexp	BOOL	false	the pattern field is a regular expression
extract	BOOL	“false”	if “true” the `main` field of the propagated Message will contain the extracted string (it can be used only if `regexp` parameter is set true)
pattern	STRING	empty	specifies the pattern that should be matched on the Message to check the condition

... | text(target="description", pattern="(#[^\\s]+)", regexp="true", extract="true") | ...

Output

If the extract parameter is “false”, the received Message will be propagated only if the specified pattern is matched in the target field of the Message.
Otherwise if extract is “true” (only regexp can be used in this case), and one or more strings matches with the pattern, the main field of the propagated Message will contain only the matched string.

Twitter

Wed, 16 Sep 2020 22:38:02 +0200

Twitter

This feeder creates a stream from tweets. It is possible to define the keywords, or the users to follow.
Based on go-twitter

Parameters

Parameter	Type	Default	Description
bearerToken	STRING	empty	Twitter Auth
keywords	STRING	empty	comma separated keywords that should match on the tweets
users	STRING	empty	comma separated users list
rules	STRING	empty	set multiple custom rules separated by the char
languages	STRING	empty	filter by language (comma separated languages)
disable_retweet	BOOL	“false”	don’t include retweets in the stream
disable_quoted	BOOL	“false”	don’t include quoted tweets in the stream

... | <twitter: users="goofy, mickeymouse",keywords="movie, cartoon", rules="rule1:movie OR cartoon|rule2:mickey mouse OR pluto"> | ...

Output

Text

The main field of the Message will contain:

Url

Wed, 16 Sep 2020 22:38:02 +0200

Url

This filter is used to search or extract URLs from a Message.
Currently supported types of URLs are:

http/s
ftp

Parameters

Parameter	Type	Default	Description
target	STRING	“main”	the field of the Message that should be used for the filter (it could be main or an extra field)
http	BOOL	“true”	if “false”, `http` scheme urls are ignored
https	BOOL	“true”	if “false”, `https` scheme urls are ignored
ftp	BOOL	“true”	if “false”, `ftp` scheme urls are ignored
extract	BOOL	“true”	if “true”, the `main` field of the propagated Message will contain the found URL

... | url(extract="true") | ...

Output

If the extract parameter is “false”, the received Message will be propagated only if at least one URL is found in it.
Otherwise, if extract is “true” and the Message contains one or more URLs, the main field of the propagated Message will contain only the extracted URLs.

Web

Wed, 16 Sep 2020 22:38:02 +0200

Web

This feeder creates a stream starting from a web page. It is possible to define how often the page should be downloaded and parsed.
Every time the page is parsed a Message is sent down the lane.

Parameters

Parameter	Type	Default	Description
url	STRING	empty	URL of the web page
freq	DURATION	60s	how often the page should be parsed
text_only	BOOL	“false”	if “true” it removes all the tags from the page
method	STRING	“GET”	HTTP method to use on the requests
headers	JSON	empty	Headers to use in the request
data	JSON	empty	POST fields to send with the requests (it’s not possible to use in combination with `rawData`)
rawData	STRING	empty	raw body of the requests (it’s not possible to use in combination with `data`)
status	STRING	empty	the filter will propagate the Message only if the returned status is this
cookies	STRING	empty	Path of the JSON file containing the cookies to use

... | <web: url="https://example.com", freq="30m", status="200", cookies="/path/to/exported.json"> | ...

Output

Text

The main field of the Message will contain the HTML source or the text of the website if the text_only parameter is set to true.