Feeders on Dripline Documentation

WebRSS

Thu, 05 Mar 2026 00:00:00 +0000

WebRSS

This feeder creates a stream by scraping a website and extracting articles using CSS selectors.
It is useful for websites that do not expose an RSS/Atom feed. It is based on Colly so you can refer to it for more info on how selectors work.

Parameters

Parameter	Type	Default	Description
url	STRING	empty	URL of the page to scrape
item_selector	STRING	empty	CSS selector matching each article/item block on the page
title_selector	STRING	empty	CSS selector for the article title (relative to the item block)
link_selector	STRING	empty	CSS selector for the article link (relative to the item block)
desc_selector	STRING	empty	CSS selector for the article description (relative to the item block)
date_selector	STRING	empty	CSS selector for the article date (relative to the item block)
link_attr	STRING	“href”	HTML attribute to read the URL from on the matched link element
freq	DURATION	60m	how often the page should be scraped

... | <webrss: url="https://website.io/blog", item_selector="a[href^='/blog/']", title_selector="h4, h3", link_selector="self", desc_selector="p", freq="1h"> | ...

Output

Text

The main field of the Message will contain the title of the article extracted via title_selector.

Sat, 10 Feb 2024 22:55:47 +0200

This feeder creates a stream using the Telegram API.
In order to use this feeder, you need to create a Telegram Application and get app_id and app_hash.
If you set the session folder the session will be stored and only the first run it will ask for the code received from the phone number specified in the config.

Based on gotd/td

Parameters

Parameter	Type	Default	Description
app_id	STRING	empty	app ID (see this)
app_hash	STRING	empty	App hash (see this)
phone_number	STRING	empty	Phone number of the account to use (it should contain the country code)
session_folder	STRING	empty	Path of the folder where storing the sessions

*events = “channel_message”,“chat_message”

Imap

Thu, 16 Nov 2023 20:08:45 +0200

Imap

This feeder creates a stream starting from emails received on the account read by an IMAP client. It is possible to define how often the email account should be checked.
Every time the email inbox is parsed a Message is sent down the lane.

Parameters

Parameter	Type	Default	Description
host	STRING	empty	Host of the IMAP server
port	STRING	empty	Port of the IMAP server
username	STRING	empty	Username of the account
password	STRING	empty	Password of the account
mailbox	STRING	“INBOX”	Name of the mailbox to read
freq	DURATION	“1m”	how often the email account should be checked
start_from_beginning	BOOL	“true”	if “true” it reads all the emails in the mailbox from the beginning
get_attachments	BOOL	“false”	if “true” it reads also the attachments

... | <imap: host="imap.gmail.com", port="993", username="test@gmail.com", password="xxxxx", get_attachments="true", freq="30m"> ...

Output

Text

The main field of the Message will contain the email’s subject.

Apt

Tue, 15 Mar 2022 11:38:24 +0100

Apt feeder

This feeder can create the stream starting from an apt repository. It supports also flat repositories.
It is possible to specify the frequency of the quering and receive a message every time a new package is published.

Parameters

Parameter	Type	Default	Description
url	STRING	empty	URL of the apt repo
freq	DURATION	60s	how often the feed should be parsed
suite	STRING	“stable”	suite of the repo to keep under control
arch	STRING	empty"	architecture of the repo, if empty the first arch returned by the Release file will be used
index	STRING	empty	URL of the Packages file (it overrides the url parameter)
insecure	BOOL	false	allow repository with insecure certificates

... | <apt: url="http://apt.modmyi.com/dists/stable/Release", insecure="true", freq="3h"> | ...

Output

Text

The main field of the Message will contain the filename of the package and all the other field will be present in extra fields.

Timer

Sat, 20 Feb 2021 18:06:02 +0200

Timer feeder

This feeder trigger a pipeline every time the timer is fired.

Parameters

Parameter	Type	Default	Description
freq	DURATION	60s	The intervals (in duration) on how often to execute the pipeline

<timer: freq="30s"> | ...

Output

Text

The main field of the Message will contain time in rfc3339 format.

Extra

Name	Description
rfc3339	time in rfc3339 format
timestamp	time in Unix timestamp

Examples

Soon…

Folder

Mon, 01 Feb 2021 13:38:02 +0200

Folder feeder

This feeder can create the stream of fsnotify events for a given folder or for cloud platform storage like Amazon S3,
Google Drive and Dropbox.

The feeder use cloudwatcher to keep track of changes on the chosen directory.

Parameters

Parameter	Type	Default	Description
name	STRING	empty	the path of the folder that it has to keep track
type	STRING	“local”	the type of service to use `local`, `dropbox`, `gdrive`, `s3` or `git`
freq	DURATION	2s	how often the directory should be checked for updates

Some services like Gdrive, S3 and Dropbox require additional configurations (you can check them from here).
You can pass them using the config file (more here) OR
define them in the rule itself: <folder: name="/", type="gdrive", client_id="xxx", client_secret="yyy", token="zzz">

<folder: name="/tmp", type="local"> | ...

Output

Text

The main field of the Message will contain the filename while the op extra the type of event.

Slack

Tue, 06 Oct 2020 22:05:07 +0200

Slack

This feeder creates a stream using the Slack Events API.
In order to use this feeder, you need to create a Slack App, define its Bot Token scopes and finally enable the Event Subscriptions.
The feeder will start a webserver to receive the events, so it should be reachable by slack. If you are behind a NAT or Firewall, you can enable localtunnel.
Based on slack-go/slack

Parameters

Parameter	Type	Default	Description
bot_token	STRING	empty	token Bot (starting with xoxb-*)
app_token	STRING	empty	App token (starting with xapp-*)
verification_token	STRING	empty	token to verify the requests (you can find it in the Basic information section)
addr	STRING	“:3000”	address and port in the form IP:PORT
socket_mode	BOOL	false	enable the Socket Mode
lt_enable	BOOL	false	enable localtunnel for the server
lt_baseurl	STRING	“https://localtunnel.me”	URL to the Proxy server
lt_subdomain	STRING	empty	specify a subdomain to use with localtunnel
events	STRING	*events	comma separated event list to handle
ignore_bot	BOOL	true	if true the bot will ignore mentions and messages created by another bot

*events = “app_mention”,“app_home_opened”,“app_uninstalled”,“grid_migration_finished”,“grid_migration_started”,“link_shared”,“message”,“member_joined_channel”,“member_left_channel”,“pin_added”,“pin_removed”,“reaction_added”,“reaction_removed”,“tokens_revoked”,“file_shared”

File

Wed, 16 Sep 2020 22:38:02 +0200

File feeder

This feeder can create the stream starting from a file. Like the tail -f command it opens the specified file and propagates a data message if a line is being added to the file.

Parameters

Parameter	Type	Default	Description
filename	STRING	empty	the path of the file that it has to keep track
toend	BOOL	“false”	the feeder will start to create data messages only for new added lines

... | <file: filename="path/of/the/file.txt", toend="false"> | ...

Output

Text

The main field of the Message will contain the new read line.

RSS

Wed, 16 Sep 2020 22:38:02 +0200

RSS

This feeder creates a stream starting from a feed RSS, ATOM or JSON.
It is based on gofeed so you can refer to it for more info and supported formats.

Parameters

Parameter	Type	Default	Description
url	STRING	empty	URL of the feed
freq	DURATION	60s	how often the feed should be parsed
start_from_beginning	BOOL	“false”	if “true” it starts to parse the feed from the beginning (the first time it will ignore the pubdate field of the feed)
ignore_pubdate	BOOL	“false”	if “true” it ignores the pubdate and it returns all the feed content every time

... | <rss: url="https://example.rss", freq="5s", start_from_beginning="false"> | ...

Output

Text

The main field of the Message will contain the item.Title string of the gofeed.Item struct.

Twitter

Wed, 16 Sep 2020 22:38:02 +0200

Twitter

This feeder creates a stream from tweets. It is possible to define the keywords, or the users to follow.
Based on go-twitter

Parameters

Parameter	Type	Default	Description
bearerToken	STRING	empty	Twitter Auth
keywords	STRING	empty	comma separated keywords that should match on the tweets
users	STRING	empty	comma separated users list
rules	STRING	empty	set multiple custom rules separated by the char
languages	STRING	empty	filter by language (comma separated languages)
disable_retweet	BOOL	“false”	don’t include retweets in the stream
disable_quoted	BOOL	“false”	don’t include quoted tweets in the stream

... | <twitter: users="goofy, mickeymouse",keywords="movie, cartoon", rules="rule1:movie OR cartoon|rule2:mickey mouse OR pluto"> | ...

Output

Text

The main field of the Message will contain:

Web

Wed, 16 Sep 2020 22:38:02 +0200

Web

This feeder creates a stream starting from a web page. It is possible to define how often the page should be downloaded and parsed.
Every time the page is parsed a Message is sent down the lane.

Parameters

Parameter	Type	Default	Description
url	STRING	empty	URL of the web page
freq	DURATION	60s	how often the page should be parsed
text_only	BOOL	“false”	if “true” it removes all the tags from the page
method	STRING	“GET”	HTTP method to use on the requests
headers	JSON	empty	Headers to use in the request
data	JSON	empty	POST fields to send with the requests (it’s not possible to use in combination with `rawData`)
rawData	STRING	empty	raw body of the requests (it’s not possible to use in combination with `data`)
status	STRING	empty	the filter will propagate the Message only if the returned status is this
cookies	STRING	empty	Path of the JSON file containing the cookies to use

... | <web: url="https://example.com", freq="30m", status="200", cookies="/path/to/exported.json"> | ...

Output

Text

The main field of the Message will contain the HTML source or the text of the website if the text_only parameter is set to true.