Client Scripts

Bash scripts for client machines (FreeBSD/Linux; need bash and curl or wget). Download the current version from the server:

curl -fsSLO https://tis.cap-net.cz/get/tis-pull.sh && chmod +x tis-pull.sh
curl -fsSLO https://tis.cap-net.cz/get/tis-push.sh && chmod +x tis-push.sh
curl -fsSLO https://tis.cap-net.cz/get/tis-hits.sh && chmod +x tis-hits.sh

The sources are in scripts/ of the repository, together with the tis-cron.sh example.

Installer

Since v1.32.0. Every release has a self-extracting installer, tis-client-<version>.run (linked from the GitLab release, in the package registry under tis-client), that installs the scripts (tis-cron.sh, tis-hits.sh, tis-pull.sh, tis-push.sh) into one directory – tis-cron.sh finds the others next to itself – together with tis-cron.d/example.sh.sample:

sh tis-client-1.32.0.run -- --dry-run          # what would change in /opt/tis
sudo sh tis-client-1.32.0.run                  # install or update /opt/tis
sh tis-client-1.32.0.run -- --dir ~/tis        # another directory

New files are installed and identical ones left alone. Your steps in tis-cron.d/*.sh are never touched, the examples (*.sample) are updated without asking. For a script that differs (e.g. a locally edited one) it asks – overwrite, keep, show the diff, or overwrite all – and keeps the replaced file as <file>.bak.<timestamp>. Without a terminal (e.g. from Ansible) differing files are kept unless --overwrite is given.

Option	Description
-d, --dir <dir>	Where to install (default /opt/tis, or TIS_INSTALL_DIR).
-n, --dry-run	Only show what would be installed or overwritten.
-y, --overwrite	Overwrite files that differ without asking.
-N, --no-overwrite	Never overwrite files that differ.
-h, --help	Help.

The options go after --; before it are the archive's own makeself options (--check, --list, --target to only extract, …). Build one locally with scripts/build-installer.sh (needs makeself).

tis-pull.sh

Downloads an export and deploys it atomically: compares it with the current file, replaces it only when it changed, runs a reload command and rolls back if the reload fails. Common targets are built in – see Deployment targets.

tis-pull.sh --token "$TOKEN" --group customer-x --type fqdn \
  --destination /usr/local/etc/namedb/master/tis-blocklist.zone --bind

tis-pull.sh --token "$TOKEN" --source 2 --type ipv4 --ipfw-table tis-blocklist

Option	Env	Description
--token *	TIS_TOKEN	API token.
--format	TIS_FORMAT	plain, hosts, rpz, zone, csv, nftables, ipset, ipfw, dnsmasq, unbound (default plain).
--type	TIS_TYPE	ipv4, ipv6, fqdn, all (default).
--tag	TIS_TAG	Only indicators with this tag.
--whitelist	TIS_WHITELIST	Comma-separated values to exclude.
--rpz-ip <response\|client\|nsip>	TIS_RPZ_IP	With --format rpz: add IP indicators as RPZ IP triggers (since 1.19.0 of the script).
--country <codes>	TIS_COUNTRY	Only IP indicators in these GeoIP countries, e.g. CZ,SK (since 1.21.0 of the script).
--exclude-country <codes>	TIS_EXCLUDE_COUNTRY	No IP indicators in these countries, e.g. CN,RU (since 1.21.0 of the script).
--group <id\|name>	TIS_GROUP	Pull a group (since 1.16.0 of the script).
--source <ids>	TIS_SOURCE	Pull specific (private) sources, e.g. 5,6. Not with --group.
--destination	TIS_DESTINATION	Target file (default: stdout).
--reloadcmd	TIS_RELOADCMD	Run after a changed file was deployed.
--revertcmd	TIS_REVERTCMD	Run after rolling back a failed reload (default: --reloadcmd again).
--checkcmd	TIS_CHECKCMD	Validate the new file before deploying it; {} is its path (appended when missing). Failure → nothing deployed, exit 3.
--ipfw-table, --pf-table, --ipset <name>		Load into a firewall table, see Deployment targets (since 1.20.0).
--ipfw-rule <num>		Add a rule to the IPFW firewall, see Deployment targets (since 1.21.0).
--ipfw-legacy	TIS_IPFW_LEGACY	Older ipfw without create … missing / or-flush (since 1.22.0).
--bind, --unbound, --dnsmasq		Validate and reload a DNS server, see Deployment targets (since 1.20.0).
--table-opts, --state-dir	TIS_TABLE_OPTS, TIS_STATE_DIR	Table creation options; where the loaded list is kept (default /var/db/tis, or /var/lib/tis without /var/db).
--config	TIS_CONFIG	Shell file with TIS_* defaults (e.g. the token).
--api-base	TIS_API_BASE	Default https://tis.cap-net.cz/api/v1/export.
--retries, --retry-delay, --connect-timeout, --max-time	TIS_RETRIES, …	Network tuning (defaults 3, 2 s, 15 s, 120 s).
--http-insecure	TIS_HTTP_INSECURE	Skip TLS certificate verification (insecure). The variable accepts 1, true, yes, on (case-insensitive) (since 1.22.0).
--force		Ignore the stored ETag: download and compare even if the server reports no change.
--dry-run, -q, -v, --color		Preview, quiet, verbose, colours.

Behaviour worth knowing:

• Without --group/--source it pulls the public export (/export).
• Network errors, 5xx and 429 are retried with back-off; other 4xx (bad token, no access) abort immediately.
• ETag: after a deploy the script stores the server's ETag in <destination>.etag together with the file's checksum and sends it as If-None-Match next time. 304 Not Modified → nothing is downloaded, written or reloaded. A list changed by hand no longer matches the checksum and is downloaded in full.
• Unchanged content (ignoring the timestamp and SOA serial lines, in every format) → nothing is written or reloaded, exit 0. Before 1.16.1 hosts lists were redeployed (and the service reloaded) on every run.
• Concurrent runs for the same destination are serialised with flock; file permissions and ownership are kept.
• Exit codes: 0 success/no change, 1 usage or download error, 2 reload failed (rolled back) or a table load failed (previous content kept), 3 validation failed (nothing deployed).

Deployment targets

Since 1.20.0 of the script. One option replaces the hand-written reload commands and pipes. Explicit --format, --reloadcmd, --revertcmd and --checkcmd still win.

Firewall tables – the download goes into a temporary table that is swapped in at once, so the live table is never empty or half-filled; if loading fails, the old content stays. Only IP addresses and networks are loaded (domains and URLs in the export are skipped). The loaded list is kept in --state-dir for the ETag and the comparison, so the table is touched only when the list changed – or when it doesn't exist (e.g. after a reboot).

Option	What it does
--ipfw-table <name>	ipfw table <name> create … missing (--ipfw-legacy: plain create, flush the temp table), fill <name>_TEMP in one batch, swap, destroy. --table-opts default type addr. IPv4 and IPv6 in one table.
--ipfw-rule <number>	ipfw add <number> deny log logamount 0 ip from "table(<name>)" to me, no default
--pf-table <name>	pfctl -t <name> -T replace -f <list> (atomic, creates the table).
--ipset <name>	ipset restore into <name>_TEMP, swap, destroy. One family per set: --type ipv4 or ipv6. --table-opts default hash:net family inet\|inet6.

DNS servers – the new file is validated before it is deployed (a broken file never reaches the server), then the service is reloaded; a failed reload is rolled back as usual. Need --destination.

Option	Format	Validation	Reload
--bind	zone (default)	named-checkconf <file>	rndc reconfig
--bind --format rpz	rpz	named-checkzone <zone> <file>	rndc reload
--unbound	unbound	unbound-checkconf <file>	unbound-control reload
--dnsmasq	dnsmasq	dnsmasq --test --conf-file=<file>	service dnsmasq restart

Keep the token out of the command line with a config file:

# /usr/local/etc/tis/tis.conf (chmod 600)
TIS_TOKEN="..."

tis-push.sh

Pushes indicators from a file, URL or stdin into an import-only source (POST /api/v1/import/:id); the source's format decides how the text is parsed.

tis-push.sh --config tis.conf --source 7 --tag honeypot --input /var/log/honeypot/ips.txt
some-command | tis-push.sh --token "$TOKEN" --source 7

Option	Env	Description
--source <id> *	TIS_SOURCE_ID	Target source.
--token *	TIS_TOKEN	Token with import/<id>.
--input <path\|url> or positional	TIS_INPUT	Data to push (default stdin).
--tag	TIS_TAG	Tags for the imported indicators.
--config, --api-base, --retries, …		As for tis-pull.sh (API base default …/api/v1/import).

Pushed indicators expire after the retention period unless pushed again – push regularly from cron.

tis-hits.sh

Since v1.34.0. Reports how often the list blocked something to POST /api/v1/hits, for the statistics on the Dashboard and the indicators. It reads the BIND RPZ log (category rpz) and counts each rewrite for its trigger – the listed entry, so the counts match the indicators: *.evil.example → evil.example, RPZ IP triggers (rpz-ip, rpz-client-ip, rpz-nsip) → the address or network. PASSTHRU rewrites (whitelist) and other zones are not counted.

tis-hits.sh --token "$TOKEN" --log /var/log/named/rpz.log --zone tis-blocklist.rpz

Since 1.1.0 of the script it also reads the ipfw log (--log-format ipfw, e.g. /var/log/security): the packets logged by the given rules, each counted for the entry of the ipfw table its source or destination matches (ipfw table … lookup) – a listed network collects the hits of all its addresses, and it doesn't matter whether the rule matches the source or the destination. Without --ipfw-table (e.g. when testing on another machine) the address on --side is counted. last message repeated N times from syslog counts N times.

tis-hits.sh --token "$TOKEN" --log /var/log/security --log-format ipfw --rule 4990 --ipfw-table tis-blocklist

Option	Env	Description
--token *	TIS_TOKEN	API token with the hits ACL (in the guest default).
--log *	TIS_HITS_LOG	BIND log file with the rpz category, or the ipfw log.
--log-format	TIS_HITS_FORMAT	rpz (default) or ipfw.
--zone * (rpz)	TIS_RPZ_ZONE	The RPZ zone of the TIS list as named in named.conf; comma-separated for more.
--rule * (ipfw)	TIS_IPFW_RULE	Number(s) of the logging rule(s), comma-separated.
--ipfw-table (ipfw)	TIS_IPFW_TABLE	Count for the matching entry of this table; other packets are skipped.
--side <src\|dst> (ipfw)	TIS_IPFW_SIDE	Without --ipfw-table: count the source or destination address.
--reporter	TIS_REPORTER	The machine's name in the statistics (default: hostname).
--state-dir	TIS_STATE_DIR	Where the read position is kept (default /var/db/tis or /var/lib/tis).
--from-start		On the first run count the whole log (default: start at its end).
--dry-run		Print the counts, send nothing, keep the position.
--config, --api-base, --retries, …		As in tis-push.sh.

Only new lines are read: the position (inode and offset) is saved after the report was accepted, so a failed run is simply repeated; a line still being written waits for the next run, and a rotated log is detected and read from its start. Run it as a tis-cron.sh step, e.g. every 5 minutes – the hits count in the hour of the report.

BIND has to log the rpz category to a file, e.g.:

logging {
    channel rpz_log { file "/var/log/named/rpz.log" versions 3 size 20m; print-time yes;
                      print-category yes; print-severity yes; severity info; };
    category rpz { rpz_log; };
};

For ipfw the rule has to log and the kernel messages end up in security.log (the FreeBSD default with net.inet.ip.fw.verbose=1); mind net.inet.ip.fw.verbose_limit, which stops logging a rule after that many packets until ipfw resetlog:

ipfw add 4990 deny log ip from table(tis-blocklist) to any
ipfw add 4991 deny log ip from any to table(tis-blocklist)

tis-cron.sh

A wrapper for cron that runs several steps, logs them and e-mails the transcript when a step fails (TIS_CRON_NOTIFY_EMAIL, needs mail).

Since v1.35.0 the steps are not in the script itself but in tis-cron.d/ next to it: every *.sh file there is run in name order, so updating tis-cron.sh never overwrites them. Start from tis-cron.d/example.sh.sample (*.sample files are not run):

# /opt/tis/tis-cron.d/10-blocklists.sh
run_step "pull zone blocklist (group customer-x) -> named" \
    tis-pull.sh --config 'tis.conf' --group 'customer-x' --type 'fqdn' \
        --destination '/usr/local/etc/namedb/master/tis-blocklist.zone' --bind

run_step "pull ipv4 blocklist (source 2) -> ipfw table tis-blocklist" \
    tis-pull.sh --config 'tis.conf' --source '2' --type 'ipv4' --ipfw-table 'tis-blocklist'

run_step <description> <command…> runs a command, run_pipe <description> '<a | b>' a pipeline that fails when any part of it fails. --steps <dir|file> (or TIS_CRON_STEPS) runs another directory or a single file – e.g. the hits more often than the lists:

*/15 * * * * /opt/tis/tis-cron.sh --steps /opt/tis/tis-cron.d/10-blocklists.sh
*/5  * * * * /opt/tis/tis-cron.sh --steps /opt/tis/tis-cron.d/50-hits.sh

Without any steps the run fails with an error. Upgrading from an older version: steps you had edited into tis-cron.sh are in the installer's backup tis-cron.sh.bak.<timestamp> – move them into a tis-cron.d/*.sh file.

Healthcheck

Since v1.28.0. tis-cron.sh can report every run to a Healthchecks.io-compatible monitor, so a cron job that stops running or keeps failing raises an alert:

*/15 * * * * /opt/tis/tis-cron.sh --healthcheck-url 'https://hc-ping.com/<uuid>' --healthcheck-qs "?rid=$(uuidgen)"

Option	Env	Description
--healthcheck-url <url>	TIS_CRON_HEALTHCHECK_URL	GET <url>/start before the steps, POST <url>/log with each step's output, finally POST <url>/<exit-code> with the whole transcript (0 = success).
--healthcheck-qs <qs>	TIS_CRON_HEALTHCHECK_QS	Appended verbatim to every ping URL – include the leading ? or &. ?rid=<uuid> ties the pings to one run.
--notify-email <addr>	TIS_CRON_NOTIFY_EMAIL	Mail the transcript when a step fails.

• Uses curl (or wget); a ping that can't be sent is logged as a warning and never fails the run.
• Bodies are cut to their last 100 kB (the Healthchecks.io limit), so the end of the output – where errors are – is kept.
• If the script dies unexpectedly, the failure is still reported with its exit code.

Integrations

Target	Format	Notes
BIND RPZ	rpz	response-policy { zone "rpz.tis"; }; + zone file from --destination; --bind --format rpz.
BIND empty zones	zone, type=fqdn	include the file in named.conf; provide master/empty.db; --bind.
Unbound	unbound (or rpz with the rpz: module)	include: <file>; --unbound.
nftables	nftables	--reloadcmd 'nft -f <file>'; reference @blocklist_v4 / @blocklist_v6 in your rules.
iptables + ipset	plain, type=ipv4/ipv6	--ipset <name>; match with -m set --match-set <name> src. (The ipset format with --reloadcmd 'ipset restore < <file>' still works.)
dnsmasq	dnsmasq	conf-file=<file>; --dnsmasq.
ipfw / pf	plain	--ipfw-table <name> / --pf-table <name>. (The ipfw format with --reloadcmd 'ipfw -q /absolute/path/to/file' loads the fixed table tis-blocklist.)
/etc/hosts, Pi-hole	hosts