Migration from the scripts¶

The agent does what tis-cron.sh with its tis-cron.d steps did: each tis-pull.sh or tis-push.sh step becomes a job, the tis-hits.sh steps become hits.logs, the cron line becomes the jobs' every. There is no automatic conversion – the steps are shell code – but the options map one to one. This page shows the mapping and a whole tis-cron.d converted.

Before you start¶

The token – the scripts' TIS_TOKEN (usually in tis.conf) works as it is; it goes to platform.token. Pushing needs import/<source>, hits need hits in its ACL.
Commands are lists, not shell strings – --reloadcmd 'rndc reload' becomes "reload": ["rndc", "reload"]. Anything using the shell (|, &&, >, variables) becomes ["sh", "-c", "…"]. The placeholder {} of --checkcmd is {file}.
One job per target – a file or table is written by one job only.
No pipelines and no e-mail – run_pipe steps and tis-pull.sh without --destination (to stdout) have no job equivalent: deploy to a file target and post-process in its reload command, or produce content for a push with a command. Instead of --notify-email use a healthcheck – Healthchecks.io can e-mail.
No shared state – the agent keeps its own state (/var/db/tis-agent, /var/lib/tis-agent), apart from the scripts' (/var/db/tis, /var/lib/tis). Its first run downloads every list in full: unchanged files stay untouched (no reload), each table is loaded once.

tis-pull.sh → a pull job and its target¶

tis-pull.sh	Agent
`--group <ref>`	`pull.list`
`--source <refs>`	`pull.sources` (a list)
neither	neither – the public export
`--format`, `--type`, `--tag`	`pull.format`, `pull.type`, `pull.tag`
`--whitelist a,b`	`pull.whitelist: ["a", "b"]`
`--country`, `--exclude-country`	`pull.country`, `pull.exclude_country` (lists)
`--rpz-ip <kind>`	`pull.rpz_ip`
`--destination <path>`	a target of type `file` (or `bind`, `unbound`, `dnsmasq`) with `path`
`--checkcmd`, `--reloadcmd`, `--revertcmd`	the target's `check`, `reload`, `revert` (lists)
`--bind`, `--unbound`, `--dnsmasq`	target `type` `bind`, `unbound`, `dnsmasq` (same default commands)
`--ipfw-table <name>`	target `{"type": "ipfw-table", "table": "<name>"}`
`--ipfw-rule <n>`, `--ipfw-legacy`	the target's `rule`, `legacy`
`--pf-table <name>`, `--ipset <name>`	target type `pf-table`, `ipset` with `table`
`--table-opts '<opts>'`	the target's `create_options` (a list)
`--format ipset` + `--reloadcmd 'ipset restore …'`	an `ipset` target (format `plain`)
`--format nftables` + `--reloadcmd 'nft -f …'`	`nftables-set` targets (format `plain`), one per family – the export's fixed `inet tis` sets `blocklist_v4`/`_v6` become any sets you name
`--force`, `--dry-run`	`tis-agent run -force`, `-dry-run`
`--state-dir`	`state_dir`
`--api-base https://…/api/v1`	`platform.url` without `/api/v1`
`--retries`, `--retry-delay 2`	`platform.retries`, `platform.retry_delay: "2s"`
`--connect-timeout 15`, `--max-time 120`	`platform.connect_timeout: "15s"`, `platform.timeout: "2m"`
`--http-insecure`	`platform.insecure`
`--config`, `--color`, `-q`, `-v`	– (one configuration file; the log goes to syslog or the journal)

The exit codes of tis-agent run are those of tis-pull.sh (0, 1, 2, 3).

tis-push.sh → a push job¶

tis-push.sh	Agent
`--source <ref>`	`push.source`
`--input <file>` or `[file]`	`push.file`
`--input <url>`	`push.url`
stdin	`push.command` – the command whose output was piped in
`--tag`	`push.tag`
`--dry-run`	`tis-agent run -dry-run <job>`

tis-push.sh sent the content on every run; the agent sends it when it changed and otherwise once a refresh (default a day), which keeps the indicators from expiring.

tis-hits.sh → hits.logs¶

tis-hits.sh	Agent
`--log <path>`	`hits.logs[].path`
`--log-format rpz`, `--zone a,b`	`"format": "rpz"`, `"zones": ["a", "b"]`
`--log-format ipfw`, `--rule 4990,4991`	`"format": "ipfw"`, `"rules": [4990, 4991]`
`--ipfw-table <name>`	`table`
`--side src\\|dst`	`side`
`--from-start`	`from_start`
`--reporter`	`reporter` (top level, for all hits)
`--state-dir`	`state_dir`
`--dry-run`	`tis-agent hits -dry-run`
the cron interval of the step	`hits.interval` (one for all logs)

The read positions don't carry over: the agent starts at the end of each log (unless from_start), so stop the script's cron line and start the agent at about the same time.

tis-cron.sh → the daemon¶

tis-cron.sh	Agent
the crontab line (`/15 * * *`)	each job's `every` (`"15m"`)
several cron lines with `--steps`	different `every` per job
`--healthcheck-url <url>`	`healthcheck` per job (and `hits.healthcheck`)
`--healthcheck-qs "?rid=…"`	– (each job has its own check)
`--notify-email`	– (a healthcheck that e-mails)
`TIS_NO_PULL`, `TIS_NO_PUSH`, `TIS_NO_HITS`	– (leave a job without `every` and run it by hand)

Example¶

The steps of tis-cron.d/example.sh.sample, run every 15 minutes with a healthcheck:

*/15 * * * * /opt/tis/tis-cron.sh --healthcheck-url 'https://hc-ping.com/<uuid>'

run_step "push file.txt to source 1" \
    tis-push.sh --config 'tis.conf' --source '1' --input 'http://url.to/file.txt'

run_step "pull zone blocklist (group customer-x) -> named" \
    tis-pull.sh --config 'tis.conf' --group 'customer-x' --type 'fqdn' \
        --destination '/usr/local/etc/namedb/master/tis-blocklist.zone' --bind

run_step "pull ipv4 blocklist (source 2) -> ipfw table tis-blocklist" \
    tis-pull.sh --config 'tis.conf' --source '2' --type 'ipv4' --ipfw-table 'tis-blocklist'

run_step "report RPZ hits -> TIS" \
    tis-hits.sh --config 'tis.conf' --log '/var/log/named/rpz.log' --zone 'tis-blocklist.rpz'

run_step "report ipfw hits -> TIS" \
    tis-hits.sh --config 'tis.conf' --log '/var/log/security' --log-format 'ipfw' \
        --rule '4990' --ipfw-table 'tis-blocklist'

become /usr/local/etc/tis-agent.json:

{
  "platform": { "token": "<the TIS_TOKEN of tis.conf>" },
  "targets": {
    "zone":      { "type": "bind", "path": "/usr/local/etc/namedb/master/tis-blocklist.zone" },
    "blocklist": { "type": "ipfw-table", "table": "tis-blocklist" }
  },
  "jobs": [
    { "name": "file-txt", "every": "15m",
      "push": { "source": "1", "url": "http://url.to/file.txt" } },
    { "name": "zone", "every": "15m",
      "pull": { "list": "customer-x", "type": "fqdn", "target": "zone" },
      "healthcheck": "https://hc-ping.com/<uuid-zone>" },
    { "name": "blocklist", "every": "15m",
      "pull": { "sources": ["2"], "type": "ipv4", "target": "blocklist" },
      "healthcheck": "https://hc-ping.com/<uuid-blocklist>" }
  ],
  "hits": {
    "interval": "15m",
    "logs": [
      { "format": "rpz", "path": "/var/log/named/rpz.log", "zones": ["tis-blocklist.rpz"] },
      { "format": "ipfw", "path": "/var/log/security", "rules": [4990], "table": "tis-blocklist" }
    ]
  }
}

The bind target uses the format zone and named-checkconf / rndc reconfig as --bind did; give each job you want monitored its own check.

Switching over¶

Install the agent package (Installation).
Write the configuration, then check it and look at what the first run would do:
```
tis-agent check
tis-agent run -dry-run
tis-agent hits -dry-run
```
Comment out the tis-cron.sh line(s) in the crontab.
Start the service (service tis_agent start or systemctl enable --now tis-agent) and watch tis-agent status and the log (/var/log/security or journalctl -u tis-agent).
Once it runs well, remove the scripts (/opt/tis) and their state (/var/db/tis or /var/lib/tis).