Groups¶

Since v1.25.0.

A group is a named set of sources that are exported together, with the group's own whitelist (never exported) and blacklist (always exported). Both lists take literal values and whole sources, and every source a group references can be narrowed by indicator tags. Client machines reference only the group, so changing what a customer gets is one change on the server instead of an edit in every machine's cron.

Before – source IDs and whitelist repeated on every machine:

tis-pull.sh --source '3,5' --format zone --type fqdn \
  --whitelist 'eu-central-1.amazonaws.com,s3.amazonaws.com,storage.googleapis.com' ...

After – the machine pulls the group; sources and lists live on the server:

tis-pull.sh --group customer-x --format zone --type fqdn ...

Managing groups¶

group:create --name customer-x --description "Customer X resolvers" \
  --sources malwarepatrol-hosts,dnjp-hosts \
  --whitelist eu-central-1.amazonaws.com,s3.amazonaws.com,storage.googleapis.com \
  --blacklist always-block.example,203.0.113.0/24

group:update --group customer-x --add-source urlhaus-hosts          # one more source for all machines
group:update --group customer-x --remove-whitelist storage.googleapis.com
group:update --group customer-x --sources 3,5                       # replace the member list
groups                                                              # list
group:detail --group customer-x                                     # members, lists, export URL, tis-pull call
indicators:export --group customer-x --format zone --type fqdn      # preview the export
group:delete --group customer-x                                     # sources are kept

Groups and sources can be referenced by ID or name.
sources, whitelist and blacklist can each be replaced (--sources, --whitelist, --blacklist; "" clears) or edited (--add-…, --remove-…).
Whitelist entries: IPs, CIDRs and domain globs (*.example.com). Blacklist entries: IPs, CIDRs, domains or URLs. Invalid entries are rejected; both lists are stored normalised (lowercased, deduplicated).
Unknown sources are reported and nothing is changed.

Sources in the whitelist and blacklist, tag and age filters¶

Since v1.29.0. Besides literal values, the whitelist and blacklist can reference whole sources: the indicators of a whitelist source are never exported for the group, those of a blacklist source always are. A source may be a member and in the whitelist or blacklist at the same time.

Every referenced source – member, whitelist or blacklist – can be narrowed by tags, written in brackets after the source:

Reference	Takes from the source
`urlhaus`	every indicator
`urlhaus[phishing]`	indicators tagged `phishing`
`urlhaus[phishing,c2]`	indicators tagged `phishing` or `c2`
`urlhaus[!test]`	every indicator except those tagged `test`
`urlhaus[phishing,!test]`	tagged `phishing`, but not tagged `test`

A tag counts whether it is set on the indicator or on its source, as with the export's ?tag=.

Since v1.32.0 a source can also be limited to recently observed indicators with < and a duration – s, m/min, h, d, w:

Reference	Takes from the source
`urlhaus[<8h]`	indicators last observed (fetched or pushed) within 8 hours
`urlhaus[phishing,<1d]`	tagged `phishing` and observed within a day

This keeps a group to what a feed currently lists even when the feed's retention is longer. It works for members as well as whitelist and blacklist sources.

group:create --name customer-x \
  --sources 'malwarepatrol-hosts,urlhaus[phishing,!test]' \
  --whitelist-sources 'google-ips' \
  --blacklist-sources 'manual-blocks[customer-x]'

group:update -g customer-x --add-whitelist-source 'office-ranges'
group:update -g customer-x --add-source 'urlhaus[!test]'     # a listed source: replaces its filter

The options --whitelist-sources / --blacklist-sources (and --add-…-source, --remove-…-source) work like --sources. In the API the lists are whitelistSources and blacklistSources; entries are refs in the syntax above or objects { "source": "urlhaus", "tags": ["phishing"], "excludeTags": ["test"], "maxAge": "8h" } (maxAge is returned as written, e.g. "8h"). In the GUI, each picked source has two tag fields, Only tagged… and Except tagged…, and a max age field.

API: /api/v1/groups – see API Reference.

What a group export contains¶

GET /api/v1/export/group/<id|name> (and indicators:export --group):

indicators of all member sources (each narrowed by its tag and age filter) – private sources included (membership is the authorisation) – deduplicated,
minus the whitelists: the group whitelist (values and whitelist sources), the ?whitelist= of the request and each source's own options.whitelist,
plus the group blacklist (values and blacklist sources).

Whitelist-source indicators behave exactly like literal whitelist entries: an IP or network covers indicators inside it, a domain matches exactly, and in RPZ, Unbound and dnsmasq they are emitted as passthru entries.

The format, type and tag parameters work as for every export. Blacklist entries honour type and the whitelists too: if a value is both blacklisted and whitelisted, the whitelist wins. A group without sources exports just its blacklist.

Countries¶

A group can keep only the IP indicators of some GeoIP countries, or drop those of others – e.g. a customer who doesn't want to block addresses in their own country:

group:update --group customer-x --countries '!CZ'
group:update --group customer-y --countries CZ,SK

In the API: countries and excludeCountries (lists of ISO codes; addCountries, removeExcludeCountries … edit them). Domains, URLs and JSON indicators aren't affected, and the blacklist isn't filtered. ?country= / ?exclude_country= of the export request apply on top.

Access for customers¶

Give each customer a token that can pull only their group:

token:create --name customer-x-pull --role guest --acl '["export/group/customer-x"]'

The rule may use the group's ID or name; both forms of the URL are allowed either way. Renaming the group rewrites rules that name it, so the token keeps its access. A token without access gets 403 – also for groups that don't exist, so tokens can't discover other customers' groups. The user role may pull every group (export/group/*); managing groups needs the groups ACL (admin by default). See Tokens & ACL.