Sources¶

A source is where indicators come from. Every indicator belongs to exactly one source; the same value may exist in several sources and is deduplicated on export.

Source kinds¶

Kind	How it gets data	Pruned?	Create with
Feed	Fetched from `url` on its cron `schedule`, or on demand.	yes	`--url … [--schedule …]`
Import-only	Pushed via `tis-push.sh` / `POST /api/v1/import/:id`; no URL.	yes – stops being exported when no longer pushed	no `--url`
Manual	Hand-curated with `indicator:add` / `POST /api/v1/indicators`. Never fetched.	no	`--manual`

Rules: a schedule requires a url; a manual source has neither. Any source can additionally be private (--private): its indicators are left out of the public export and are only available through per-source, multi-source or group exports.

Changes of a source (create/update/delete via CLI or API) are signalled to the worker through Redis, so new or changed schedules apply immediately. Without Redis, restart the worker or run scheduler:resync.

Creating sources¶

# feed fetched every hour
source:create --name urlhaus-hosts --url https://urlhaus.abuse.ch/downloads/hosts/ \
  --format hosts --tags malware --schedule "0 * * * *"

# feed with authentication and a timeout
source:create --name premium-feed --url https://api.example.com/v1/malware.txt \
  --schedule "*/30 * * * *" \
  --options '{"headers":{"Authorization":"Bearer <secret>"},"timeout":10000}'

# import-only, private (fed by tis-push.sh)
source:create --name internal-honeypot --private

# manual, private blocklist for one customer
source:create --name customer-x-blocklist --manual --private

Names are slugs (a-z, 0-9, -), unique among existing sources (a deleted source's name can be reused). Schedules are cron expressions in server time (UTC in Docker); see crontab.guru – or presets, below.

Schedule presets¶

Since v1.32.0. Instead of a cron expression, a schedule can be a preset:

Preset	Runs
`@every 15m`, `@every 30m`	every 15 / 30 minutes
`@hourly`	once an hour
`@every 2h`, `@every 3h`, `@every 4h`, `@every 6h`, `@every 8h`, `@every 12h`	12×, 8×, 6×, 4×, 3×, 2× a day
`@daily`, `@weekly`	once a day / a week

The scheduler turns a preset into cron at a minute (and, for the longer ones, an hour or weekday) derived from the source name, so a hundred hourly feeds don't all fetch at :00 – e.g. @every 6h for urlhaus runs as 14 2-23/6 * * *. The time stays the same until the source is renamed. source:detail and the GUI show the resulting cron; the API's GET /sources/schedule?schedule=@every 6h&name=urlhaus returns it with the next run.

source:update --id 3 --schedule "@every 6h"

Retention and purge¶

Since v1.32.0. Indicators a source no longer lists are removed after CONFIG_SCHEDULER_DAYS_RETENTION days and deleted for good CONFIG_SCHEDULER_DAYS_PURGE days later (see Configuration). A source can set its own periods – e.g. a feed fetched every hour, whose stale entries should go after a day:

source:update --id 3 --days-retention 1 --days-purge 7
source:update --id 3 --days-retention default     # back to the configured default

daysPurge: 0 never purges the source's removed indicators. In the API the fields are daysRetention and daysPurge (null = the default, GET /sources/defaults returns the defaults); the GUI shows the defaults as placeholders. Manual sources are never pruned. The former options.retentionDays was moved to daysRetention by the migration and is still accepted.

Other commands: sources (list), source:detail --id, source:update --id … [--manual|--no-manual] [--private|--no-private], source:delete --id, source:fetch --id [--dry-run]. See CLI Reference.

Deleting a source soft-deletes it together with its indicators; they disappear from all exports.

Formats¶

The format decides how fetched (or imported) text is turned into indicator values. Each value is then classified:

Value looks like	Indicator type
`http://…` / `https://…`	`url`
IPv4 address, optionally `/prefix`	`ipv4` (default netmask 32)
IPv6 address, optionally `/prefix`	`ipv6` (default netmask 128)
domain name (`a.example.com`, trailing `.` allowed)	`fqdn`, lowercased
JSON object (json format only)	`json`
anything else	skipped (counted as `skipped`)

`plain`¶

One value per line; the first token of a line (split on whitespace or ,) is used. Empty lines and lines starting with # are ignored; text after # is a comment.

# comment
1.2.3.4
10.0.0.0/8      # inline comment
evil.example.com, extra columns are ignored

`hosts`¶

/etc/hosts style: every name after the first (address) column is a value.

0.0.0.0 evil.example.com www.evil.example.com
127.0.0.1 tracker.example.net   # comment

`json`¶

options.parser is a jq expression returning the values, e.g. map(.ioc) or .data[].indicator. Required for this format.

`csv`¶

Parsed with csv-parse; configure under options.csv:

Key	Default	Meaning
`column`	`0`	Header name or 0-based index of the value column.
`header`	`true`	First row is a header. Use numeric `column`/`tags` when `false`.
`tags`	`false`	Header name or index of a column with comma/space-separated tags.
`delimiter`	`,`	Field delimiter.
`comment`	`#`	Comment prefix.
any other	–	Passed to csv-parse as-is.

Per-item tags from the feed¶

Indicators get the tags of their source automatically (on export and in filters). Feeds can also carry tags per item:

Format	`options.tags`
`plain`, `hosts`	`true` – the inline `#` comment of a line is a comma/space-separated tag list: `1.2.3.4 # malware, c2`.
`json`	jq expression yielding one entry per item (in `parser` order): a tag, an array of tags or `null`, e.g. `map(.tags)`.
`csv`	use `options.csv.tags` instead.

Tags are added to an indicator's existing tags, never replacing them.

Ports can be tagged by service name with @service-<port>/<protocol> (see Indicators), e.g. for the Microsoft 365 endpoints feed (https://endpoints.office.com/endpoints/worldwide?clientrequestid=<uuid>):

{
  "parser": "x => x.flatMap(i => [...(i.ips || []), ...(i.urls || [])].map(e => e.replace('.*.', '.').split('*.').pop()))",
  "tags": "x => x.flatMap(i => { const t = ['o365-' + i.serviceArea.toLowerCase(), ...[['tcp', i.tcpPorts], ['udp', i.udpPorts]].flatMap(([k, p]) => (p || '').split(',').map(p => p.trim()).filter(Boolean).map(p => '@service-' + p + '/' + k))]; return [...(i.ips || []), ...(i.urls || [])].map(() => t); })"
}

13.107.6.152/31 then gets o365-exchange, http, https.

Options reference¶

--options / options is a JSON object:

Key	Applies to	Description
`headers`	fetch	Extra HTTP headers (e.g. `Authorization`, `Key`, `Accept`). Override the default `User-Agent: @capnet/threat-intel-scrany/<version>`.
`method`	fetch	HTTP method, default `GET`.
`body`	fetch	Request body (string).
`timeout`	fetch	Abort the request after this many milliseconds.
`parser`	json	jq expression extracting values.
`tags`	plain, hosts, json	Per-item tags, see above.
`csv`	csv	CSV settings, see above.
`expireOnMissing`	fetch	`true` for feeds listing their current state: after a successful fetch, indicators the feed no longer contains are removed right away (skipped when a fetch yields nothing, so an empty response can't wipe the source).
`whitelist`	export	Array of IPs/CIDRs/domain globs never exported from this source; see Export.

Options may contain secrets (API keys in headers). sources lists only option names; source:detail shows the values.

Fetching and troubleshooting¶

source:fetch --id 3 --dry-run downloads and parses the feed and shows a table of what would be stored (normalised), without writing.
source:fetch --id 3 runs a real fetch now (same as the scheduled one). API: POST /api/v1/sources/3/fetch.
A fetch that fails (HTTP error, timeout, parse error) is logged ([CRON FAILED]) and reported to Sentry; the previous indicators stay until they expire.
Overlapping runs of the same source are skipped while the previous one is still running.

Fetch status¶

Since v1.26.0. Every fetch attempt – scheduled, source:fetch, API or GUI – is recorded on the source: when it ran, success or error, the error message, how long it took and how many indicators it yielded. lastFetchedAt keeps the time of the last successful fetch.

Field	Meaning
`lastFetchedAt`	last successful fetch
`lastFetchAt`	last attempt
`lastFetchStatus`	`success` or `error`
`lastFetchError`	error message, or the warning of a suspicious success
`lastFetchDurationMs`	duration of the last attempt
`lastFetchItems`	indicators in the last successful fetch

A fetch that succeeds but returns no indicators, or more than CONFIG_FETCH_DROP_WARNING_PERCENT (default 50 %) fewer than the previous one, is usually a broken feed (expired subscription, changed format, error page served with HTTP 200). It is logged as [FETCH WARNING], reported to Sentry as a warning and kept in lastFetchError; the indicators are still processed normally.

source:list shows the status in the Status column (ok, warning, error), source:detail shows the last attempt and its message, and the GUI marks failing and shrunk sources on the Sources page and the dashboard.