sniffarr

sniffarr is a private classification database of BitTorrent torrents discovered on the DHT, with an API to search it. Its web UI is a catalogue of movies and series as TMDB knows them, each with the releases of it the DHT carries; a search looks in the database and on TMDB together, so it also finds what has no release yet.

The database and the API are meant to stay private: run them for yourself, not as a public index. Peer IP addresses are never sent to the server or stored: peers are counted in HyperLogLog sketches of keyed hashes, from which no address comes back. The sniffer fetches metadata only: it never downloads pieces, never uploads, and never announces itself into a swarm. The fetcher does so only with -probe-dir: it downloads the first pieces of a stored release's main video file to read its tracks, and seeds them back; see Probes.

Components

sniffarr is one binary whose components are switched on by flags, any number of them in one process; with none on, it exits with a usage error. It runs until SIGINT or SIGTERM.

• -api is the centre of a work queue. It queues the infohashes sniffers sight, hands them out as fetch tasks, stores the metadata that comes back in SQLite, classifies stored torrents with TypeSafe's Jev model, and serves the search API and the Torznab indexer. It is the only component that touches the database.
• -ui serves the web UI, a client of the browse API.
• -sniff walks the DHT for infohashes, by BEP 51 sample_infohashes and by the announce_peer queries its own DHT node receives, and reports them to the ingest API, with the announcing peers counted in sketches. Its node answers the queries of other nodes in turn.
• -fetch takes tasks from the ingest API, finds each torrent's peers with get_peers on its own DHT node, fetches the metadata from them by BEP 9, and returns the name, size and file list, with the peers counted in sketches. With -probe-dir it also takes probe tasks, reading the tracks of stored releases.

-ui, -sniff and -fetch are clients of the API. In a process with -api they reach it over an in-process connection; without -api they dial SNIFFARR_URL, and fail at startup when it is unset. Any number of sniffing and fetching processes can run against one API. -sniff and -fetch in one process share one DHT node.

The queue serves the infohashes Radarr or Sonarr grabbed first, then announced ones before sampled ones, then the oldest first. It lives in the same SQLite database and survives restarts; a task whose fetcher goes away, or whose lease expires, goes back to the queue. A task whose fetcher finds no peers, or fewer than its -min-peers, is dropped at once. A task whose metadata times out is retried after a backoff and dropped after a few attempts. The queue is bounded: once a minute the API evicts the infohashes that have waited too long, then the ones beyond the size limit, sampled ones before announced ones and the oldest first. A dropped or evicted infohash is kept out of the queue for a while, a shorter one when it had no peers, then forgotten. It is not rejected: an announce or a Radarr or Sonarr event queues it again at once, and a sample once it is forgotten.

The server passes fetched metadata through a first filter by name and file extensions, rejecting what is never worth classifying: adult content, software and games, books and comics, courses, torrents under 50 MiB, torrents with neither video nor audio files, and torrents whose names have a letter outside the allowed scripts, Latin and Cyrillic by default. Only the name is judged by script, not file paths; digits, punctuation, symbols, and letters of the Common and Inherited scripts always pass. A rejected infohash is remembered with its reason only, so it is never queued again.

Accepted torrents are classified in layers, cheapest first, and every release records which layer decided its kind:

1. Rules read the name: episode and season markers, in Latin and Cyrillic (S01E02, 1x05, Season 3, Сезон: 1, Серии: 1-8 из 8), make a series; a title with a year and a quality tag makes a movie. A name that says neither leaves the kind unknown.
2. Reuse: the rules also reduce the name to a title key, the series title or the title and year. A torrent of unknown kind takes the kind of a stored torrent with the same key, now or once one is classified. Only the kind is shared; audio and the video source belong to each release.
3. TypeSafe's Jev examines a torrent for audio in each audio language, video source, and kind when it is still unknown. It does so lazily, for the candidates of a search before the answer, and in the background only for torrents of unknown kind whose names carry a marker or a script hint of an audio language. Both share a daily token budget, 2,000,000 tokens by default, some 1,500 torrents a day; the background stops at 75% of it by default, leaving the rest for searches. A search waits for Jev for a short while at most; answers that come later are recorded for the next search, and a slow or failing Jev never fails a search.

Adult and software kinds never show in search results. Music, books and software are classified like any other kind, and the web UI shows them only as "other" among the torrents: they have no title page and no top.

The rules also read a release's features from its name: the year, the season, the episode (S02E05, 1x05, E05, Серия 5, 5 серия; none for a whole season or a range such as S1E1-8 or 1-8 серии) and the quality, by the resolution the name gives: 2160p for 2160p, 4K or UHD, 1080p for 1080p or 1080i, 720p, and sd for 576p, 480p, SD or an SD source such as DVDRip; none when the name gives no resolution. A release stored before episodes and qualities were read gets them once, at the first start that has them.

Audio is an extension over the layers, off until SNIFFARR_AUDIO_LANGUAGES names languages; ru is the only language built in. For each configured language every stored torrent records a marker, a word or studio of the language in its name or file paths (Rus, Дубляж, LostFilm for ru), and a script hint, letters of the language's script there (Cyrillic for ru). A marker sets the probability of an audio track in the language to 1; otherwise Jev answers it, one question per language. A marker records its mark as written in the name, or else in a stored file path: a studio when there is one, wherever it stands, else a word, so Silo.S03E10.rus.LostFilm marks LostFilm; stored markers get their marks recomputed once, at the first start of a version whose marking changed. While the extension is off, nothing about audio is recorded or returned, Jev is asked about kind and source only, the background classifier has nothing to select, BrowseService.Settings returns no languages, and the web UI shows no voice filter, no per-language columns in search results and tops, and no audio rows on a release's card.

TMDB is a second extension, on by default with the TMDB read access token sniffarr ships with; TMDB_API_KEY replaces it with a v3 API key or a read access token of your own, and set empty turns the extension off. It matches title keys, not torrents, to works, so every release under a key shares its match. A background matcher walks the keys of movies and series without a match, newest torrents first, and a search matches the unmatched keys of its candidates within the same wait as its classification. Each title of the newest torrent's name, both in a Русское / Original name, is searched on TMDB as a movie or a series after the torrent's kind, in English and, when it has Cyrillic letters, in Russian too. A candidate fits when one of its titles equals one of the name's, ignoring case and punctuation, and, when the key has a year, its year is the key's or one off. One fit is the match; among several, the one with a TMDB popularity of at least 20 and ten times every other's is; else Jev picks one or none, shown each fit's titles, year, language, countries, popularity, votes and the first sentence of its overview, within the classifier's daily budget and its search reserve. Without TYPESAFE_API_KEY several fits with no such favourite match nothing. A key that matched nothing is looked up again after SNIFFARR_TMDB_RETRY, and one that matched never is. Without the key nothing is matched and releases carry no IDs.

A work is a movie or a series on TMDB, stored once per TMDB type and ID, with or without releases. Title keys are matched to works many to one: the keys of Чужой (1979) and Alien.1979 both match the movie 348, and every release of either key belongs to it. A release belongs to the work its key is matched to, at its storing when the key is matched already and at the match otherwise; a release whose key is matched to nothing belongs to no work. The API names a work m and its TMDB ID for a movie and tv and its TMDB ID for a series, m348 or tv1399, as TMDB numbers the two apart.

A search of movies and series asks TMDB as well, through /3/search/movie and /3/search/tv in the request's language, both when the search names neither kind: two requests, under SNIFFARR_TMDB_RATE. An answer is kept 6 hours per kind and query, its words lowercase, and a search within them asks TMDB nothing. Every work an answer has is stored with the answer's title, original title, year, poster and popularity, its details left to the backfill.

A match recorded, the work's page is fetched twice, as en-US and ru-RU, each with its external IDs appended. The first answer gives the work's details: its type, original title, poster path, rating and votes, runtime (a typical episode's for a series), a series' status and seasons, its origin countries and original language, and its IMDb, TVDB (series) and Wikidata IDs; each answer gives its title, overview, genres and poster in that language, stored as en and ru texts as TMDB returns them, which is the original where TMDB has no translation. The API serves the texts and the poster of the request's lang; a language without a poster takes the other's, then the work's. A work's page is fetched once, for the first key matched to it. A work whose page could not be fetched, or that a search stored, keeps what it has, and the TMDB worker's backfill fetches the details of every work without them, those with releases first, then the oldest, under SNIFFARR_TMDB_RATE and SNIFFARR_TMDB_CONCURRENCY, until none is left; it looks again every 10 seconds. A page TMDB refuses to serve marks the work fetched with what a search told of it, so it is not asked about again. The web UI shows the details on a movie's or series' page, on the card, in the top and in search results, with posters from image.tmdb.org; every page carries the TMDB logo and the notice "This product uses the TMDB API but is not endorsed or certified by TMDB."

Peers are a third extension, always on. A peer is an IP:port; -sniff sees one in every announce_peer its node receives, -fetch in every get_peers answer and every connection of a fetch. Each client hashes a peer with SipHash-2-4 under a 128-bit key the API generated once and keeps in the store, and adds the hash to a HyperLogLog sketch of the infohash and the current UTC hour; every few minutes it sends the sketches to the API, which drops those of infohashes it has not stored and unions the rest into the stored sketch of the torrent and hour. Nothing else about a peer is kept, and a sketch gives nothing back but an estimate of how many distinct peers it saw, within a few percent. Hour sketches are kept 8 days, then unioned into day sketches kept 65 days; every count is over a sliding window of 24 hours, 7 days or 30 days ending now, the 30 days by the day beyond the hours kept.

Every 10 minutes the API ranks, for movies and for series and for each window, the 100 works whose releases the most distinct peers share, with the change against the window before, and again, for each configured audio language, narrowed to the releases with a marker of the language or a probability of at least 0.5 of an audio track in it, and to those with a marker. A release that belongs to no work is in no top; the episodes and seasons of a series all count towards it. It refreshes the distinct peers over the last 7 days of each torrent and each work at the same time, so a page can sort by them.

Probes

A probe reads the track list of a stored release's main video file from its container header, which lies in the file's first pieces. Only -fetch with -probe-dir probes, and the API hands out the tasks.

The fetcher finds the torrent's peers with get_peers, fetches its metadata by BEP 9, and picks the largest file ending in .mkv, .webm, .mp4, .m4v or .mov; for a season pack that is one episode. It reads the file through the pieces each read covers, downloading them from the peers and verifying each against its hash before reading it. In Matroska and WebM it reads the EBML header and the Segment's top-level element headers up to Tracks, reading no body but those of SeekHead and Tracks, and follows SeekHead when Tracks comes after the first Cluster; in MP4 and MOV it reads the top-level box headers up to moov. A probe that would download more than the task's max_bytes stops there, and the tracks stay unknown. Of each track it records the kind, the language as ISO 639-2 and as BCP-47, the name, the codec, an audio track's channels, and a video track's size and dynamic range: DV for a Dolby Vision configuration (Matroska's BlockAdditionMapping, MP4's dvcC, dvvC or dvwC), else HLG or HDR10 by the transfer characteristics, or HDR10 for mastering metadata.

Candidates are stored movies and series with a stored file of those formats and at least one distinct peer over the last 7 days. Releases a Torznab search with a query or an ID returned go first, the most recently returned first; then the rest, the most peers over 7 days first. A release without peers waits until it has some. Each release is probed once, and the outcome is stored, the tracks in their own table: read, unknown (no file of those formats, a track list beyond max_bytes, or a header the probe cannot read), or failed (fewer than -min-peers peers within -peer-wait, the metadata not within -fetch-timeout, or the pieces not before -probe-timeout). A failed probe is repeated after SNIFFARR_PROBE_BACKOFF, doubled for each failure before, until SNIFFARR_PROBE_ATTEMPTS have failed. A task is leased for SNIFFARR_PROBE_LEASE; leases live in the API's memory, so a restart frees them.

The API meters the piece data probes download per UTC day, as it meters the classifier's tokens: SNIFFARR_PROBE_DAILY_BYTES, 100 GiB by default. A lease counts as SNIFFARR_PROBE_MAX_BYTES until its result reports what it downloaded. Releases no search returned stop at 75% of the day's bytes by default, SNIFFARR_PROBE_SEARCH_RESERVE leaving the rest for searched ones.

With the audio extension on, a probe that read the tracks sets the probability of an audio track in each configured language, under layer probe: 1 when an audio track is in it; 0 when every audio track gives another language, unless a marker in the name decided it, since a release may carry that audio in a file of its own. A track that gives no language leaves the probability as it was. Jev does not override a probe.

What a probe downloaded is seeded back. The pieces stay in -probe-dir and the torrent in the client, which uploads them to its peers and announces itself on the DHT for the torrent every 30 minutes. The fetcher removes a torrent and deletes its pieces once it has uploaded as much as it downloaded; before that, once it has been seeded for -seed-max-age, or, oldest first, while the torrents kept hold more than -seed-max-size of piece data. It looks every minute, and tells the API by ProbeUpload what the torrents uploaded since the last look; a report the API refuses goes with the next. The seeded torrents, what each downloaded and uploaded, and which pieces are complete survive a restart. Metadata fetches upload nothing.

API

Both API groups are ConnectRPC services, defined in proto/sniffarr/v1. The server speaks HTTP/1.1 and HTTP/2 without TLS, so Connect, gRPC and gRPC-Web clients all work; Fetch streams both ways and needs HTTP/2 end to end. The services authenticate no caller: whoever reaches the port can call every RPC, so the network has to keep others out, as the chart's api.networkPolicy does.

Service	RPC
sniffarr.v1.IngestService	Sight
sniffarr.v1.IngestService	Fetch
sniffarr.v1.IngestService	HashKey
sniffarr.v1.IngestService	Peers
sniffarr.v1.IngestService	ProbeTask
sniffarr.v1.IngestService	ProbeResult
sniffarr.v1.IngestService	ProbeUpload
sniffarr.v1.SearchService	Search
sniffarr.v1.SearchService	Get
sniffarr.v1.BrowseService	SearchTitles
sniffarr.v1.BrowseService	Title
sniffarr.v1.BrowseService	TitleReleases
sniffarr.v1.BrowseService	Torrents
sniffarr.v1.BrowseService	Card
sniffarr.v1.BrowseService	Stats
sniffarr.v1.BrowseService	Home
sniffarr.v1.BrowseService	Top
sniffarr.v1.BrowseService	Settings

• Sight is client-streaming: it queues every infohash the server has neither stored, rejected nor queued, and returns how many were new and how many known, and, per infohash sent, whether it is a stored torrent.
• Fetch is bidirectional: the client grants credit, one unit per task it can take, the server answers with a task per unit, and the client returns each task's result, metadata or a failure, on the same stream.
• HashKey returns the 16-byte SipHash-2-4 key peers are hashed with.
• Peers is client-streaming: each message is an infohash, the start of a UTC hour and a sketch in the Aggregate Knowledge storage specification v1.0.0 (log2m 10, regwidth 5) of the peers seen sharing it in that hour, hashed with the key; the server returns how many sketches it merged and how many it dropped for an infohash it has not stored.
• ProbeTask leases the next release to probe and the most piece data its probe may download, or nothing while there is none or the day's budget is spent.
• ProbeResult records a probe's outcome: the file's path, format and tracks, or why there are none, and the piece data it downloaded.
• ProbeUpload records the piece data a fetcher uploaded to the swarms of the releases it probed since its last report.
• Search matches every word of the query against names and stored file paths, and filters by kind and by the probability of an audio track in one language. It has Jev examine a bounded number of its candidates first, so the filters apply to what is known once that is done. Each release carries the layer that decided its kind, whether Jev has examined it, its audio entries with their marks, its quality, season and episode, its work as work, m348 or tv1399, the work's IDs, the probed tracks of its main video file, and tmdb, a TMDBWork: what TMDB knows of the work its title is matched to, with the title, overview and genres in the request's lang, ru or en (en when empty), falling back to the other language and the title to the original; absent while the title was never looked up, with id 0 when nothing matched. Given one ID, Search returns the releases of that work.
• Get returns one release by infohash, with tmdb in lang likewise.
• BrowseService reads the store as the web UI shows it and classifies nothing first. A work in its answers is a WorkSummary: its ID, kind, tmdb, stored releases, distinct peers over the last 7 days, place in the 7-day top of its kind, and grabs_week, the grabs Radarr and Sonarr reported over the last 7 days. A voice narrows releases by their audio in one language: VOICE_MARKER to those with a marker of it, VOICE_LIKELY to those with a marker or a probability of at least 0.5.

RPC	Request	Response
SearchTitles	query, kind (movie, series or both), lang	works: those whose titles, or whose releases' names and paths, have every word, and those TMDB finds, with releases first by peers, then the rest by TMDB popularity; ours, tmdb_only
Title	id, lang	work; ranks over 24 hours, 7 days and 30 days; peers of all its releases over the three; audio per configured language: releases marked, their marks, releases likely
TitleReleases	id; quality, source, min_size, max_size, audio_language and voice, season, episode; sort by peers, first sighting or size; offset, limit	releases with their peers over 7 days, quality, source, season, episode and audio; total, matched, seasons
Torrents	query, words or an infohash; kind (KIND_OTHER takes music, books and software in) or unknown_kind; matched; quality; audio_language and voice; source, layer, min_size; sort; offset, limit; lang	rows, each a release with its work when matched, adult releases left out; total
Card	infohash, lang	the release, with tmdb the state of its title's match; discovery, every stored file, peers over the day, week and month; its work and rank_in_work, its place among the work's releases by peers; its rules, those that recognised features in the name (kind, year, season, episode, quality) or audio markers in the name or file paths, each with the matched fragment, recomputed when the card is opened
Top	kind (movie or series), window, voice and voice_language (the first audio language by default), offset, limit, lang	entries: rank, work, peers, change against the window before, audio, sources and first sighting of the releases the voice admits; total, computed_at
Stats	days	counts by kind (movie, series, other, unknown; music, books and software are other), layer, source and audio language, rejections by reason, pending, tokens spent today and the day's limit, piece data probes downloaded today, the day's limit on it, and what the fetchers uploaded back today, new torrents per day, computed_at
Home		stored, new in 24 hours and pending, computed_at
Settings		the API's audio languages, BCP-47 tags in its order, empty while the audio extension is off; the web UI asks it at start

Stats and Home serve the counts the API computed last, at start and then every SNIFFARR_STATS_INTERVAL, as of computed_at; only tokens_used_day, probe_bytes_rx_day and probe_bytes_tx_day are read at the request.

lang is ru or en, en when empty: the language of tmdb texts, and of TMDB's search for SearchTitles. A work's id that is not one fails with INVALID_ARGUMENT, one of no stored work with NOT_FOUND.

Torznab

With SNIFFARR_TORZNAB_TOKEN set, the API port also serves a Torznab indexer at /torznab/api, for Prowlarr, Radarr and Sonarr; without it the path answers 404. Each request carries the token as its apikey parameter.

t	Parameters
caps
search	q, cat, offset, limit
movie	q, tmdbid, imdbid, cat, offset, limit
tvsearch	q, tvdbid, tmdbid, season, ep, cat, offset, limit

It answers through Search, so its candidates are classified and matched as a search's are, and returns movies and series only: movie searches movies, tvsearch series, search both, and cat narrows them to Movies (2000s) or TV (5000s). An ID parameter selects the releases matched to that work, tmdbid among movies for movie and among series for tvsearch; imdbid takes either tt0078748 or 78748. season selects the releases whose name gives that season, and ep with it that episode and the season's whole packs; a name that gives no season matches no season, and an ep in the daily form MM/DD matches nothing. limit is 100 by default and offset and limit together 500 at most, counted among the releases that match.

Each item is a release: its name, its magnet as guid, link and enclosure, its first sighting as pubDate, and the attributes category, size, infohash, magneturl, seeders, peers, and tmdbid, imdbid (without tt) and tvdbid when its title is matched. The category is Movies or TV, /UHD for a name with 2160p, 4K or UHD, /HD for 1080p, 1080i or 720p, /SD otherwise. seeders and peers are both the release's distinct peers over the last 7 days, since the DHT does not tell seeders apart.

The title is the release's name with markers in brackets after it, for the release-title custom formats of Radarr and Sonarr:

• A probed Russian audio track: the labels the Russian tracks' names say, Dub for a dub (Дубляж), MVO, DVO, AVO and VO for many-voice, two-voice, author's and one-voice voice-over (Многоголосый, Двухголосый, Авторский, Одноголосый, or the abbreviation), each left out when the name already has it as a word, then RUS: Coraline.2009.2160p.HDR.BDRip.Master5 [Dub, MVO, RUS].
• Probed audio tracks all in other languages, every one giving a language, and no Russian marker: NORUS and their ISO 639-2 codes in the order of the tracks, [NORUS FRA, ENG]. No release name has NORUS, and it does not read as Russian.
• Otherwise a Russian marker, or a probability of at least 0.9 of a Russian track: [RUS].

RUS is followed by the Russian studios in the name or stored file paths, [RUS LostFilm]. A release that has none of these keeps its name.

In Prowlarr, add an indexer of type Generic Torznab with URL http://<host>:8080/torznab, API path /api, and the token as API key.

Radarr and Sonarr

With SNIFFARR_WEBHOOK_TOKEN set, the API port also serves a webhook for Radarr and Sonarr at POST /webhooks/arr; without it the path answers 404. In each app, add a connection of type Webhook under Settings, Connect:

Field	Value
URL	https://<api>/webhooks/arr
Method	POST
Username	anything
Password	SNIFFARR_WEBHOOK_TOKEN
Triggers	On Grab, On File Import; On File Upgrade if you like

A request without the token as its Basic password answers 401. The app is the query's app, radarr or sonarr, or else the payload's: a movie is Radarr's, a series Sonarr's. A Grab is recorded as a grab, a Download, which Radarr and Sonarr send on import and upgrade, as an import; Test and every other event answer 200 and record nothing.

Each event is recorded once per app, event and download ID, so the same download reported twice counts once. A downloadId of 40 hex digits, as torrent clients such as qBittorrent report it, is the torrent's infohash; any other, such as a Usenet client's, leaves the event without one. The event's work is Radarr's TMDB ID, or Sonarr's TMDB ID or else its TVDB ID, which TMDB's /3/find turns into a TMDB ID once, the mapping kept with the work's IDs; a work new to the store is stored without details, for the backfill to fetch.

With a work and an infohash, the stored release of the hash belongs to the work, and its title key is matched to the work under layer arr with confidence 1, TMDB_LAYER_ARR in the API, which every other release of the key follows. Neither TMDB's search nor Jev replaces an arr match; a later event does. A release not stored is queued, ahead of announced infohashes, and gets the work and the match when its metadata is stored; one the DHT never yields, such as a private tracker's, stays unfetched. A rejected infohash is not queued.

Every grab is demand, with or without an infohash: grabs_week in a WorkSummary, in Title, SearchTitles, Top and Card, counts the work's grabs over the last 7 days. Imports are recorded and not counted.

With SNIFFARR_RADARR_URL and SNIFFARR_RADARR_API_KEY, or the SNIFFARR_SONARR_ pair, the API imports the app's grabs through its /api/v3/history: the whole history once per database, oldest first, and then at start and every 24 hours the grabs since the latest recorded one through /api/v3/history/since; a round that fails is retried after an hour. The URL is the app's base, with its URL base if it has one, such as http://radarr:7878 or http://nas/sonarr. A history record and a webhook event of one download are one event, and the records of a season pack's episodes one grab. An event without a download ID is never taken for another.

Web UI

With -ui, sniffarr serves a web UI on SNIFFARR_UI_ADDR, in English or Russian. The UI touches no database: it reads BrowseService, in-process when -api is on and over the network at SNIFFARR_URL otherwise. At start it asks Settings for the API's audio languages, retrying every 5 seconds until the API answers, and listens only then; the languages are kept for the life of the process, so a change of the API's needs the UI restarted.

Route	Page
GET /	The store's counts: kinds, layers, sources, audio per language, rejections by reason, the classifier's pending torrents and tokens used, the piece data probes downloaded against the day's limit and what the fetchers uploaded back, and new torrents per day and per week, with how many minutes ago the API computed them
GET /w/{id}	A movie or series, id m348 or tv1399: what TMDB knows of it, its places in the tops over 24 h, 7 days and 30 days, its releases and peers, the audio of its releases, and its releases under quality (2160p, 1080p, 720p, sd), source (web, rip, remux, disc), min and max in GB, vl and voice, a series' season and episode, sorted by sort (peers, first_seen, size), 50 a page from o
GET /torrents	Every release, matched or not, by name or infohash in q, under kind (movie, series, other, unknown), matched (yes, no), quality, vl and voice, source, layer, min in GB, sorted by sort (peers, last_seen, size), 50 a page from o; a matched release links to its movie or series
GET /top	The 100 movies or series with the most peers over a window: kind is movie or series, win is 24h, 7d or 30d, voice is any, p50 or explicit in the audio language vl, narrowing each to its releases with a marker or a probability of at least 0.5, or to those with a marker; the first 25 until all=1. A row leads to its movie's or series' page
GET /top/w/{id}	A permanent redirect to /w/{id}
GET /t/{infohash}	A release: its magnet link, peers over the last day, 7 days and 30 days, its movie or series with its place in the top and the release's place among its releases, or the state of its title's lookup with TMDB and IMDb searches while it is unmatched, classification, audio signs per language with their marks, and the first 8 stored files, every one with files=all
GET /static/…	Stylesheet, scripts and the TMDB logo. The pages link each with a version in v, a hash of its content, at which it is cached for a year; at any other URL the browser revalidates it by ETag

A parameter value outside those listed is the default: every kind, quality, source and layer, the audio language of the page's language when it is configured and else the first, voice=any, sort=peers, kind=movie for the top, win=7d. A fragment of a page is served alone when htmx asks for it with HX-Request: true and HX-Target naming it: results, releases, torrents, top, files or data.

The UI authenticates nobody: an authenticating proxy, such as oauth2-proxy, goes in front of it. Its language is the one picked with ?lang=ru or ?lang=en, on any page, remembered in a cookie for a year, else, with SNIFFARR_OIDC_ISSUER set, the locale claim of the provider's userinfo for the access token in the X-Forwarded-Access-Token header, which oauth2-proxy can pass on; a locale is kept for 10 minutes per token. Adult releases do not show in it.

The chart serves the UI with ui.enabled, behind the Service <release>-ui; see Helm.

Configuration

Each component reads its own variables, listed below, and only when it is on. A variable several components read, such as SNIFFARR_URL, is read once and means the same to each.

Every component

Flag or variable	Default	Meaning
-api	off	Serve the API
-ui	off	Serve the web UI
-sniff	off	Walk the DHT and report infohashes
-fetch	off	Take fetch tasks and return metadata
-log-level	info	Level to log at: debug, info, warn or error
SNIFFARR_LOG_LEVEL		The level when -log-level is not given; the flag wins when both are set
SNIFFARR_URL		Base URL of the API, such as http://sniffarr-api:8080; required by -ui, -sniff and -fetch without -api, and must be unset with it

-api

Variable	Default	Meaning
SNIFFARR_DB		Path of the SQLite database; required
SNIFFARR_ADDR	:8080	Listen address
TYPESAFE_API_KEY		TypeSafe API key; when unset, only rules and reuse classify
SNIFFARR_CLASSIFY_CONCURRENCY	4	Background classification requests at once
SNIFFARR_CLASSIFY_DAILY_TOKENS	2000000	Classifier tokens per UTC day at most; 0 is no limit
SNIFFARR_CLASSIFY_SEARCH_RESERVE	25	Percentage of the daily tokens background classification leaves for searches, 0 to 100
SNIFFARR_SEARCH_CLASSIFY_MAX	20	Candidates one search has classified at most
SNIFFARR_SEARCH_CLASSIFY_WAIT	2s	How long a search waits for them
SNIFFARR_FETCH_LEASE	1m	How long a fetcher may hold a task
SNIFFARR_FETCH_ATTEMPTS	3	Timed-out fetches after which an infohash is dropped
SNIFFARR_FETCH_BACKOFF	10m	Wait before retrying a timed-out task, doubled per timeout
SNIFFARR_QUEUE_MAX_SIZE	1000000	Infohashes the queue keeps
SNIFFARR_QUEUE_MAX_AGE	24h	How long an infohash may wait in the queue
SNIFFARR_QUEUE_DEAD_FOR	168h	How long an evicted infohash, or one dropped after its last attempt, stays out
SNIFFARR_QUEUE_NO_PEERS_FOR	24h	How long an infohash dropped for having no peers stays out
SNIFFARR_STORE_MAX_FILES	20	Files stored per torrent, the largest ones
SNIFFARR_RETENTION	2160h	How long a stored torrent is kept without being seen
SNIFFARR_RETENTION_IDLE	168h	How long a stored torrent with no TMDB work is kept without being seen
SNIFFARR_STATS_INTERVAL	5m	How often the counts of Stats and Home are recomputed
SNIFFARR_SCRIPTS	Latin,Cyrillic	Scripts, by Go unicode.Scripts name and comma-separated, whose letters a torrent name may have; empty is the default
SNIFFARR_AUDIO_LANGUAGES		Audio languages, BCP-47 tags and comma-separated; unset or empty leaves the extension off. An unknown tag fails startup
TMDB_API_KEY	built-in token	TMDB API key or read access token; set empty, no titles are matched
SNIFFARR_TMDB_CONCURRENCY	2	Background title matches at once
SNIFFARR_TMDB_RATE	20	TMDB requests a second at most
SNIFFARR_TMDB_RETRY	168h	How long a title key that matched nothing waits before it is looked up again
SNIFFARR_SEARCH_TMDB_MAX	5	Title keys one search has matched at most
SNIFFARR_TORZNAB_TOKEN		API key of the Torznab endpoint; when unset, there is none
SNIFFARR_WEBHOOK_TOKEN		Basic password of the Radarr and Sonarr webhook; when unset, there is none
SNIFFARR_RADARR_URL		Base URL of the Radarr whose grab history is imported; set with SNIFFARR_RADARR_API_KEY or not at all
SNIFFARR_RADARR_API_KEY		That Radarr's API key
SNIFFARR_SONARR_URL		Base URL of the Sonarr whose grab history is imported; set with SNIFFARR_SONARR_API_KEY or not at all
SNIFFARR_SONARR_API_KEY		That Sonarr's API key
SNIFFARR_PROBE_DAILY_BYTES	100GiB	Piece data probes download per UTC day at most, in bytes or with a unit of B, KiB, MiB, GiB or TiB; 0 is no limit
SNIFFARR_PROBE_SEARCH_RESERVE	25	Percentage of the daily bytes left to releases a Torznab search returned, 0 to 100
SNIFFARR_PROBE_MAX_BYTES	64MiB	Piece data one probe downloads at most, units as above
SNIFFARR_PROBE_LEASE	15m	How long a fetcher may hold a probe task
SNIFFARR_PROBE_ATTEMPTS	3	Failed probes after which a release is not probed again
SNIFFARR_PROBE_BACKOFF	6h	Wait before repeating a failed probe, doubled per failure

The database runs in WAL mode and its schema migrates on start; leases left over from a previous run are freed on start. Once a minute the API truncates the WAL when it has reached 1 GiB, waiting up to 10 seconds for readers; a reader that outlasts the wait is logged with the frames it pins. The WAL's size is the metric sniffarr.db.wal.size. A torrent keeps its true file count, but only its largest files are stored and searched. Once a minute the API deletes, in small batches, the torrents nobody has sighted within the retention period, or within the idle retention period for those matched to no TMDB work, with their peer sketches; a later sighting queues them again. Once an hour it unions the hour sketches older than 8 days into day sketches and deletes the day sketches older than 65 days, and every 10 minutes, and once at start, it recomputes the tops of works and every torrent's peers over the last 7 days.

-ui

Variable	Default	Meaning
SNIFFARR_UI_ADDR	:8081	Listen address
SNIFFARR_OIDC_ISSUER		OpenID Connect issuer whose userinfo gives the UI a person's locale; empty asks none

-sniff and -fetch

Flag	Default	Meaning
-port	42069	Port for peers and the DHT, TCP and UDP; 0 picks one
-upnp	false	Forward the port, TCP and UDP, by UPnP
-stun	stun.l.google.com:19302	STUN server asked for the node's public address, for BEP 42; empty asks none
-fetchers	2000	Metadata fetches at once
-rate	2000	DHT queries, and new peer connections, a second
-fetch-timeout	15s	How long to wait for metadata once a torrent has peers
-peer-wait	8s	How long to look for a torrent's peers on the DHT
-min-peers	1	Peers needed within -peer-wait to fetch the metadata
-walkers	16	sample_infohashes queries at once
-probe-dir		Directory of the pieces -fetch probes and seeds; empty probes nothing, and it needs -fetch
-probers	4	Probes at once
-probe-timeout	10m	How long one probe may take, from looking for peers to the tracks
-seed-max-size	200GiB	Piece data kept for seeding, units as SNIFFARR_PROBE_MAX_BYTES; the oldest torrents go first beyond it
-seed-max-age	168h	How long a probed torrent is seeded at most

With -upnp the node asks the gateways it finds on the local network to forward the port it listens on, TCP and UDP. Without -sniff the node is passive by BEP 43: it answers no DHT query and asks other nodes to keep it out of their routing tables.

With -sniff the node answers ping, find_node, get_peers and announce_peer, and holds the peers announced to it in memory: one per IP and infohash, served in get_peers for 30 minutes after the announce, at most 50 a reply, up to 100 peers of each of up to 200,000 infohashes, the oldest evicted first. Nothing of it is persisted. It answers BEP 51 sample_infohashes with up to 20 of the infohashes it holds, their count, the nodes nearest the target and an interval of 1800 seconds, one reply per source every 10 seconds; and a get_peers with scrape set gets the BEP 33 BFsd and BFpe bloom filters of the seeds and the other peers it holds for the infohash, a peer being a seed when its announce_peer said seed.

The node's ID comes from the API's HashKey with the hash key: the API generates 20 random bytes once and keeps them in its store, so the node keeps its place in other nodes' routing tables across restarts. The node waits for the ID before it starts, retrying while the API is unreachable and warning at every attempt after the first. Then it asks the -stun server for its public address, with 5 seconds to answer, and the ID takes the BEP 42 prefix of that address, so other nodes that check it accept the node; the rest of the ID stays, so the same address gives the same ID, and a new address a new one. When the server does not answer the ID is used as stored, with a warning, and so it is when -stun is empty; nodes enforcing BEP 42 may not keep it. The passive node of -fetch alone takes a random ID at each start: it answers nothing and is in no routing table.

Both components count the peers they see in sketches of the current UTC hour and send them to the API's Peers every 3 minutes, and at once when 100,000 sketches wait; a send that fails is retried, and while the API is unreachable the oldest unsent batch makes room for new sketches. -sniff counts an announcing peer only for an infohash Sight has reported as a stored torrent, remembering up to 1,000,000 of them, so a torrent's first announce before that answer is not counted; -fetch counts the peers of every torrent it fetched, and the API drops the sketches of those it rejected. The key comes from the API's HashKey at start, and peers seen before it arrives are not counted.

Discovery rests on sample_infohashes: in a measured run on a server with a public address, no announce_peer query reached the node, and the sniffer still fetched metadata for about 22 torrents a second. Behind NAT it fetches far fewer, a few a second at most.

Telemetry

sniffarr exports traces and metrics with the OpenTelemetry SDK over OTLP, configured only by the standard variables: OTEL_EXPORTER_OTLP_ENDPOINT and the rest of OTEL_EXPORTER_OTLP_*, OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, OTEL_SERVICE_NAME (sniffarr when unset), OTEL_RESOURCE_ATTRIBUTES, OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG. Without an endpoint the exporters go to the specification's default on localhost. OTEL_SDK_DISABLED=true turns telemetry off: no exporter is created and nothing connects anywhere.

Every RPC is traced and measured on both ends by otelconnect. Logs stay JSON on stderr and carry trace_id and span_id when written inside a span. No BitTorrent peer address appears in spans, attributes or logs.

The API's SQLite statements are instrumented by otelsql: inside a traced request each statement is a child span carrying its text with placeholders, never its arguments, and every statement is measured in db.client.operation.duration (s), by db.pool (write or read), db.operation.name and code.function.name, the Go function that issued it. The pools' connection counts are the db.sql.connection.* metrics.

Component	Metric	Unit	Attributes
-api	sniffarr.db.wal.size	By
-api	sniffarr.ingest.sightings	{sighting}	sniffarr.sighting.result
-api	sniffarr.queue.depth	{infohash}	sniffarr.discovery
-api	sniffarr.queue.leases	{lease}
-api	sniffarr.queue.lease_expiries	{lease}
-api	sniffarr.queue.evictions	{infohash}	sniffarr.queue.eviction
-api	sniffarr.queue.dead	{infohash}
-api	sniffarr.store.retention.deletions	{torrent}	sniffarr.store.retention.class
-api	sniffarr.ingest.results	{result}	sniffarr.fetch.outcome
-api	sniffarr.ingest.rejections	{torrent}	sniffarr.rules.reason
-api	sniffarr.ingest.stored	{torrent}
-api	sniffarr.classify.classifications	{torrent}	sniffarr.kind, sniffarr.classify.layer
-api	sniffarr.search.lazy.duration	s
-api	sniffarr.search.lazy.classifications	{torrent}	sniffarr.search.lazy.result
-api	sniffarr.classify.tokens	{token}
-api	sniffarr.classify.errors	{error}	sniffarr.classify.error
-api	sniffarr.tmdb.matches	{title}	sniffarr.tmdb.layer
-api	sniffarr.tmdb.requests	{request}	sniffarr.tmdb.status
-api	sniffarr.search.lazy.titles	{title}	sniffarr.search.lazy.result
-api	sniffarr.arr.events	{event}	sniffarr.arr.app, sniffarr.arr.event, sniffarr.arr.source
-api	sniffarr.arr.outcomes	{event}	sniffarr.arr.app, sniffarr.arr.outcome
-api	sniffarr.probe.tasks	{task}	sniffarr.probe.priority
-api	sniffarr.probe.results	{result}	sniffarr.probe.outcome
-api	sniffarr.probe.downloaded	By
-api	sniffarr.probe.uploaded	By
-sniff, -fetch	sniffarr.dht.nodes	{node}
-sniff, -fetch	sniffarr.dht.sample_answers	{answer}
-sniff	sniffarr.dht.replies	{reply}	sniffarr.dht.query
-sniff	sniffarr.dht.infohashes	{infohash}
-sniff	sniffarr.dht.peers	{peer}
-sniff	sniffarr.sniff.sightings	{infohash}	sniffarr.discovery
-fetch	sniffarr.sniff.fetch.in_flight	{task}
-fetch	sniffarr.sniff.fetch.outcomes	{task}	sniffarr.fetch.outcome
-fetch	sniffarr.sniff.fetch.duration	s	sniffarr.fetch.outcome
-fetch	sniffarr.sniff.probe.outcomes	{task}	sniffarr.probe.outcome
-fetch	sniffarr.seed.downloaded	By
-fetch	sniffarr.seed.uploaded	By
-fetch	sniffarr.seed.torrents	{torrent}
-fetch	sniffarr.seed.stored	By
-fetch	sniffarr.seed.removals	{torrent}	sniffarr.seed.reason

sniffarr.fetch.outcome is torrent, no_peers or timeout; sniffarr.discovery is announce or sample, and in the queue arr too; sniffarr.dht.query is sample_infohashes or scrape; sniffarr.sighting.result is new or known; sniffarr.classify.error is permanent or transient; sniffarr.queue.eviction is age, size or no_peers; sniffarr.store.retention.class is idle or expired; sniffarr.classify.layer is rules, reuse or jev; sniffarr.search.lazy.result is in_time, late or failed; sniffarr.tmdb.layer is tmdb, jev or none; sniffarr.tmdb.status is the HTTP status of TMDB's answer, or error without one; sniffarr.arr.app is radarr or sonarr; sniffarr.arr.event is grab or import; sniffarr.arr.source is webhook or history; sniffarr.arr.outcome is matched (the stored release got the work), queued (the infohash was queued), unknown (no infohash, no work, or a rejected torrent) or duplicate (recorded before); sniffarr.probe.priority is search or background; sniffarr.probe.outcome is tracks, unknown, no_peers or timeout; sniffarr.seed.reason is ratio (uploaded as much as downloaded), age or size, the last two removals before the ratio.

Container image

The image's entrypoint is /sniffarr, so the container's arguments are the component flags.

docker build -t sniffarr .
docker run -e SNIFFARR_DB=/data/sniffarr.db sniffarr -api -ui

Helm

Each vX.Y.Z tag publishes the image as sarak.sh/arr/sniffarr:X.Y.Z and the chart in charts/sniffarr at the same version, without the v.

helm install sniffarr oci://sarak.sh/arr/charts/sniffarr --version X.Y.Z \
  --set api.existingSecret=sniffarr

api.existingSecret names a Secret the api pod takes as its environment, such as TYPESAFE_API_KEY, SNIFFARR_TORZNAB_TOKEN and SNIFFARR_WEBHOOK_TOKEN; without it the pod takes none. With api.litestream.enabled, api.litestream.existingConfigMap is required. Every value is described in values.yaml.

With api.networkPolicy.enabled, true by default, a NetworkPolicy <release>-api admits to the API port, 8080, only the release's own pods and the peers listed in api.networkPolicy.from, which are NetworkPolicyPeers: Prowlarr, Radarr and Sonarr reach Torznab and the webhook on that port, so their namespaces or pods belong there. The api pod's other ports stay open. Kubernetes leaves it to the network plugin whether a pod in the host network, as sniff is by default, matches a pod selector; where it does not, list its nodes in api.networkPolicy.from or turn the policy off.

api:
  networkPolicy:
    from:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: media

The chart installs in one of two modes, both of the one image.

A Deployment per component

By default each component is a Deployment of its own, reaching the API at the Service <release>-api. <release>-api runs sniffarr -api behind that Service. <release>-sniff runs sniffarr -sniff, sniff.discover and sniff.fetch rendering -sniff and -fetch, in the host's network by default so that announce_peer queries reach its DHT port. <release>-fetch runs sniffarr -fetch, in the pod network by default, with fetch.replicas replicas and the URL of sniff unless its own is set. A fetcher's DHT node is passive and the fetcher dials the peers it fetches from, so it needs no open port.

With fetch.probe.enabled, <release>-fetch probes too, rendering -probe-dir=/seeds and fetch.probe's probers, timeout, maxSize and maxAge as -probers, -probe-timeout, -seed-max-size and -seed-max-age. /seeds is fetch.probe.persistence: an emptyDir, a hostPath, or a claim, <release>-fetch-seeds when the chart creates it. The directory is one process's, so the chart refuses fetch.probe.enabled with fetch.replicas other than 1. The fetcher reaches the peers it seeds to and is reached by them on its DHT port, so it belongs in the host's network, fetch.hostNetwork, or behind fetch.service.

With ui.enabled, <release>-ui runs sniffarr -ui, with ui.replicas replicas, behind the Service <release>-ui on ui.service.port (8081). ui.addr is SNIFFARR_UI_ADDR, and ui.env carries the rest of the UI's variables, SNIFFARR_OIDC_ISSUER. ui.extraContainers and ui.extraVolumes add an authenticating proxy to the pod, and ui.service.targetPort names its port. With ui.addr on a loopback address, such as 127.0.0.1:8081, the UI is reachable only through that proxy, and ui.service.targetPort must name it.

Either DHT Deployment reaches the DHT from the pod network through <component>.service: a Service of the component's name exposing its port, TCP and UDP, NodePort by default. With service.nodePort the pod listens on that port, rendered as -port, so the port peers reach the node on is the port it uses; flags.port must then be unset or equal. <component>.upnp renders -upnp.

One process

With single.enabled, the chart renders one Deployment, <release>, in place of the component Deployments. Its container runs sniffarr -api, and -ui, -sniff and -fetch for each of ui, sniff and fetch with enabled true, sniff.discover and sniff.fetch still deciding the sniff component's flags. The Services <release>-api and, with ui.enabled, <release>-ui select that pod. The components keep their values and the pod takes the union of them: api.persistence, api.litestream, api.existingSecret, the env of every enabled component, ui.addr, the flags and upnp of sniff and fetch, fetch.probe, their hostNetwork, and the extraContainers and extraVolumes of api and ui. The pod's resources, podAnnotations, podLabels, nodeSelector, tolerations and affinity come from the single block; the components' own are not read. With sniff.hostNetwork, every listener binds on the host, the API's and the UI's too.

One process has one DHT node, so sniff.service or fetch.service, not both, exposes it as the Service <release>-dht. The chart refuses values the one process cannot honour: single.replicas other than 1, since the process owns the SQLite database; ui.url, sniff.url or fetch.url, since the clients reach the API in-process and the binary refuses SNIFFARR_URL with -api; and one env or flags key set to different values by two components.

Calling the API from a shell

hack/rpc sends one unary request to the API over ConnectRPC and prints the response as indented protojson. It is a developer tool, not part of the image.

go run ./hack/rpc [flags] <Service>/<Method> [json|-]
go run ./hack/rpc -l

The method is Service/Method of sniffarr.v1 or its full name, such as sniffarr.v1.SearchService/Search; -l lists them, and streaming methods are refused. The request is protojson, from the argument, or from stdin when the argument is - or absent and stdin is not a terminal; empty is the zero message. --url names the API, such as http://127.0.0.1:8080. Without it, rpc runs kubectl port-forward svc/<release>-api on a free local port, in the kube context --context, the current one when empty, and the namespace -n or --namespace, sniffarr by default, with --release sniffarr by default, and stops it on exit. -v prints the method, the URL and the time taken to stderr; --timeout, 30 seconds by default, bounds the call.

go run ./hack/rpc --context saraksh SearchService/Search '{"query":"House of the Dragon","limit":5}'
go run ./hack/rpc BrowseService/Home

Generating code

The generated code under internal/apis is committed. After changing a .proto file, regenerate it with buf; the protoc plugins come from the Go tool dependencies in go.mod.

buf lint
go generate ./...

Releases

CHANGELOG.md decides the version. To release, rename ## [Unreleased] to ## [X.Y.Z] - YYYY-MM-DD, open a fresh ## [Unreleased] above it, and merge on the date you wrote. mikluko/action-changelog validates the file on every pull request.

The repository lives at https://forge.sarak.sh/arr/sniffarr, and its workflows in .forgejo/workflows run on Forgejo Actions. On a push to main, release-cut.yaml runs CI, then tags the commit vX.Y.Z when the newest entry names a version no tag carries. The tag starts release-roll.yaml, which fails unless the tag is the newest entry of CHANGELOG.md at that commit, then builds and pushes the image sarak.sh/arr/sniffarr:X.Y.Z and the chart oci://sarak.sh/arr/charts/sniffarr at X.Y.Z, and creates the Forgejo release from the entry last. A tag whose release-roll.yaml run failed is not cut again: re-run that run. Run by hand, release-roll.yaml builds the newest version, a prerelease included, and publishes nothing. Nothing else sets a version.

Setting up

The arr organisation holds two Actions secrets, both tokens of the Forgejo user agent:

• RELEASE_TOKEN, with write:repository. release-cut.yaml pushes the tag with it, since a tag pushed with the automatic token starts no workflow.
• REGISTRY_TOKEN, with write:package, which pushes the image and the chart.

License

MIT; see LICENSE.