About the Threadiverse Catalog
The Threadiverse Catalog is a search engine for forums/communities on Lemmy, PieFed, Mbin, Catodon, NodeBB, Discourse, Friendica, Mobilizon and similar platforms. It visits servers, reads the public list of forums each one hosts, and lets you search all of them in one place. It is built for Catodon, which uses it as its in-app directory, but anyone can use it.
Coverage is partial
The catalog only knows the servers it has been pointed at, and only the platforms it has learned to read (so far, Lemmy, PieFed, Mbin, NodeBB, Catodon, Discourse with its ActivityPub plugin, the groups of Friendica and the groups of Mobilizon). Mobilizon groups organise events: people elsewhere can follow one, but only its members write in it. Mobilizon servers carry many advertising groups with nobody in them, so a Mobilizon group is listed only when somebody besides its founder has joined or followed it (at least two members, or at least one follower). Discourse sites do not say what software they run in the way the catalog asks, so it cannot find them by itself: each one has to be added by hand. A forum that is not listed may well exist. The status page shows how many servers and forums are covered right now. Forums that stop being found, or whose server asks not to be crawled, disappear from the catalog.
A few platforms cannot be read in bulk. Hubzilla is the main one: a server there publishes no list of its own forums, and each forum is a channel that decides for itself whether to speak ActivityPub. A forum on such a platform can still be listed when someone suggests it. The catalog then asks that forum's own address again about every 30 days to see whether it is still a public forum, and if it no longer answers as one, it is hidden. These forums carry no activity numbers, so they are ranked low after six months without activity and hidden after a year.
How to read the numbers
- Order. “Most active” and “Best match” favour forums where new threads are still appearing over forums that are merely big: the number of people active this week counts most, the effect of a forum’s last new thread halves every 14 days, subscribers add a small bonus, and quiet or long-dead forums are pushed down. With a search, how well the words match is multiplied in, and a forum whose name is exactly your search comes first.
- Active this week. This is the number the “Most active” list counts most. Lemmy reports more active users than there really are: a silent forum can still show a user active today. The catalog passes Lemmy’s numbers on as they are and marks them “as reported”. Treat them as an upper estimate. Catodon’s numbers count people who posted or replied, and PieFed’s are shown as PieFed reports them. Mbin, NodeBB, Discourse, Friendica and Mobilizon give no such numbers at all.
- Posts and comments. Where a platform reports them, a result shows how many posts and comments the forum holds. Lemmy, Catodon, PieFed, Mbin, NodeBB and Discourse report both; Friendica, Mobilizon and the suggestion-only platforms do not, and the result then says “not reported”.
- Last activity. Many platforms do not say when a forum was last used. The catalog then works it out from the activity numbers, or from noticing that a forum’s totals grew between two visits. For many Lemmy and all Mbin forums it says “not known yet” until the catalog has visited twice. NodeBB says when the last post in each category was written; a category whose posts are all in its sub-categories is dated by those visits too. The list order follows the last new thread rather than the last reply: a forum kept alive only by comments on old posts drifts down.
- Topics. Each forum carries up to three rough topic tags, such as
gamingorscience. Where the platform has tags of its own (Mbin) those are shown as they are. Otherwise they come from a short list of keywords looked for in the forum’s name, title and description. They are a help for browsing, not a judgement: a forum can be missing a tag it deserves, or carry one that fits badly. - Language. A forum’s language is partly its own setting and partly detected automatically from the forum’s title and description; on NodeBB and Mbin, where a forum has no language of its own, it can be the language of the whole server. On Lemmy and Catodon the catalog asks each forum which languages its owner chose, and shows that setting when the owner chose one to three languages; it asks each forum at most once in 90 days, so a new forum can wait some days for its turn and a changed setting can take that long to show. On PieFed, Mbin, NodeBB and Discourse, a forum that still has no language gets one from its newest posts where they say enough: from the language labels on the posts where the platform has them, otherwise detected from the posts’ titles. Nothing of the posts is kept. So a language shown here may be wrong: a description written in two languages gets one of them, and a forum can be written in another language than its description. When the text is short or the detection is not sure, the catalog says nothing rather than guess. When you choose a language on the search page, forums whose language is not known are left out, so the list is shorter than what the catalog holds; choose “Any language” to see them all.
- Adult (NSFW) forums can be hidden, shown with the rest, or shown on their own, with the choice on the search page; they are hidden by default. Forums with no activity for a year are not shown at all.
- Images are never copied. Forum icons are loaded straight from the forum’s own server, and only when they scroll into view.
Descriptions
Each result shows the description the forum itself publishes, as its owner wrote it: headings appear as bold lines, lists and emphasis as meant, and links are clickable. A picture inside a description is offered as a link rather than loaded onto this page.
The crawler
The catalog visits servers with a program that introduces itself like this:
ThreadiverseCatalog/0.1 (+https://catalog.catodon.rocks/about)
That address leads back to this page. The crawler is polite:
- It reads a server’s
robots.txtbefore anything else and follows it, with one exception explained below. - It makes one request at a time to a server, waits at least two seconds between requests (longer if the server’s
Crawl-delayasks for it) and stops when told to slow down. - It asks only for a server’s public description of itself (its NodeInfo document, which names the software it runs) and, on platforms it can read, the public list of forums. On NodeBB and Discourse, where each category decides for itself whether it federates, it also asks each federated category’s public federation profile, about once a month (on Discourse a tag can federate too; tags are not listed, and nothing is asked about them). On Friendica the only public list is the server’s directory of the accounts that chose to be listed in it, people as well as groups: the catalog keeps the groups and nothing about the other accounts. It never logs in and never reads posts or comments. A server that shows its forums only after a login is left alone and asked again after 30 days.
- On Lemmy and Catodon it also asks each listed forum which languages its owner chose, and the server for its list of languages. This is kept small on purpose: a forum is asked at most once in 90 days, and a server gets at most 20 such requests in any 24 hours (a server with 1,260 forums or more gets one for every 60 of them, and none ever more than 100), one at a time and with the server’s
Crawl-delaybetween them. If a server answers with an error, too slowly or not at all, the questions stop for a day; that is never counted against the server. The answer to this question also names the forum’s moderators: the catalog reads the languages and keeps nothing about people. - On PieFed, Mbin, NodeBB and Discourse it reads recent posts, only for forums that still have no language: on PieFed and Mbin one page of the forum’s 25 newest posts, on NodeBB and Discourse the server’s one list of its newest topics, which serves all of the server’s forums. Of each post it reads the title and the language label and nothing else: not the text, not the author. Posts that came from other servers, adult posts and removed posts are passed over. The language is worked out and the titles are dropped at once: nothing of what people wrote is stored or logged, only the forum’s language. A forum is looked at this way at most once in 90 days, a NodeBB or Discourse server gets that one request at most once in 90 days (if the request fails, it is tried again after 7 days), and all of it counts towards the same 20 requests in 24 hours and stops for a day at the first error, as above. On Mbin the server is also asked once in 90 days which language it is set to, as the last resort.
- It learns of servers from four public lists of server names (see Credits). Each new server is asked once what software it runs; a server running something the catalog cannot read is left alone and asked again after 30 days.
- Active servers are visited about once a day, quiet ones weekly or monthly.
robots.txt and the one exception
Since version 1.0, Lemmy ships with a robots.txt that tells all crawlers to stay out of /api. That is meant to keep search engines from wasting the server’s power on web pages, but /api is also how programs are meant to read Lemmy, and reading the same list as web pages would cost the server more. So the catalog does not treat a blanket Disallow: /api for all crawlers as an opt-out. There are two platforms the exception does not apply to: Friendica and Mobilizon ship no such rule, so on their servers a blanket Disallow: /api is an opt-out. Every other rule is followed: Disallow: /, a rule aimed at the forum list, and any rule in a group that names us. Crawls made under this exception are recorded and counted on the status page.
The exception covers three questions to a Lemmy or Catodon server and nothing more: the forum list, the question to one forum about its languages, and the server’s list of languages (/api/v3/site). A narrower rule against either of the two language questions is followed. To stop only the language questions, add Disallow: /api/v3/site to your robots.txt, in the group for all crawlers or in a group naming ThreadiverseCatalog. The catalog reads a server’s robots.txt at most once a day, so the rule takes effect within 24 hours: from then on it asks the server nothing about languages, the server’s forums stay listed with whatever language the catalog already has for them, and the file is looked at again after 30 days. Lists of posts are not covered by the exception at all: on a server whose robots.txt has a rule against all of /api, the catalog reads no posts and asks nothing else for languages on PieFed, Mbin and NodeBB. To stop only the reading of posts, add a rule against the address: Disallow: /api/alpha/post/ on PieFed, Disallow: /api/magazine/ on Mbin, Disallow: /api/recent on NodeBB, Disallow: /latest on Discourse. The forums stay listed with the language the catalog already has for them.
Opting out and requesting removal
A server administrator can opt out in either of two ways:
- With robots.txt, no need to contact anyone. Add this to the server’s
robots.txt:User-agent: ThreadiverseCatalogThe catalog then makes no further request to the server (apart from the re-check below), and all its forums are deleted from the catalog at once. About every 30 days it reads that one file again; if the rule is gone, the server is crawled and listed again.
Disallow: / - By request, permanently. Use the removal request form or write to the contact address below. An opt-out by request is never re-checked.
Settings a forum’s own owner made are honoured too: on Mbin, a magazine that is not “discoverable” or not “indexable by search engines” is not listed.
A moderator or administrator who wants a single forum, or a whole server, removed can use the removal request form, or email. A person reads every request; nothing is removed automatically. Once a request is approved the forum disappears from the catalog at once and is never listed again. The form asks for a contact detail only if you want to give one, and it is deleted as soon as the request is decided.
What is not listed
The catalog does not list forums for sexual content involving children, drawn material included, and does not answer searches for them. Servers known for such content are never contacted. A list of words, in several languages, is checked against every forum’s name, title and description: a forum whose name or title holds one is refused, and a forum that only mentions one in its description (often in its rules, to forbid it) is listed after a person has looked. No list of words catches everything. If you find a forum here that should not be listed, use the “Report this forum/community” link under it on the search page, or the removal request form; a person reads every report.
Suggesting a forum/community
A forum the crawler cannot reach (its server shows the forum list only after a login, or runs a platform the catalog cannot read) can be suggested with the suggestion form, by its address or its handle. The catalog then asks the forum’s address once, under the same rules as the crawler, to make sure it really is a public forum, and a person decides whether it is listed. A Discourse site can be brought in the same way: suggest one of its federated categories and name the platform. A forum on a platform the catalog cannot read in bulk - Hubzilla, for example - is brought in this way, and is asked again about every 30 days to see whether it is still there.
Contact
For opt-outs, removals and anything else, write to dreamsareimmortal[at]hotmail[dot]com. Replace “[at]” with “@” and “[dot]” with “.”; the address is written this way to keep it away from spam robots.
Credits
This catalog builds on the work of others:
- Lemmy Explorer (source) showed that a forum directory for Lemmy is worth having, and its published data files are a source of server names.
- The Fediverse nodes list from Minoru’s Fediverse Crawler (source) is a public list of servers on the fediverse, and another source of server names.
- The Friendica Directory is a public, opt-in index of Friendica accounts. It is asked only for its groups, and only the hostname of each group’s own server is kept, so no person’s entry is fetched or stored — another source of server names.
- FediDB is a general index of servers on the fediverse. It is asked once a day, through its open API, for the servers of each platform the catalog reads, and only their hostnames are kept — another source of server names.
Source code
The catalog is open source under the AGPL-3.0 licence: codefloe.com/catodon/threadiverse-catalog. Programs can read the catalog through the API; no sign-up is needed.
