About the Threadiverse Catalog

The Threadiverse Catalog is a search engine for forums/communities on Lemmy, PieFed, Mbin, Catodon, NodeBB, Discourse, Friendica, Mobilizon and similar platforms. It visits servers, reads the public list of forums each one hosts, and lets you search all of them in one place. It is built for Catodon, which uses it as its in-app directory, but anyone can use it.

Coverage is partial

The catalog only knows the servers it has been pointed at, and only the platforms it has learned to read (so far, Lemmy, PieFed, Mbin, NodeBB, Catodon, Discourse with its ActivityPub plugin, the groups of Friendica and the groups of Mobilizon). Mobilizon groups organise events: people elsewhere can follow one, but only its members write in it. Mobilizon servers carry many advertising groups with nobody in them, so a Mobilizon group is listed only when somebody besides its founder has joined or followed it (at least two members, or at least one follower). Discourse sites do not say what software they run in the way the catalog asks, so it cannot find them by itself: each one has to be added by hand. A forum that is not listed may well exist. The status page shows how many servers and forums are covered right now. Forums that stop being found, or whose server asks not to be crawled, disappear from the catalog.

How to read the numbers

• Order. “Most active” and “Best match” favour forums where people were recently active over forums that are merely big: the number of people active this month counts most, the effect of past activity halves every 30 days, subscribers add a small bonus, and quiet or long-dead forums are pushed down. With a search, how well the words match is multiplied in, and a forum whose name is exactly your search comes first.
• Active this month. Lemmy reports more active users than there really are: a silent forum can still show a user active today. The catalog passes Lemmy’s numbers on as they are and marks them “as reported”. Treat them as an upper estimate. Catodon’s numbers count people who posted or replied, and PieFed’s are shown as PieFed reports them. Mbin, NodeBB, Discourse, Friendica and Mobilizon give no such numbers at all.
• Last activity. Many platforms do not say when a forum was last used. The catalog then works it out from the activity numbers, or from noticing that a forum’s totals grew between two visits. For many Lemmy and all Mbin forums it says “not known yet” until the catalog has visited twice. NodeBB says when the last post in each category was written; a category whose posts are all in its sub-categories is dated by those visits too.
• Topics. Each forum carries up to three rough topic tags, such as gaming or science. Where the platform has tags of its own (Mbin) those are shown as they are. Otherwise they come from a short list of keywords looked for in the forum’s name, title and description. They are a help for browsing, not a judgement: a forum can be missing a tag it deserves, or carry one that fits badly.
• Adult (NSFW) forums are hidden unless you switch them on. Forums with no activity for a year are not shown at all.
• Images are never copied. Forum icons are loaded straight from the forum’s own server, and only when they scroll into view.

The crawler

The catalog visits servers with a program that introduces itself like this:

ThreadiverseCatalog/0.1 (+https://catalog.catodon.rocks/about)

That address leads back to this page. The crawler is polite:

• It reads a server’s robots.txt before anything else and follows it, with one exception explained below.
• It makes one request at a time to a server, waits at least two seconds between requests (longer if the server’s Crawl-delay asks for it) and stops when told to slow down.
• It asks only for a server’s public description of itself (its NodeInfo document, which names the software it runs) and, on platforms it can read, the public list of forums. On NodeBB and Discourse, where each category decides for itself whether it federates, it also asks each federated category’s public federation profile, about once a month (on Discourse a tag can federate too; tags are not listed, and nothing is asked about them). On Friendica the only public list is the server’s directory of the accounts that chose to be listed in it, people as well as groups: the catalog keeps the groups and nothing about the other accounts. It never logs in and never reads posts or comments. A server that shows its forums only after a login is left alone and asked again after 30 days.
• It learns of servers from two public lists of server names (see Credits). Each new server is asked once what software it runs; a server running something the catalog cannot read is left alone and asked again after 30 days.
• Active servers are visited about once a day, quiet ones weekly or monthly.

robots.txt and the one exception

Since version 1.0, Lemmy ships with a robots.txt that tells all crawlers to stay out of /api. That is meant to keep search engines from wasting the server’s power on web pages, but /api is also how programs are meant to read Lemmy, and reading the same list as web pages would cost the server more. So the catalog does not treat a blanket Disallow: /api for all crawlers as an opt-out. There are two platforms the exception does not apply to: Friendica and Mobilizon ship no such rule, so on their servers a blanket Disallow: /api is an opt-out. Every other rule is followed: Disallow: /, a rule aimed at the forum list, and any rule in a group that names us. Crawls made under this exception are recorded and counted on the status page.

Opting out and requesting removal

A server administrator can opt out in either of two ways:

• With robots.txt, no need to contact anyone. Add this to the server’s robots.txt: User-agent: ThreadiverseCatalog
Disallow: / The catalog then makes no further request to the server (apart from the re-check below), and all its forums are deleted from the catalog at once. About every 30 days it reads that one file again; if the rule is gone, the server is crawled and listed again.
• By request, permanently. Use the removal request form or write to the contact address below. An opt-out by request is never re-checked.

Settings a forum’s own owner made are honoured too: on Mbin, a magazine that is not “discoverable” or not “indexable by search engines” is not listed.

A moderator or administrator who wants a single forum, or a whole server, removed can use the removal request form, or email. A person reads every request; nothing is removed automatically. Once a request is approved the forum disappears from the catalog at once and is never listed again. The form asks for a contact detail only if you want to give one, and it is deleted as soon as the request is decided.

What is not listed

The catalog does not list forums for sexual content involving children, drawn material included, and does not answer searches for them. Servers known for such content are never contacted. A list of words, in several languages, is checked against every forum’s name, title and description: a forum whose name or title holds one is refused, and a forum that only mentions one in its description (often in its rules, to forbid it) is listed after a person has looked. No list of words catches everything. If you find a forum here that should not be listed, use the “Report this forum/community” link under it on the search page, or the removal request form; a person reads every report.

Suggesting a forum/community

A forum the crawler cannot reach (its server shows the forum list only after a login, or runs a platform the catalog cannot read) can be suggested with the suggestion form. The catalog then asks the forum’s address once, under the same rules as the crawler, to make sure it really is a public forum, and a person decides whether it is listed. A Discourse site can be brought in the same way: suggest one of its federated categories and name the platform.

Contact

For opt-outs, removals and anything else, write to dreamsareimmortal[at]hotmail[dot]com. Replace “[at]” with “@” and “[dot]” with “.”; the address is written this way to keep it away from spam robots.

Credits

This catalog builds on the work of others:

• Lemmy Explorer (source) showed that a forum directory for Lemmy is worth having, and its published data files are a source of server names.
• The Fediverse nodes list from Minoru’s Fediverse Crawler (source) is a public list of servers on the fediverse, and a second source of server names.

Source code

The catalog is open source under the AGPL-3.0 licence: codefloe.com/catodon/threadiverse-catalog. Programs can read the catalog through the API; no sign-up is needed.