Swarm Lens finds groups of accounts that act together to push a message, flags automated accounts, and explains why. This page tracks how accurate it is, where it fails, and what changed in each version. All numbers are measured on accounts the models never saw during training.
“False alarms” is the share of real people wrongly flagged. Thresholds are set to keep it low, so we accept missing some bots.
| Detector | Caught | False alarms | Notes |
|---|---|---|---|
| Bot detection, with profile dataIs this account automated? | 99.2% of bots caught | 2.4% | ROC-AUC 0.998. At a realistic 10% bot share, 82% of flags are right. |
| Bot detection, posts onlySame, without profile data | 94.9% of bots caught | 1.2% | Misses one bot family it has never seen (0%); the swarm detector catches that family. |
| Swarm detectionAre these accounts run together? | 83.4% of campaign accounts found | 0.3% | Precision 99.0%. Invented 0 swarms in 1,500 real accounts. |
| Propaganda languageDoes this post use propaganda language? | 35% of propaganda sentences found | 8.7% | Weak baseline (ROC-AUC 0.75). Context only; never flags an account by itself. |
Nemotron 3.5 Super VL plays the attacker. It writes campaigns for invented products, people and places at four difficulty levels; a script turns them into accounts and posting schedules. Six campaigns are used for training, four are held out. The chart shows the share of held-out campaign accounts placed in a swarm, among 150 real accounts per test batch.
| Level | Caught before | Caught after | Real people flagged before | after |
|---|---|---|---|---|
| Copy-paste | 100% | 100% | 1.3% | 1.3% |
| Light rewording | 100% | 100% | 1.2% | 1.2% |
| Personas | 0% | 99% | 0.8% | 2.5% |
| Hidden among real posts | 0% | 89% | 0.8% | 2.5% |
The cost: at the hardest level a few real people get pulled into swarms. They are marked “needs review”, never “coordinated inauthentic”, but the share of real people flagged in these batches went up.
Nemotron 3.5 Super VL reviews each flagged swarm and answers coordinated, organic or unclear, names the techniques with quotes, and explains its call. We tested it on 19 groups with known answers: 12 real or synthetic campaigns and 7 groups of real people who only look alike (same viral repost, same hashtag, random).
| Prompt | Campaigns called coordinated | Real-people groups called coordinated | Unclear | Quotes kept / dropped | Median time | Failed calls |
|---|---|---|---|---|---|---|
| Prompt v1 | 33% | 0% | 16% | 17 / 3 | 12.0 s | 0 |
| Prompt v2 | 33% | 0% | 21% | 14 / 0 | 11.9 s | 0 |
| Group | Truth | Coordinated | Organic | Unclear |
|---|---|---|---|---|
| nemotron campaign blackout | coordinated | 1 | 0 | 0 |
| nemotron campaign dsa12 | coordinated | 1 | 0 | 0 |
| nemotron campaign lumo | coordinated | 0 | 1 | 0 |
| nemotron campaign starforge | coordinated | 1 | 0 | 0 |
| organic random people | organic | 0 | 2 | 0 |
| organic same hashtag | organic | 0 | 3 | 0 |
| organic same viral repost | organic | 0 | 2 | 0 |
| real campaign social spambots 1 | coordinated | 0 | 1 | 1 |
| real campaign social spambots 2 | coordinated | 0 | 2 | 0 |
| real campaign social spambots 3 | coordinated | 0 | 1 | 1 |
| real campaign traditional spambots 1 | coordinated | 1 | 0 | 1 |
| Policy | Campaigns flagged | Real-people groups flagged |
|---|---|---|
| Swarm model alone | 50% | 0% |
| Swarm model, unless Nemotron (prompt v1) says organic | 33% | 0% |
| Swarm model or Nemotron (prompt v1) says coordinated | 67% | 0% |
Several accounts posted the exact same messages promising thousands of followers within hours and included spam links. This kind of verbatim repetition across different users is not typical of normal fans or news sharing. It indicates the accounts are working together to spread spam.
We train without one bot family, then test on it. This is the closest we can get to “a new bot operation shows up”.
| Held-out family | Bot model (with profile) | Swarm model |
|---|---|---|
| fake followers | 98% | – |
| social spambots 1 | 99% | 98% |
| social spambots 2 | 99% | 97% |
| social spambots 3 | 94% | 89% |
| traditional spambots 1 | 100% | 30% |
What changed. Every flagged swarm can be sent to Nemotron 3.5 Super VL, which answers coordinated, organic or unclear, names techniques with quotes and explains its call. Quotes not found word for word in the posts are dropped. Tested on 19 groups with known answers, with two prompt versions.
Why. Give moderators a readable reason for each verdict and a second check against false alarms.
Effect. Nemotron never called a group of real people coordinated (0 of 7), and with prompt v2 every quote it gave was real. It is cautious: it called most real spambot groups organic or unclear, so it must not be used as a veto. Used as a second net it raised recall from 50% to 67% with no extra false alarms. Raising the token budget fixed 7 of 38 calls that failed because reasoning used up the whole budget.
What changed. Nemotron wrote 10 fictional campaigns at 4 difficulty levels. The swarm model was retrained on 6 of them; 4 are held out for testing only.
Why. Before this, campaigns that reworded their message in personal voices went completely undetected.
Effect. Persona campaigns: 0% to 99% caught. Hidden campaigns: 0% to 89%. Real people flagged in those batches rose from about 1% to about 2.5%.
What changed. Dropped the 'number of posts' feature and trained on the same 50-post window the live pipeline uses.
Why. Post count mostly reflected how many tweets the dataset authors collected, not how an account behaves.
Effect. Slightly better and, more importantly, explanations no longer cite a meaningless feature.
What changed. Swarms are now split into communities (Louvain), and every member must have two strong ties inside its own community.
Why. A few similar-looking accounts were bridging separate campaigns and dragging real people into swarms.
Effect. Each campaign is its own swarm; no real people left inside any swarm.
What changed. Bot model, swarm model and propaganda-language model trained on cresci-2017 and the SemEval-2020 propaganda corpus. Verdicts combine the bot and swarm signals.
Why. Get a working baseline with honest held-out numbers before adding anything clever.
Effect. All 4 campaigns found, but two different campaigns were merged into one swarm that also pulled in 3 real people.