BotLens: A Security and Privacy Auditing Framework for Third-Party Community-Based Bots on Social Platforms

Paper: Kofi Ampomah and Tu Le. Measuring Users' Security and Privacy Concerns About Third-Party Community Bots on Discord. Network and Distributed System Security (NDSS) Symposium 2027. https://dx.doi.org/10.14722/ndss.2027.230535

Artifact reviewers: start with ARTIFACT-EVALUATION.md. It maps every claim in the paper to the exact command that reproduces it, lists expected outputs and runtimes, and says which components are out of scope.

Overview

This repository contains the source code for the BotLens pipeline, which analyzes security and privacy (S&P) concerns in third-party community-based bots on Discord through large-scale review analysis and metadata auditing. The pipeline processes 183,142 user reviews across 23,102 bots scraped from Top.gg, identifies 8,832 reviews as S&P-relevant — an observed pipeline yield of 4.82% — and maps them onto a 31-theme S&P taxonomy.

Repository Structure

BotLens/
├── README.md                          # This file
├── ARTIFACT-EVALUATION.md             # Claim → command → expected output
├── LICENSE                            # See License below
├── CITATION.cff                       # How to cite (also used by Zenodo)
├── botlens-ndss-data.zip              # The data; unpack to data/ (see Data below)
├── requirements-lock.txt              # Exact versions used for the paper
├── requirements.txt                   # Looser version bounds
│
├── config/                            # Pipeline configuration
│   ├── __init__.py
│   ├── settings.py                    # Paths, model params, thresholds
│   └── taxonomy.py                    # 31-theme S&P taxonomy (single source of truth)
│
├── scrapers/                          # Data collection scripts
│   ├── topgg_metadata_scraper.py      # Bot metadata scraper (Top.gg pages)
│   ├── topgg_reviews_scraper.py       # Bot review scraper (Top.gg reviews)
│   └── discord_server_scanner.py      # Server bot scanner — OUT OF SCOPE, see below
│
├── scripts/                           # Pipeline stages + analysis
│   ├── stage0_preprocess.py           # Merge, clean, detect language, translate
│   ├── stage1_llm_labeling.py         # Binary S&P classification via Ollama
│   ├── stage1b_active_learning.py     # Active learning loop
│   ├── stage2_prepare_data.py         # Train/val/test splits with balancing
│   ├── stage3_finetune_deberta.py     # Fine-tune DeBERTa-v3-large classifier
│   ├── stage4_classify_all.py         # Classify all 183K reviews
│   ├── stage5_theme_assignment.py     # Assign 31-theme taxonomy themes
│   ├── stage6_emotion_classification.py # GoEmotions → Ekman-style groups
│   ├── stage7_analysis.py             # Sec. IV-A: SPRR, TPR, per-tag/per-bot tables
│   ├── stage8_visualizations.py       # Figs. 2–9
│   ├── rq3_permissions_metadata.py    # RQ3 (Sec. IV-D): statistics + Figs. 10–14, 18
│   ├── language_affective_analysis.py # Appendix B: Fig. 17
│   ├── experiment_labeling_pct.py     # Labeling-budget experiment (10–45%)
│   ├── create_irr_sheets.py           # Inter-rater reliability labeling sheets
│   ├── compute_kappa.py               # Cohen's kappa (Sec. III-C3)
│   ├── check_langs.py                 # Language distribution (Table IV; needs raw crawl)
│   └── prep_30pct_for_al.py           # Prepare labels for active learning
│
├── tests/
│   └── test_smoke.py                  # Fast integrity checks, no data required
│
└── mcp_server/                        # BotLens MCP server (Sec. V)
    ├── server.py                      # The six tools of Sec. V-B2 (+ internal helpers)
    ├── example_client.py              # MCP SDK client reproducing Use Cases 1 and 2
    └── benchmark_latency.py           # Sec. V-D latency session

Every figure the scripts write is named after its paper figure number (figures/fig02_sprr_over_time.png is Fig. 2, and so on). The scripts produce exactly the figures in the paper: Figs. 2–14, 17 and 18. Fig. 1 is the pipeline schematic and Figs. 15–16 are screenshots of the BotLens report UI.

Getting the artifact

The artifact is released at https://github.com/ua-spectral/BotLens. Each GitHub release is archived permanently on Zenodo with a DOI; the DOI is given in the paper's artifact appendix and on the repository's release page.

Data

The data ships in the repository as botlens-ndss-data.zip (41 MB; one file unpacks to 515 MB, beyond GitHub's per-file limit). Unpack it once, in the repository root:

python -m zipfile -e botlens-ndss-data.zip .

This creates data/ beside config/; nothing else needs downloading. The layout is:

BotLens/
└── data/
    ├── processed/
    │   ├── all_reviews_classified.csv    # 183,142 rows — required by the MCP server
    │   ├── privacy_reviews_full.csv      # 8,832 S&P reviews w/ themes + affect
    │   ├── bot_metadata_cleaned.csv      # 23,102 bots (Table I, MCP fallback)
    │   ├── train.csv / val.csv / test.csv
    │   └── label_map.json
    ├── results/                          # Tables behind the paper's numbers
    ├── reference_figures/                # Our output of every generated figure
    └── irr/                              # Annotator workbooks (Sec. III-C3)

Verify integrity with sha256sum -c SHA256SUMS.txt from inside data/. data/README.md describes every file.

Pipeline Architecture

Stage	Script	Description	Key Output
0	stage0_preprocess.py	Merge reviews + metadata, language detection, Argos Translate	data/raw/reviews.csv
1	stage1_llm_labeling.py	Ollama/Llama 3 8B binary S&P classification	phase1_binary_labels.csv
1b	stage1b_active_learning.py	Uncertainty sampling + majority-vote labels	Improved training set
2	stage2_prepare_data.py	Confidence filtering, 5:1 downsampling, stratified splits	train/val/test.csv
3	stage3_finetune_deberta.py	DeBERTa-v3-large fine-tuning (F1 = 0.938 at 30% labeling)	best_model/
4	stage4_classify_all.py	Batch inference on all 183,142 reviews	all_reviews_classified.csv
5	stage5_theme_assignment.py	Ollama assigns 31-theme taxonomy themes	privacy_reviews_themed.csv
6	stage6_emotion_classification.py	GoEmotions (RoBERTa) → 8 affect groups	privacy_reviews_full.csv
7	stage7_analysis.py	SPRR, TPR, per-tag and per-bot SPRR, affect by theme	data/results/analysis_summary.json
8	stage8_visualizations.py	Figs. 2–9, including the Fig. 2 OLS trend (R² = 0.35)	figures/fig02_* … fig09_*

Key Metrics

• SPRR (Security and Privacy Review Rate): the percentage of reviews that our pipeline identified and retained as S&P-relevant. Overall SPRR = 4.82%. This is a pipeline-identified review rate, not a measure of verified vulnerabilities, confirmed misconduct, or bot-level risk.
• TPR (Theme Prevalence Rate): distribution of S&P reviews across the 31-theme taxonomy, excluding reviews labeled unassigned.

Taxonomy

The 31-theme S&P taxonomy (paper Table V) comprises 20 themes inherited from Akgul et al. (USENIX Security 2024) and 11 themes developed inductively in this study — 2 additional Security & Safety themes and 9 Discord-specific themes. config/taxonomy.py is the single source of truth; stage8_visualizations.py derives its groupings from it.

Theme Group 1 — Data Practices (7, all inherited): Data Collection, Data Sharing, Data Deletion, Remove Personally Identifiable Information, Data Exposure, Data Hiding, Location and Tracking

Theme Group 2 — User Rights & Controls (7, all inherited): Consent, Privacy Controls, Anonymity / Identification, Advertising, Privacy Policies and Laws, Privacy Invasion, Positive Privacy