AI Security Literacy (AISL) — Grounding Pipeline

A reproducible analysis pipeline that grounds the six AI Security Literacy competency dimensions in public benchmarks and nationally representative survey data. Runs in Google Colab. Every artifact — dataset cache, model checkpoint, figure, results table — persists to Google Drive, and every stage is resumable.

---

Why this exists

AI arrived in schools faster than any guidance about how to use it safely. Students are handed black-box systems and expected to intuit what those systems can be trusted to do. Teachers are split between adopting AI and fearing it. Districts are writing acceptable-use policies while the tools underneath them change every few months.

The literacy frameworks that exist teach students to understand and use AI. The cybersecurity frameworks that exist were written before AI-specific threats existed. Neither treats operating securely in an AI-mediated environment as a competency in its own right — with its own dimensions, its own developmental progression, and its own teachable skills.

That competency is what AISL defines, across six dimensions spanning the whole school ecosystem rather than students alone.

Why grounding it matters

A competency framework that nobody has tested against data is just a taxonomy. Anyone can assert that students should "recognize manipulation" or "calibrate trust." The harder question is whether those phrases name real, measurable phenomena — and if so, what the measurements actually show.

That is what this pipeline is for. It asks what public data can establish about each dimension, and the answers are specific:

• A detection label is a threshold decision, not a property of text. Moving a toxicity cutoff from 0.3 to 0.7 swings the share of items labelled toxic by more than twenty percentage points. On real scam text messages, the same move cuts false alarms from 15 to 2 while doubling missed scams from 18 to 36. No threshold removes both errors; someone has to choose.
• Harm often sits in the part of the text you have not read yet. In a corpus built to contain toxicity in equal measure across four bands, most prompt halves score in the lowest band.
• Fluent is not the same as reliable. A small language model picks the true answer on a misconception benchmark only modestly above chance.
• A robustness number describes the evaluation, not just the model. The same classifiers attacked two ways give two very different verdicts, and the spread across independently trained models is several times wider than the sampling interval on the test set.
• The policy vacuum is measurable. In a nationally representative 2025 survey, about a third of U.S. public school teachers report that their school has an AI policy, and a quarter do not know whether it does. Teachers expect their own AI use to make teaching easier and students' use to make it harder.
• The surrounding population is exposed and uneasy. Most U.S. adults call online scams a major problem, receive scam texts weekly, and are concerned about AI misusing personal information — while reporting little control over AI in their lives.

Each of those is a lesson a learner can be walked through with real models and real data, which is the point: the framework is meant to be taught, not just cited.

Why the engineering is part of the point

A pipeline can produce figures that look right and are not: a robustness curve from a model that was never properly trained, or benchmark results attached to the wrong categories because two files were joined by row position. Those are the same kinds of failure the framework's most advanced dimension teaches students to catch.

So the pipeline is built to make them structurally hard: attack evaluation is gated on a clean-accuracy sanity check, joins are made on content rather than position, blocked stages announce themselves rather than emitting empty output, sampling is seeded, and every reported number is written to a machine-readable results file instead of transcribed by hand. Reproducibility here is not housekeeping — it is the subject matter.

---

What it does

The pipeline runs in two tiers.

Tier 1 grounds the attitudinal dimensions from nationally representative survey micro-data that you download yourself under free registration. Until the files are present, its stages mark themselves skipped. See Tier 1 setup.

Tier 2 grounds the technical dimensions from public sources. No configuration needed — open and run. Every dataset downloads automatically.

	Dimension	Tier 1 (surveys)	Tier 2 (public data)
D1	Permission & Norms	RAND ATP Fall 2025, Pew ATP W142	—
D2	Data & Privacy Security	Pew ATP W152	—
D3	Threat & Manipulation Recognition	Pew ATP W152, W168	RealToxicityPrompts, SMS Spam Collection, BBQ
D4	Trust Calibration & Verification	Pew ATP W142, W152	TruthfulQA + GPT-2 baseline, BBQ
D5	Adversarial & Technical Understanding	—	CIFAR-10 + FGSM/PGD, 3 seeds
D6	Disposition & Agency	RAND ATP Fall 2025, Pew ATP W152	—

---

Quick start

Open AISL_analysis_poc.ipynb in Colab and run all cells top to bottom.

Runtime > Change runtime type > GPU     # optional; D5 training is ~10x faster
Runtime > Run all

The first run downloads CIFAR-10 (~170 MB), the Hugging Face corpora, GPT-2 and the SMS Spam Collection. All of them cache to Drive and are never re-downloaded. The run is safe to interrupt at any point — see Resuming a run.

If you add survey files to Drive while a Colab session is open, restart the session (Runtime > Restart session) before rerunning: Colab does not always see files added after Drive was mounted.

---

Output layout

Everything persistent lives under MyDrive/AISL_data/. Nothing of value is written to the ephemeral Colab disk.

AISL_data/
├── cache/
│   ├── hf/                    # Hugging Face datasets and GPT-2
│   └── torch/                 # CIFAR-10
├── checkpoints/
│   └── cifar10_cnn_seed{N}.pt # one per seed, written after every epoch, atomically
├── figures/                   # 300-dpi JPGs
├── tables/                    # CSV + JSON + TeX macro exports
├── logs/
│   └── status.json            # per-stage completion record
└── microdata/
    ├── sms/                   # SMS Spam Collection (downloaded automatically)
    ├── pew/                   # <- you place the Pew ATP .sav files here
    └── rand/                  # <- you place the RAND ATP .dta file here

Results are exported in three formats: aisl_results.json (machine-readable), per-stage .csv files, and aisl_results.tex (a set of TeX macros, one per reported value, for downstream typesetting).

---

Stages