Vulnhalla

Automated CodeQL Analysis with LLM Classification

For a detailed overview of the research and motivation behind Vulnhalla, see the official CyberArk Threat Research blog post:

Vulnhalla: Picking the True Vulnerabilities from the CodeQL Haystack

Vulnhalla automates the complete security analysis pipeline:

1. Fetching repositories of a given programming language from GitHub
2. Downloading their corresponding CodeQL databases (if available)
3. Running CodeQL queries on those databases to detect security or code-quality issues
4. Post-processing the results with an LLM (ChatGPT, Gemini, etc.) to classify and filter issues

---

🚀 Quick Start

Step 1: Prerequisites

Before starting, ensure you have:

• Python 3.10 – 3.13 (Python 3.11 or 3.12 recommended)

  • Python 3.14+ is not supported (this tool uses grpcio which is not supported by Python 3.14+)
  • Download from python.org

• CodeQL CLI

  • Download from CodeQL CLI releases
  • Make sure codeql is in your PATH, or you'll set the path in .env (see Step 2)

• (Optional) GitHub API token

  • For higher rate limits when downloading databases
  • Get from GitHub Settings > Tokens

• LLM API key

  • OpenAI, Azure, Gemini, or Bedrock credentials (depending on your provider)

Step 2: Configure Environment

All configuration is in a single file: .env

1. Clone the repository:

git clone https://github.com/cyberark/Vulnhalla
cd Vulnhalla

2. Copy .env.example to .env:

cp .env.example .env # macOS / Linux
Copy-Item .env.example .env # Windows (PowerShell)

3. Edit .env and fill in your values:

Example for OpenAI:

CODEQL_PATH=codeql
GITHUB_TOKEN=ghp_your_token_here
PROVIDER=openai
MODEL=gpt-4o
OPENAI_API_KEY=your-api-key-here
LLM_TEMPERATURE=0.2
LLM_TOP_P=0.2

# Optional: Logging Configuration
LOG_LEVEL=INFO                  # DEBUG, INFO, WARNING, ERROR
LOG_FILE=                       # Optional: path to log file (e.g., logs/vulnhalla.log)
LOG_FORMAT=default              # default or json
# LOG_VERBOSE_CONSOLE=false     # If true, WARNING/ERROR use full format (timestamp - logger - level - message)

📖 For complete configuration reference: See Configuration Reference below for all supported providers (OpenAI, Azure, Gemini, Bedrock), required/optional variables, and detailed examples.

Step 3: Install Poetry (Recommended: pipx)

Windows (PowerShell):

# List available Python versions
py -0p

# Pick any supported Python: 3.10 / 3.11 / 3.12 / 3.13
py -3.12 -m pip install --user -U pipx
py -3.12 -m pipx ensurepath
# Close and reopen terminal (required)
pipx install poetry
poetry --version

macOS / Linux:

# Check your Python version
python3 --version

# Use any supported Python: 3.10 / 3.11 / 3.12 / 3.13
python3 -m pip install --user -U pipx
python3 -m pipx ensurepath
# Restart terminal (required)
pipx install poetry
poetry --version

Step 4: Install Dependencies and Setup

Windows (PowerShell):

# Pick one supported version you have: 3.10 / 3.11 / 3.12 / 3.13
poetry env use 3.12  # Force Poetry to use a supported Python version if you have multiple versions installed
poetry install
poetry run vulnhalla-setup

macOS / Linux:

# Pick one supported version you have: 3.10 / 3.11 / 3.12 / 3.13
poetry env use 3.12  # Force Poetry to use a supported Python version if you have multiple versions installed
poetry install
poetry run vulnhalla-setup

Step 5: Run the Pipeline

# Analyze a specific repository, for example:
poetry run vulnhalla redis/redis

# Re-download even if database already exists
poetry run vulnhalla redis/redis --force

# Show help
poetry run vulnhalla --help

This will automatically:

1. Fetch CodeQL databases
2. Run CodeQL queries on all downloaded databases
3. Analyze results with LLM and save to output/results/
4. Open the UI to browse results

Using a Local CodeQL Database

If you already have a CodeQL database on disk (e.g., created manually or from a previous run), you can skip the GitHub fetch step using the --local / -l flag:

Windows (PowerShell):

poetry run vulnhalla --local C:\path\to\my-codeql-db

macOS / Linux:

poetry run vulnhalla --local /path/to/my-codeql-db

Note: The --local flag expects a CodeQL database directory, not a source code folder. You can verify by checking that the folder contains a codeql-database.yml file.

Additional Commands

# Open UI to view existing results (without running analysis)
poetry run vulnhalla-ui

# Validate configuration: CodeQL, LLM, Logging (without running analysis)
poetry run vulnhalla-validate

# List analyzed repositories and their issue counts
poetry run vulnhalla-list

# Run example pipeline (analyzes videolan/vlc and redis/redis)
poetry run vulnhalla-example

---

🖥️ User Interface (UI)

Vulnhalla includes a full-featured User Interface for browsing and exploring analysis results.

Running the UI

poetry run vulnhalla-ui

UI Layout

The UI displays a two-panel top area with a controls bar at the bottom:

Top Area (side-by-side, resizable):

• Left Panel (Issues List):

  • DataTable showing: ID, Repo, Issue Name, File, LLM decision, Manual decision
  • Issues count and sort indicator
  • Search input box at the bottom, updates as you type (case-insensitive).

• Right Panel (Details):

  • LLM decision Section: Shows the LLM's classification (True Positive, False Positive, or Needs More Data)
  • Metadata Section: Issue name, Repo, File, Line, Type, Function name
  • Code Section:
    • 📌 Initial Code Context (first code snippet the LLM saw)
    • 📥 Additional Code (code that the LLM requested during the conversation) - only shown if additional code exists
    • Vulnerable line highlighted in red
  • Summary Section: LLM final answer/decision
  • Manual Decision Select: Dropdown at the bottom to set manual verdict (True Positive, False Positive, Uncertain, or Not Set)

Bottom Controls Bar:

• Language: C (only language currently supported)
• Filter by llm desicion dropdown: All, True Positive, False Positive, Needs more Info to decide
• Action buttons: Refresh, Run Analysis
• Key bindings help text

Key Bindings

• ↑/↓ - Navigate issue list (row-by-row)
• Tab / Shift+Tab - Switch focus between panels
• Enter - Show details for selected issue
• / - Focus search input box (in left panel)
• Esc - Clear search and return focus to issues table
• r - Reload results from disk
• [ / ] - Resize left/right panels (adjust split position)
• q - Quit application

Interactive Features

Column Sorting

• Click any column header to sort by that column
• Default sorting: by Repo (ascending), then by ID (ascending)

Resizable Panels

• Draggable divider between Issues List and Details panels
• Mouse: Click and drag the divider to resize
• Keyboard: Use [ to move divider left, ] to move divider right
• Split position is remembered during the session

---

📊 Output Structure

After running the pipeline, results are organized in output/results/<LANG>/<ISSUE_TYPE>/: