osintbuddy/webEntity graphs, OSINT data mining, and plugins. Connect unstructured and public data for transformative insights 🌐 (Web version)
I have no data yet. It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.
🚧 ⚠️ Work in progress; Not ready for use, yet ⚠️ 🚧
Welcome to the OSINTBuddy project where you can connect, combine, and get insight from unstructured and public data as results that can be explored step-by-step. An easy-to-use plugin system allows any developer to quickly integrate new data sources so you can focus on discovering, interacting, and visualizing what's important to you
Not much, yet.
But here's the concept:
An almost incomprehensible amount of data is created every day. And each year, figures are growing at an ever-increasing rate. These data sources can be divided up into six different categories of information flow:
OSINT is a method of working with, assessing and ranking information — We are living in the information age and the volume of information OSINT research has to deal with (information explosion) and the issue of low quality data leaves researchers drowning. The majority of this data is unstructured introducing many challenges to analyzing it and producing actionable intelligence since most data analytics databases are designed for structured data. We are surrounded by information and nearly none of it is useful.
I've decided to do something about it. The rapid developments in technologies such as AI and big data analytics have opened new horizons for OSINT which weren't previously available. We want all of that information at our fingertips. We want actionable intelligence quickly and to the point, that's why we're building an open-source OSINT tool that's free software. Built with plain old web technologies anyone can learn. But this isn't really the project. It's been three decades since its inception and the internets only proved to be a mirror of us, our society and politics. We face a series of significant challenges that directly threaten democratic values and processes. The question of who owns, designs, and controls technology has never been more urgent.
What are the technical frameworks that ensure that a technology is open and equitable; how do we "reconfigure" digital infrastructures to serve positive social transformation rather than corporate interests?s We need your help to design it, to program it, and to build it. Let's hear your suggestions, your ideas, and we're going to be building right in front of your eyes. The notion of a "needle in a haystack" is taken to the extreme on the internet. Let's build a magnet.
[!CAUTION]
⚠️ Experimental Software (pre-alpha) ⚠️
OSINTBuddy is currently experimental software. It's not quite ready for use yet! If you're interested in development you'll need the
pluginsdocker service cloned:git clone https://github.com/osintbuddy/plugins osintbuddy-pluginsto the root of this repo
As is, this project is more of a toy environment that's been useful while prototyping out this idea and gaining insight into the problem space. For this project to be truly successful we must ultimately be able to be used to do serious, original, and creative work. That's the basic test for whether our tools are genuinely working rather than merely telling a good story. What powerful, generalizable ideas can we learn from this project? How should the next wave of systems build on this? The system has to be shaped in a way which allows you to ask the questions you want to ask.
| Repository | Description | Language | Purpose | ETA |
|---|---|---|---|---|
osintbuddy | The core application (this repo) | Rust, TypeScript/Preact | Web interface and backend services | 2-4 months from alpha |
plugins | The Python plugin system package on PyPi | Python | Extensible data collection framework | 2-4 months |
entities | The default OSIB entity definitions | Python | Default entity types and schemas | 4-5 months |
I have this conviction that a set of opportunities is available today. I believe it's now possible, if not today then soon, that will allow us to create a new immersive medium for not just traditional OSINT and research work but a medium for discovering, learning, researching, and coming to actionable insights in any number of contexts you can imagine. A medium such as photoshop is different from any of the individual tools photoshop contains. The range of possibilities in such immersive mediums are staggering and these possibilities emerge from the basic objects and actions in that medium.
Want to help out? Writing down and sharing your ideas, having discussions, and letting me know what works and doesn't, all act as a way to help bootstrap this project. It's a way of identifying points of leverage that might help us develop new tools for OSINT. So let's get started, here are a few of my ideas:
.env.example file to .env, adjust the settings as needed.
cp .env.example .env
frontend/.env.example to frontend/.env, adjust as needed.
cp frontend/.env.example frontend/.env
Once we iron out a few more bugs and a few more features we intend to setup a github actions workflow to build and package up the built frontend and Rust services. Follow the project on discord for the latest updates :)
This Rust rewrite exists because I got tired of Python and I decided to rewrite the entire thing to learn Rust. Now it's faster and crashes less (hopefully).
If you want to start developing for OSINTBuddy, create or pick up an issue and follow these steps:
Clone the repo
git clone https://github.com/osintbuddy/osintbuddy.git
cd osintbuddy
# using ssh?
# git clone [email protected]:osintbuddy/osintbuddy.git
Initialize default entities (aka plugins) for OSIB:
git clone https://github.com/osintbuddy/plugins.git osintbuddy-plugins
python3 -m venv venv
source ./venv/bin/activate
pip install -e ./osintbuddy-plugins/
git clone https://github.com/osintbuddy/entities.git plugins
Start the stack.
docker compose up db ui
Migrate database.
cargo install sqlx-cli --no-default-features --features native-tls,postgres
cargo install cargo-watch
cd crates/ && sqlx migrate run && cd ..
Run OSIB
Run the backend api and worker on your host system in separate terminals:
# NOTE: Must run all terminals with Python (3.12.9) venv activated!
# In one terminal run:
source ./venv/bin/activate
cargo watch -q -c -w crates/api -x "run -p api"
# In another terminal run:
source ./venv/bin/activate
cargo watch -q -c -w crates/worker -x "run -p worker"
Access OSIB through the URLs provided for the frontend, backend, and documentation:
http://localhost:55173http://localhost:48997/apihttp://localhost:55173/docs/overviewCtrl+C/kill the worker and api terminals, then docker compose down to clean up.docker compose down --volumes.We are using the GNU Affero General Public License v3.0 (AGPL) as we want to guarantee freedom of use, reuse, copy, modification and re-publication of modifications. Because freedom doesn’t mean permissiveness, it means reciprocity. If you build on it, you give back. That’s how the commons stays healthy. This is a technopolitical decision encoded into the social contract. Suffice to say here that the Affero GPLv3 license legally binds the service providers to give direct access to any user to the computer code that runs in a given instance.
Empowering investigators & cybersecurity enthusiasts to uncover truths, fight injustice, and create a safer world.
Start with one lead. Close cases 12x faster with OSINT.
LinkScope allows you to perform online investigations by representing information as discrete pieces of data, called Entities.
the first automatically GPU-accelerated platform, harness all of your data without worrying about scale, and pivot on the fly to follow anywhere your investigation leads you. Ideal for everything from security, fraud, and IT investigations to 360° views of customers and supply chains to mapping cancer mutations, Graphistry turns the potential of your data into human insight and value.
Open an issue if you need to get in touch with me or send an email to [email protected].