osintbuddy/webEntity graphs, OSINT data mining, and plugins. Connect unstructured and public data for transformative insights 🌐 (Web version)
[![Contributors][contributors-shield]][contributors-url]
[![Forks][forks-shield]][forks-url]
[![Stargazers][stars-shield]][stars-url]
[![Issues][issues-shield]][issues-url]
I have no data yet. It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.
🚧 ⚠️ Work in progress; Not ready for use, yet ⚠️ 🚧
Welcome to the OSINTBuddy project where you can connect, combine, and get insight from unstructured and public data as results that can be explored step-by-step. An easy-to-use plugin system allows any developer to quickly integrate new data sources so you can focus on discovering, interacting, and visualizing what's important to you
Not much, yet.
But here's the concept:
An almost incomprehensible amount of data is created every day. And each year, figures are growing at an ever-increasing rate. These data sources can be divided up into six different categories of information flow:
OSINT is a method of working with, assessing and ranking information — We are living in the information age and the volume of information OSINT research has to deal with (information explosion) and the issue of low quality data leaves researchers drowning. The majority of this data is unstructured introducing many challenges to analyzing it and producing actionable intelligence since most data analytics databases are designed for structured data. We are surrounded by information and nearly none of it is useful.
I've decided to do something about it. The rapid developments in technologies such as AI and big data analytics have opened new horizons for OSINT which weren't previously available. We want all of that information at our fingertips. We want actionable intelligence quickly and to the point, that's why we're building an open-source OSINT tool that's free software. Built with plain old web technologies anyone can learn. But this isn't really the project. It's been three decades since its inception and the internets only proved to be a mirror of us, our society and politics. We face a series of significant challenges that directly threaten democratic values and processes. The question of who owns, designs, and controls technology has never been more urgent.
What are the technical frameworks that ensure that a technology is open and equitable; how do we "reconfigure" digital infrastructures to serve positive social transformation rather than corporate interests?s We need your help to design it, to program it, and to build it. Let's hear your suggestions, your ideas, and we're going to be building right in front of your eyes. The notion of a "needle in a haystack" is taken to the extreme on the internet. Let's build a magnet.
[!CAUTION]
⚠️ Experimental Software (pre-alpha) ⚠️
OSINTBuddy is currently experimental software. It's not quite ready for use yet! If you're interested in development you'll need the
pluginsdocker service cloned:git clone https://github.com/osintbuddy/plugins osintbuddy-pluginsto the root of this repo
As is, this project is more of a toy environment that's been useful while prototyping out this idea and gaining insight into the problem space. For this project to be truly successful we must ultimately be able to be used to do serious, original, and creative work. That's the basic test for whether our tools are genuinely working rather than merely telling a good story. What powerful, generalizable ideas can we learn from this project? How should the next wave of systems build on this? The system has to be shaped in a way which allows you to ask the questions you want to ask.
| Repository | Description | Language | Purpose | ETA |
|---|---|---|---|---|
osintbuddy | The core application (this repo) | Rust, TypeScript/Preact | Web interface and backend services | 2-4 months from alpha |
plugins | The Python plugin system package on PyPi | Python | Extensible data collection framework | 2-4 months |
entities | The default OSIB entity definitions | Python | Default entity types and schemas | 4-5 months |
I have this conviction that a set of opportunities is available today. I believe it's now possible, if not today then soon, that will allow us to create a new immersive medium for not just traditional OSINT and research work but a medium for discovering, learning, researching, and coming to actionable insights in any number of contexts you can imagine. A medium such as photoshop is different from any of the individual tools photoshop contains. The range of possibilities in such immersive mediums are staggering and these possibilities emerge from the basic objects and actions in that medium.
Want to help out? Writing down and sharing your ideas, having discussions, and letting me know what works and doesn't, all act as a way to help bootstrap this project. It's a way of identifying points of leverage that might help us develop new tools for OSINT. So let's get started, here are a few of my ideas:
