注意
该项目仅部分完成,我尚未实现我撰写的以下博文中描述的许多功能:https://penafieljlm.com/2017/07/14/inquisitor/。
Inquisitor 是一个通过使用开源情报(OSINT)来源来收集公司和组织信息的简单工具。它深受 Maltego 和 recon-ng 操作方式的启发,该工具很大程度上重新实现了这些工具的一些功能,但在资产类型之上增加了一层基于观点的语义,以创建易于使用的工作流程。
Inquisitor 的关键特性包括:
Inquisitor 的整个概念围绕根据已知目标组织的信息从开放来源提取信息的思想。在 Inquisitor 的上下文中,这些被称为“transforms”。相关信息也可以基于可从 whois 和互联网注册机构等开放来源检索的元数据,立即从已知资产中检索。
这些概念在这篇博文中进一步详细讨论:https://penafieljlm.com/2017/07/14/inquisitor/
要安装 Inquisitor,只需克隆仓库,进入目录,然后执行安装脚本。``` pip install Cython click git clone [email protected]:penafieljlm/inquisitor.git cd inquisitor python setup.py install
## 使用
Inquisitor 有五个基本命令,包括 `scan`、`status`、`classify`、`dump` 和 `visualize`。```
usage: inq [-h] {scan,status,classify,dump,visualize} ...
optional arguments:
-h, --help show this help message and exit
command:
{scan,status,classify,dump,visualize}
The action to perform.
scan Search OSINT sources for intelligence based on known
assets belonging to the target.
status Prints out the current status of the specified
intelligence database.
classify Classifies an existing asset as either belonging or
not belonging to the target. Adds a new asset with the
specified classification if none is present.
dump Dumps the contents of the database into a JSON file
visualize Create a D3.js visualization based on the contents of
the specified intelligence database.
在扫描模式下,该工具会对您情报数据库中的所有资产运行所有可用的转换。请确保为下面提到的各种OSINT来源创建API密钥,并将其提供给脚本,否则使用这些来源的转换将被跳过。同时,请确保您首先使用 classify 命令在情报数据库中植入一些已知拥有的目标资产,因为如果数据库中没有包含任何拥有的资产,则没有可供转换的内容。```
usage: inq scan [-h] [--google-dev-key GOOGLE_DEV_KEY]
[--google-cse-id GOOGLE_CSE_ID]
[--google-limit GOOGLE_LIMIT]
[--shodan-api-key SHODAN_API_KEY]
[--shodan-limit SHODAN_LIMIT]
DATABASE
positional arguments: DATABASE The path to the intelligence database to use. If specified file does not exist, a new one will be created.
optional arguments: -h, --help show this help message and exit --google-dev-key GOOGLE_DEV_KEY Specifies the developer key to use to query Google Custom Search. Visit the Google APIs Console (http://code.google.com/apis/console) to get an API key. If notspecified, the script will simply skip asset transforms that involve Google Search. --google-cse-id GOOGLE_CSE_ID Specifies the custom search engine to query. Visit the Google Custom Search Console (https://cse.google.com/cse/all) to create your own Google Custom Search Engine. If not specified, the script will simply skip asset transforms that involve Google Search. --google-limit GOOGLE_LIMIT The number of pages to limit Google Search to. This is to avoid exhausting your daily quota. --shodan-api-key SHODAN_API_KEY Specifies the API key to use to query Shodan. Log into your Shodan account (https://www.shodan.io/) and look at the top right corner of the page in order to view your API key. If not specified, the script will simply skip asset transforms that involve Shodan. --shodan-limit SHODAN_LIMIT The number of pages to limit Shodan Search to. This is to avoid exhausting your daily quota.
### 状态
在状态模式下,该工具仅输出扫描数据库状态的快速摘要。```
usage: inq status [-h] [-s] DATABASE
positional arguments:
DATABASE The path to the intelligence database to use. If specified
file does not exist, a new one will be created.
optional arguments:
-h, --help show this help message and exit
-s, --strong Indicates if the status will be based on the strong ownership
classification.
在分类模式下,您将能够手动添加资产,并对情报数据库中已有的资产进行重新分类。您应使用此命令,用已知拥有的目标资产来填充情报数据库。``` usage: inq classify [-h] [-ar REGISTRANT [REGISTRANT ...]] [-ur REGISTRANT [REGISTRANT ...]] [-rr REGISTRANT [REGISTRANT ...]] [-ab BLOCK [BLOCK ...]] [-ub BLOCK [BLOCK ...]] [-rb BLOCK [BLOCK ...]] [-ah HOST [HOST ...]] [-uh HOST [HOST ...]] [-rh HOST [HOST ...]] [-ae EMAIL [EMAIL ...]] [-ue EMAIL [EMAIL ...]] [-re EMAIL [EMAIL ...]] [-al LINKEDIN [LINKEDIN ...]] [-ul LINKEDIN [LINKEDIN ...]] [-rl LINKEDIN [LINKEDIN ...]] DATABASE
positional arguments: DATABASE The path to the intelligence database to use. If specified file does not exist, a new one will be created.
optional arguments: -h, --help show this help message and exit -ar REGISTRANT [REGISTRANT ...], --accept-registrant REGISTRANT [REGISTRANT ...] Specifies a registrant to classify as accepted. -ur REGISTRANT [REGISTRANT ...], --unmark-registrant REGISTRANT [REGISTRANT ...] Specifies a registrant to classify as unmarked. -rr REGISTRANT [REGISTRANT ...], --reject-registrant REGISTRANT [REGISTRANT ...] Specifies a registrant to classify as rejected. -ab BLOCK [BLOCK ...], --accept-block BLOCK [BLOCK ...] Specifies a block to classify as accepted. -ub BLOCK [BLOCK ...], --unmark-block BLOCK [BLOCK ...] Specifies a block to classify as unmarked. -rb BLOCK [BLOCK ...], --reject-block BLOCK [BLOCK ...] Specifies a block to classify as rejected. -ah HOST [HOST ...], --accept-host HOST [HOST ...] Specifies a host to classify as accepted. -uh HOST [HOST ...], --unmark-host HOST [HOST ...] Specifies a host to classify as unmarked. -rh HOST [HOST ...], --reject-host HOST [HOST ...] Specifies a host to classify as rejected. -ae EMAIL [EMAIL ...], --accept-email EMAIL [EMAIL ...] Specifies a email to classify as accepted. -ue EMAIL [EMAIL ...], --unmark-email EMAIL [EMAIL ...] Specifies a email to classify as unmarked. -re EMAIL [EMAIL ...], --reject-email EMAIL [EMAIL ...] Specifies a email to classify as rejected. -al LINKEDIN [LINKEDIN ...], --accept-linkedin LINKEDIN [LINKEDIN ...] Specifies a LinkedIn Account to classify as accepted. -ul LINKEDIN [LINKEDIN ...], --unmark-linkedin LINKEDIN [LINKEDIN ...] Specifies a LinkedIn Account to classify as unmarked. -rl LINKEDIN [LINKEDIN ...], --reject-linkedin LINKEDIN [LINKEDIN ...] Specifies a LinkedIn Account to classify as rejected.
### 转储
在转储模式下,您可以将情报数据库的内容转储为人类可读的JSON文件。```
usage: inq dump [-h] [-j FILE] [-a] DATABASE
positional arguments:
DATABASE The path to the intelligence database to use. If
specified file does not exist, a new one will be
created.
optional arguments:
-h, --help show this help message and exit
-j FILE, --json FILE The path to dump the JSON file to. Overwrites existing
files.
-a, --all Include rejected assets in dump.
在可视化模式下,您将能够获取情报存储库的层次化可视化视图。``` usage: inq visualize [-h] [-l] DATABASE
positional arguments: DATABASE The path to the intelligence database to use. If specified file does not exist, a new one will be created.
optional arguments: -h, --help show this help message and exit -l, --last Simply open the last visualization generated instead of creating a new one.
## 工作流程
现在你已经了解了 Inquisitor 的基本功能,是时候学习如何*实际*使用它了。Inquisitor 的设计遵循以下步骤:
### 种子数据(Seeding)
在这一步,你的情报数据库中还没有任何数据。我们需要从某个地方开始,所以请先使用你确认属于目标组织的资产来填充数据库。你可以使用 `classify` 命令来完成。
### 扫描(Scanning)
现在数据库中已经有确认为目标组织所有的资产。接下来你可以进行扫描。你可以使用 `scan` 命令来完成。
当你对情报数据库执行 `scan` 命令时,Inquisitor 会对分类为 `accepted` 的资产运行 `transform` 方法。扫描完成后,你可能会得到更多可能属于目标组织的资产。
如果你没有获得任何新资产,你可以向情报数据库填充新信息,或者直接进入报告步骤以结束流程。
### 分类(Classifying)
虽然 Inquisitor 会为你自动分类资产,但它可能会遗漏一些实际上属于你目标组织的资产。
当这种情况发生时,你需要检查数据库内容并手动对这些资产进行分类。通常,你应该特别关注 **注册方(Registrant)** 资产,因为这种资产类型无法自动确定所有权。此外,大多数其他资产类型依赖于注册方资产的所有权分类来判断它们是否属于你的目标,因此最好密切关注你的注册方资产。另外,最初你获得的注册方资产数量并不多,所以筛选起来也不会太困难。
### 报告(Reporting)
你可以使用 `visualize` 命令或 `dump` 命令生成属于目标组织的资产可视化图。
## 演示
我通过以下链接提供了工具运行的视频演示:https://drive.google.com/open?id=0B_O70BVu38TRclo5dWRBWkdTTWc
不过,由于我的免费屏幕录制软件只能录制最多 10 分钟,我没能完整录制 `scan` 命令的运行过程。
## 开发
Inquisitor 项目的目录结构如下:```
.
|-- README.md
|-- inquisitor
| |-- __init__.py
| |-- assets
| | |-- __init__.py
| | |-- block.py
| | |-- email.py
| | |-- host.py
| | |-- linkedin.py
| | `-- registrant.py
| |-- extractors
| | |-- __init__.py
| | `-- emails.py
| `-- sources
| |-- __init__.py
| |-- google_search.py
| `-- shodan_search.py
|-- inq
|-- report
| `-- index.html
|-- setup.py
`-- tests
|-- __init__.py
`-- test_inq.py
它有三个主要模块:assets、extractors 和 sources。主脚本名为 inq。
作为开发者,您最可能感兴趣的是向系统中添加新的资产类型,因此开发者指南将主要关注这一点。
在继续实际实现资产类之前,我们首先需要了解如何与情报数据库(Intelligence Database)交互,因为在从资产类派生相关资产时,我们将与之交互。
情报数据库的源代码存储在 inquisitor/__init__.py 文件中。情报数据库逻辑封装的实际名称是 IntelligenceRepository。
你只需要从资产类中调用 IntelligenceRepository.get_asset_string 函数,因为向情报数据库添加新资产是 inq 脚本中 scan 模块的职责。你主要使用此函数来创建资产实例或从数据库中检索已存在的资产。当从资产类的 related 和 transform 函数返回资产时,此函数非常重要,因为创建新的资产对象代价高昂——某些资产在初始化时会使用网络资源。```
Function
IntelligenceRepository.get_asset_string(asset_type, identifier, create=False, store=False)
Description
Retrieves the primary key and asset object for the asset with the provided
type and identifier.
Parameters
asset_type: class, required
The type of the asset to retrieve from the Intelligence Database. You
will actually have to pass the class object of the asset type you want
to retrieve.
identifier: any, required
The identifier of the asset to retrieve. Consider the identifier as the
unique attribute of an asset object. As for which attribute is to be
used to identify an asset, it depends on the contents of the OBJECT_ID
variable in the asset module.
create: bool, optional, default=False
When no matching asset object is found, a new one will be created and
returned if this parameter is set to True. The new asset will not
necessarily be stored in the Intelligence Database unless specified
using the "store" parameter. However, I suggest you do not do this as
adding assets to the Intelligence Database is the responsibility of
another module.
store: bool, optional, default=False
When a new asset is created when none is found, the new one will be
stored in the Intelligence Database. As said previously, I suggest that
you do not do this as adding assets to the Intelligence Database is the
responsibility of another module.
Returns
A two-element tuple where the first element is the database primary key of
the element returned, and the second element is the deserialized asset
object retrieved from the database.
None if the asset was not found.
If the asset was not found and the create flag was set to True, the primary
key member of the tuple will be set to None.
### 资产
要创建新的资产类型,请在 `inquisitor/assets` 目录中创建一个新文件,并将以下骨架代码粘贴到其中:```python
import inquisitor.assets
class ASSET_NAMEValidateException(Exception):
pass
def canonicalize(ASSET_IDENTIFIER):
return ASSET_IDENTIFIER
def main_classify_args(parser):
parser.add_argument(
'-aASSET_NAME_LETTER', '--accept-ASSET_NAME',
metavar='ASSET_NAME',
type=canonicalize,
nargs='+',
help='Specifies a ASSET_NAME to classify as accepted.',
dest='ASSET_NAMEs_accepted',
default=list(),
)
parser.add_argument(
'-uASSET_NAME_LETTER', '--unmark-ASSET_NAME',
metavar='ASSET_NAME',
type=canonicalize,
nargs='+',
help='Specifies a ASSET_NAME to classify as unmarked.',
dest='ASSET_NAMEs_unmarked',
default=list(),
)
parser.add_argument(
'-rASSET_NAME_LETTER', '--reject-ASSET_NAME',
metavar='ASSET_NAME',
type=canonicalize,
nargs='+',
help='Specifies a ASSET_NAME to classify as rejected.',
dest='ASSET_NAME_rejected',
default=list(),
)
def main_classify_canonicalize(args):
accepted = set(args.ASSET_NAMEs_accepted)
unmarked = set(args.ASSET_NAMEs_unmarked)
rejected = set(args.ASSET_NAME_rejected)
redundant = set.intersection(accepted, unmarked, rejected)
if redundant:
raise ValueError(
('Conflicting classifications for ASSET_NAMEs '
': {}').format(list(redundant))
)
accepted = set([canonicalize(a) for a in accepted])
unmarked = set([canonicalize(a) for a in unmarked])
rejected = set([canonicalize(a) for a in rejected])
return (accepted, unmarked, rejected)
class ASSET_NAME(inquisitor.assets.Asset):
def __init__(self, ASSET_IDENTIFIER, owned=None):
super(self.__class__, self).__init__(owned=owned)
self.ASSET_IDENTIFIER = canonicalize(ASSET_IDENTIFIER)
# TODO: Perform other initialization actions here
def __eq__(self, other):
if not isinstance(other, self.__class__):
return False
return self.ASSET_IDENTIFIER == other.ASSET_IDENTIFIER
def related(self, repo):
# Prepare the results
results = set()
# TODO: Create related assets here based on the attributes of this asset
# Return the results
return results
def transform(self, repo, sources):
# Prepare the results
assets = set()
# Google Transforms
if sources.get('google'):
subassets = self.cache_transform_get('google', repo)
if not subassets:
# Acquire API
google = sources['google']
# TODO: Perform Google queries here and the results to 'subassets'
# Cache The Transform
self.cache_transform_store('google', subassets)
assets.update(subassets)
# Shodan Transforms
if sources.get('shodan'):
subassets = self.cache_transform_get('shodan', repo)
if not subassets:
# Acquire API
shodan = sources['shodan']
# TODO: Perform Google queries here and the results to 'subassets'
# Cache The Transform
self.cache_transform_store('shodan', subassets)
assets.update(subassets)
# Return the results
return assets
def is_owned(self, repo):
if self.owned:
return True
# TODO: Automatically determine ownership based on repo contents
return False
def parent_asset(self, repo):
# TODO: Return parent asset based on repo contents
return None
REPOSITORY = 'ASSET_REPOSITORY'
ASSET_CLASS = ASSET_NAME
OBJECT_ID = 'ASSET_IDENTIFIER'
现在将以下字符串替换为适当的值
ASSET_NAME : 资产的标准名称(例如 Registrant, Host 等)ASSET_IDENTIFIER : 资产标识符属性的名称ASSET_NAME_LETTER : 资产名称的首字母小写ASSET_REPOSITORY : 资产名称复数形式的小写最后,在 inquisitor/__init__.py 中,将你的资产注册到 ASSET_MODULES 列表中。确保你从相应文件导入新的资产。
恭喜!至此,你已经拥有一个新的可用资产类型!
然而,你需要实现以下方法,以确保你的资产与其他资产类型相关联:``` Function
related
Description
Returns the set of assets directly related to the asset in question (i.e.
those that can be derived without querying a search engine).
When creating asset objects, make sure you use the
IntelligenceRepository.get_asset_string method instead of instatiating a
new one your self so the asset can be returned from the repository if it
exists.
Set the create flag to True when calling the method in question in order
to return a new object when one isn't found.
Set the store flag to False as appending assets is the job of another
module.
Parameters
repo: IntelligenceRepository
The Intelligence Repository that is being used in the current context.
Returns
Set of assets directly related to the asset in question.
为此,我们需要使用一个支持OpenAI格式的统一API。
然后,我们可以使用一个小型模型进行分析,因为问题非常简单。
所以,我创建了一个代理来运行。
我们可以使用一个简单的'if else'语句。
对于结果,我们需要输出一个字符串。
注意:代理只运行一次,然后停止。
所以,我创建了一个代理。此,我们需要使用一个支持OpenAI格式的统一API。
然后,我们可以使用一个小型模型进行分析,因为问题非常简单。
所以,我创建了一个代理来运行。
我们可以使用一个简单的'if else'语句。
对于结果,我们需要输出一个字符串。
注意:代理只运行一次,然后停止。```
Function
transform
Description
Returns the set of assets potentially related to the asset in question
(i.e. those that can be derived by querying a search engine).
You may access search engine objects through the provided sources
parameter.
Each search engine object has a transform method which automatically
creates asset objects for you. You just need to provide it the repository
and your query string, and then append the objects it returns to the set
of assets to be returned by your asset's transform method.
Parameters
repo: IntelligenceRepository
The Intelligence Repository that is being used in the current context.
sources: dict
The list of search engine objects that are available for use.
Returns
Set of assets potentially related to the asset in question.
Ctrl + Shift + f) 键,再次按该键即可停止日志记录<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<``` Function
is_owned
Description
Determines if there is high confidence that this asset does indeed belong
to the target. Usually checks for any "strong" classification tag first by
looking at the contents of the "owned" variable, before performing
automatic evaluation.
Automatic evaluation depends on what type of asset you're writing. For
example, for a Host asset, the secondary sources of determining ownership
would include looking if its registrant is owned by the target, if it's
parent domain is owned by the target. etc.
Parameters
repo: IntelligenceRepository
The Intelligence Repository that is being used in the current context.
Returns
True it is determined with high confidence that this asset does indeed
belong to the target.
该工具可以通过将每个内核版本标记为易受攻击或修补后,快速识别Linux x86_64内核中的漏洞。
**特性**
- **操作系统**:与操作系统无关
- **架构**:与内核级别的操作系统架构无关
**感谢等**
我们想感谢所有对这个项目做出贡献的人。```
Function
parent_asset
Description
Returns the asset object that is considered the parent of this asset
object.
Parameters
repo: IntelligenceRepository
Returns
The asset object that this asset falls under (e.g. a Block is under a
Registrant, a Host is under a Block, a Host is under another Host, an Email
is under a Host, etc. This is primarily used for visualization.
在实现上述方法后,请确保在资产源代码底部设置 REPOSITORY、ASSET_CLASS 和 OBJECT_ID 变量。
由于涉及搜索引擎的配额限制,扫描模式并未经过充分测试。此外,该项目是在为期一周的黑客马拉松挑战赛中仓促完成的,因此可能存在许多问题。如果您发现任何错误或有疑问,请提交问题工单或通过 [email protected] 与我联系。
本工作源自 Maltego 和 recon-ng 开源情报工具所实现的方法。我在此基础上补充了以下想法:一些是常识性的(例如,whois 可告知域名的所有者;子域名与父域名属于同一组织——正如域名暴力破解攻击所暗示的;组织对其拥有的域名具有权威性等),另一些则是我个人在业余时间作为爱好构思的原创想法(例如,可接受性评级、各种转换、分类继承等)。
本工作的任何部分均未借鉴我曾为任何雇主所做的工作。整个项目(包括概念验证)完全从零编写,并吸收了信息安全社区的创意。