subgraph "Process"
    C1["Sampling based of ratio"]
    C2["Packet Process"]
    C3["Dispatcher"]
    O11["Output1"]
    O12["Domain Skip (optional)"]
    O13["Domain Allow (optional)"]
    O21["Output2"]
    O22["Domain Skip (optional)"]
    O23["Domain Allow (optional)"]
    O31["Output3"]
    O32["Domain Skip (optional)"]
    O33["Domain Allow (optional)"]
end

B1 --> Process
B2 --> Process
B3 --> Process

C1 --> C2
C2 --> C3
C3 --> O11
C3 --> O21
C3 --> O31

O11 --> O12 --> O13
O21 --> O22 --> O23
O31 --> O32 --> O33

subgraph Output
    Splunk
    Syslog
    H["ClickHouse"]
    Postgres
    Kafka
    I["JSON File"]
    Influx
    Elastic
    J["stdout"]
    Parquet
    Sentinel
end

O13 --> H
O23 --> I
O33 --> J

# 主要特性

- 能够使用Linux的`afpacket`和零拷贝数据包捕获。
- 支持BPF
- 能够掩盖IP地址以增强隐私
- 支持预处理采样率
- 支持“跳过”`fqdn`列表，避免将某些域/后缀/前缀写入存储
- 支持“允许”域列表，用于记录对某些域的访问
- 跳过和允许域文件/URL的热重载
- 模块化输出，每个输出流可配置逻辑。
- 使用ClickHouse的TTL属性自动数据保留策略
- 内置的Grafana仪表板，用于ClickHouse输出。
- 能够作为单个静态链接的二进制文件分发
- 能够使用环境变量、命令行选项或配置文件进行配置
- 能够使用ClickHouse的SAMPLE功能对输出进行采样
- 能够使用`prometheus`和`statstd`发送指标
- 得益于ClickHouse内置的LZ4存储，具有高压缩比
- 支持DNS over TCP、分片DNS（udp/tcp）和IPv6
- 支持通过Unix套接字或TCP的[dnstap](https://github.com/dnstap/golang-dnstap)
- 与Splunk和Microsoft Sentinel的内置SIEM集成

# 安装

## Linux

开始使用`dnsmonster`的最佳方式是从发布版块下载二进制文件。该二进制文件针对`musl`静态构建，因此应该可以在许多发行版上开箱即用。对于`afpacket`支持，您必须使用内核3.x+。任何现代Linux发行版（CentOS/RHEL 7+、Ubuntu 14.0.4.2+、Debian 7+）都附带3.x+版本，因此应该可以开箱即用。如果您的发行版无法与预编译版本一起使用，请提交包含详细信息的问题，并使用本节的[手动构建](#build-manually)手动构建`dnsmonster`。

### 容器

由于`dnsmonster`使用原始数据包捕获功能，Docker/Podman守护程序必须向容器授予该能力```
sudo docker run --rm -it --net=host --cap-add NET_RAW --cap-add NET_ADMIN --name dnsmonster ghcr.io/mosajjal/dnsmonster:latest --devName lo --stdoutOutputType=1

- 不使用 `libpcap`：
`dnsmonster` 只使用了 `libpcap` 的一个函数，即把 `tcpdump` 风格的过滤器转换为 BPF 字节码。如果你可以接受没有 BPF 支持，你可以在没有 `libpcap` 的情况下构建 `dnsmonster`。请注意，对于任何其他平台，数据包捕获会回退到 `libpcap`，因此它成为一个硬性依赖（*BSD、Windows、Darwin）```sh
git clone https://github.com/mosajjal/dnsmonster --depth 1 /tmp/dnsmonster 
cd /tmp/dnsmonster
go get
go build -o dnsmonster -tags nolibpcap ./cmd/dnsmonster

有关如何创建静态链接二进制文件的更多信息，请查看[此](https://github.com/fenkohq/dnsmonster/blob/HEAD/Dockerfile) Dockerfile。

## Windows

在 Windows 上构建与 Linux 大致相同。只需确保已安装 `npcap`。克隆仓库（`--history 1` 即可），然后运行 `go get` 和 `go build ./cmd/dnsmonster`

如前所述，Windows 版本的二进制文件依赖于 [npcap](https://nmap.org/npcap/#download) 的安装。安装后，二进制文件应该可以直接使用。它已在 Windows 10 环境中测试过，且运行无问题。要查找接口名称以提供给 `--devName` 参数并开始嗅探，您需要执行以下操作：

  - 以管理员身份打开 cmd.exe，运行以下命令：`getmac.exe`，您将看到一个包含接口 MAC 地址的表格，以及一个类似如下的“传输名称”列：`\Device\Tcpip_{16000000-0000-0000-0000-145C4638064C}`
  - 在 `cmd.exe` 中像这样运行 `dnsmonster.exe`：```sh
dnsmonster.exe --devName \Device\NPF_{16000000-0000-0000-0000-145C4638064C}

# 架构

## 使用 Docker 进行一体化安装

![Basic AIO Diagram](https://raw.githubusercontent.com/fenkohq/dnsmonster/HEAD/static/dnsmonster-basic.svg)

在示例图中，捕获了DNS服务器流量的出/入口，之后在到达DNSMonster服务器之前添加了一个可选的流量聚合层。从DNS服务器出去的出站数据对于对DNS集群进行缓存和性能分析非常有用。如果聚合器不可用，可以将两个TAP直接连接到DNSMonster，并由两个DNSMonster代理查看流量。

运行 `./autobuild.sh` 会创建多个容器：

* 多个 `dnsmonster` 实例用于查看任何接口上的流量。接口列表将在 `autobuild.sh` 运行时提示。
* 一个 `clickhouse` 实例用于收集 `dnsmonster` 的输出，并将所有日志/数据保存到数据和日志目录。两者将在 `autobuild.sh` 运行时提示。
* 一个 `grafana` 实例用于查看 `clickhouse` 数据，并带有预构建的仪表板。

### 一体化演示

[![AIO Demo](https://raw.githubusercontent.com/fenkohq/dnsmonster/HEAD/static/aio_demo.svg)](static/aio_demo.svg)

## 企业部署

![Basic AIO Diagram](https://raw.githubusercontent.com/fenkohq/dnsmonster/HEAD/static/dnsmonster-enterprise.svg)

# 配置

DNSMonster 可以通过 3 种不同的方式进行配置：命令行选项、环境变量和配置文件。优先级顺序：

- 命令行选项（不区分大小写）
- 环境变量（始终大写）
- 配置文件（区分大小写，小写）
- 默认值（无配置）

## 命令行选项

注意，从 v0.9.5 开始，命令行参数不区分大小写

[//]: <> (start of command line options)```sh
# [capture]
# Device used to capture
--devname=

# Pcap filename to run
--pcapfile=

# dnstap socket path. Example: unix:///tmp/dnstap.sock, tcp://127.0.0.1:8080
--dnstapsocket=

# Port selected to filter packets
--port=53

# Capture Sampling by a:b. eg sampleRatio of 1:100 will process 1 percent of the incoming packets
--sampleratio=1:1

# Cleans up packet hash table used for deduplication
--dedupcleanupinterval=1m0s

# Set the dnstap socket permission, only applicable when unix:// is used
--dnstappermission=755

# Number of routines used to handle received packets
--packethandlercount=2

# Size of the tcp assembler
--tcpassemblychannelsize=10000

# Size of the tcp result channel
--tcpresultchannelsize=10000

# Number of routines used to handle tcp packets
--tcphandlercount=1

# Size of the channel to send packets to be defragged
--defraggerchannelsize=10000

# Size of the channel where the defragged packets are returned
--defraggerchannelreturnsize=10000

# Size of the packet handler channel
--packetchannelsize=1000

# Afpacket Buffersize in MB
--afpacketbuffersizemb=64

# BPF filter applied to the packet stream. If port is selected, the packets will not be defragged.
--filter=((ip and (ip[9] == 6 or ip[9] == 17)) or (ip6 and (ip6[6] == 17 or ip6[6] == 6 or ip6[6] == 44)))

# Use AFPacket for live captures. Supported on Linux 3.0+ only
--useafpacket

# The PCAP capture does not contain ethernet frames
--noetherframe

# Deduplicate incoming packets, Only supported with --devName and --pcapFile. Experimental 
--dedup

# Do not put the interface in promiscuous mode
--nopromiscuous

# [clickhouse_output]
# Address of the clickhouse database to save the results. multiple values can be provided.
--clickhouseaddress=localhost:9000

# Username to connect to the clickhouse database
--clickhouseusername=

# Password to connect to the clickhouse database
--clickhousepassword=

# Database to connect to the clickhouse database
--clickhousedatabase=default

# Table which data will be stored on clickhouse database
--clickhousetable=DNS_LOG

# Interval between sending results to ClickHouse. If non-0, Batch size is ignored and batch delay is used
--clickhousedelay=0s

# Clickhouse connection LZ4 compression level, 0 means no compression
--clickhousecompress=0

# Debug Clickhouse connection
--clickhousedebug

# Use TLS for Clickhouse connection
--clickhousesecure

# Save full packet query and response in JSON format.
--clickhousesavefullquery

# Use DNSTap identity field instead of ServerName for the identity field in ClickHouse
--clickhouseusednstapidentity

# What should be written to clickhouse. options:
#	0: Disable Output
#	1: Enable Output without any filters
#	2: Enable Output and apply skipdomains logic
#	3: Enable Output and apply allowdomains logic
#	4: Enable Output and apply both skip and allow domains logic
--clickhouseoutputtype=0

# Minimum capacity of the cache array used to send data to clickhouse. Set close to the queries per second received to prevent allocations
--clickhousebatchsize=100000

# Number of Clickhouse output Workers
--clickhouseworkers=1

# Channel Size for each Clickhouse Worker
--clickhouseworkerchannelsize=100000

# [elastic_output]
# What should be written to elastic. options:
#	0: Disable Output
#	1: Enable Output without any filters
#	2: Enable Output and apply skipdomains logic
#	3: Enable Output and apply allowdomains logic
#	4: Enable Output and apply both skip and allow domains logic
--elasticoutputtype=0

# elastic endpoint address, example: http://127.0.0.1:9200. Used if elasticOutputType is not none
--elasticoutputendpoint=

# elastic index
--elasticoutputindex=default

# Send data to Elastic in batch sizes
--elasticbatchsize=1000

# Interval between sending results to Elastic if Batch size is not filled
--elasticbatchdelay=1s

# [file_output]
# What should be written to file. options:
#	0: Disable Output
#	1: Enable Output without any filters
#	2: Enable Output and apply skipdomains logic
#	3: Enable Output and apply allowdomains logic
#	4: Enable Output and apply both skip and allow domains logic
--fileoutputtype=0

# Path to output folder. Used if fileoutputType is not none
--fileoutputpath=

# Interval to rotate the file in cron format
--fileoutputrotatecron=0 0 * * *

# Number of files to keep. 0 to disable rotation
--fileoutputrotatecount=4

# Output format for file. options:json, csv, csv_no_header, gotemplate. note that the csv splits the datetime format into multiple fields
--fileoutputformat=json

# Go Template to format the output as needed
--fileoutputgotemplate={{.}}

# [influx_output]
# What should be written to influx. options:
#	0: Disable Output
#	1: Enable Output without any filters
#	2: Enable Output and apply skipdomains logic
#	3: Enable Output and apply allowdomains logic
#	4: Enable Output and apply both skip and allow domains logic
--influxoutputtype=0

# influx Server address, example: http://localhost:8086. Used if influxOutputType is not none
--influxoutputserver=

# Influx Server Auth Token
--influxoutputtoken=dnsmonster

# Influx Server Bucket
--influxoutputbucket=dnsmonster

# Influx Server Org
--influxoutputorg=dnsmonster

# Minimum capacity of the cache array used to send data to Influx
--influxoutputworkers=8

# Minimum capacity of the cache array used to send data to Influx
--influxbatchsize=1000

# [kafka_output]
# What should be written to kafka. options:
#	0: Disable Output
#	1: Enable Output without any filters
#	2: Enable Output and apply skipdomains logic
#	3: Enable Output and apply allowdomains logic
#	4: Enable Output and apply both skip and allow domains logic
--kafkaoutputtype=0

# kafka broker address(es), example: 127.0.0.1:9092. Used if kafkaOutputType is not none
--kafkaoutputbroker=

# Kafka topic for logging
--kafkaoutputtopic=dnsmonster

# Minimum capacity of the cache array used to send data to Kafka
--kafkabatchsize=1000

# Output format. options:json, gob. 
--kafkaoutputformat=json

# Kafka connection timeout in seconds
--kafkatimeout=3

# Interval between sending results to Kafka if Batch size is not filled
--kafkabatchdelay=1s

# Compress Kafka connection
--kafkacompress

# Compression Type Kafka connection [snappy gzip lz4 zstd]; default(snappy).
--kafkacompressiontype=snappy

# Use TLS for kafka connection
--kafkasecure

# Path of CA certificate that signs Kafka broker certificate
--kafkacacertificatepath=

# Path of TLS certificate to present to broker
--kafkatlscertificatepath=

# Path of TLS certificate key
--kafkatlskeypath=

# [parquet_output]
# What should be written to parquet file. options:
#	0: Disable Output
#	1: Enable Output without any filters
#	2: Enable Output and apply skipdomains logic
#	3: Enable Output and apply allowdomains logic
#	4: Enable Output and apply both skip and allow domains logic
--parquetoutputtype=0

# Path to output folder. Used if parquetoutputtype is not none
--parquetoutputpath=

# Number of records to write to parquet file before flushing
--parquetflushbatchsize=10000

# Number of workers to write to parquet file
--parquetworkercount=4

# Size of the write buffer in bytes
--parquetwritebuffersize=256000

# [psql_output]
# What should be written to Microsoft Psql. options:
#	0: Disable Output
#	1: Enable Output without any filters
#	2: Enable Output and apply skipdomains logic
#	3: Enable Output and apply allowdomains logic
#	4: Enable Output and apply both skip and allow domains logic
--psqloutputtype=0

# Psql endpoint used. must be in uri format. example: postgres://username:password@hostname:port/database?sslmode=disable
--psqlendpoint=

# Psql table which data will be stored on database
--psqltable=DNS_LOG

# Number of PSQL workers
--psqlworkers=1

# Psql Batch Size
--psqlbatchsize=1

# Interval between sending results to Psql if Batch size is not filled. Any value larger than zero takes precedence over Batch Size
--psqlbatchdelay=0s

# Timeout for any INSERT operation before we consider them failed
--psqlbatchtimeout=5s

# Save full packet query and response in JSON format.
--psqlsavefullquery

# [sentinel_output]
# What should be written to Microsoft Sentinel. options:
#	0: Disable Output
#	1: Enable Output without any filters
#	2: Enable Output and apply skipdomains logic
#	3: Enable Output and apply allowdomains logic
#	4: Enable Output and apply both skip and allow domains logic
--sentineloutputtype=0

# Sentinel Shared Key, either the primary or secondary, can be found in Agents Management page under Log Analytics workspace
--sentineloutputsharedkey=

# Sentinel Customer Id. can be found in Agents Management page under Log Analytics workspace
--sentineloutputcustomerid=

# Sentinel Output LogType
--sentineloutputlogtype=dnsmonster

# Sentinel Output Proxy in URI format
--sentineloutputproxy=

# Sentinel Batch Size
--sentinelbatchsize=100

# Interval between sending results to Sentinel if Batch size is not filled. Any value larger than zero takes precedence over Batch Size
--sentinelbatchdelay=0s

# [splunk_output]
# What should be written to HEC. options:
#	0: Disable Output
#	1: Enable Output without any filters
#	2: Enable Output and apply skipdomains logic
#	3: Enable Output and apply allowdomains logic
#	4: Enable Output and apply both skip and allow domains logic
--splunkoutputtype=0

# splunk endpoint address, example: http://127.0.0.1:8088. Used if splunkOutputType is not none, can be specified multiple times for load balanace and HA
--splunkoutputendpoint=

# Splunk HEC Token
--splunkoutputtoken=00000000-0000-0000-0000-000000000000

# Splunk Output Index
--splunkoutputindex=temp

# Splunk Output Proxy in URI format
--splunkoutputproxy=

# Splunk Output Source
--splunkoutputsource=dnsmonster

# Splunk Output Sourcetype
--splunkoutputsourcetype=json

# Send data to HEC in batch sizes
--splunkbatchsize=1000

# Interval between sending results to HEC if Batch size is not filled
--splunkbatchdelay=1s

# [stdout_output]
# What should be written to stdout. options:
#	0: Disable Output
#	1: Enable Output without any filters
#	2: Enable Output and apply skipdomains logic
#	3: Enable Output and apply allowdomains logic
#	4: Enable Output and apply both skip and allow domains logic
--stdoutoutputtype=0

# Output format for stdout. options:json,csv, csv_no_header, gotemplate. note that the csv splits the datetime format into multiple fields
--stdoutoutputformat=json

# Go Template to format the output as needed
--stdoutoutputgotemplate={{.}}

# Number of workers
--stdoutoutputworkercount=8

# [syslog_output]
# What should be written to Syslog server. options:
#	0: Disable Output
#	1: Enable Output without any filters
#	2: Enable Output and apply skipdomains logic
#	3: Enable Output and apply allowdomains logic
#	4: Enable Output and apply both skip and allow domains logic
--syslogoutputtype=0

# Syslog endpoint address, example: udp://127.0.0.1:514, tcp://127.0.0.1:514. Used if syslogOutputType is not none
--syslogoutputendpoint=udp://127.0.0.1:514

# [victoria_output]
# Victoria Output Endpoint. example: http://localhost:9428/insert/jsonline?_msg_field=rcode_id&_time_field=time
--victoriaoutputendpoint=

# What should be written to Microsoft Victoria. options:
#	0: Disable Output
#	1: Enable Output without any filters
#	2: Enable Output and apply skipdomains logic
#	3: Enable Output and apply allowdomains logic
#	4: Enable Output and apply both skip and allow domains logic
--victoriaoutputtype=0

# Victoria Output Proxy in URI format
--victoriaoutputproxy=

# Number of workers
--victoriaoutputworkers=8

# Victoria Batch Size
--victoriabatchsize=100

# Interval between sending results to Victoria if Batch size is not filled. Any value larger than zero takes precedence over Batch Size
--victoriabatchdelay=0s

# [zinc_output]
# What should be written to zinc. options:
#	0: Disable Output
#	1: Enable Output without any filters
#	2: Enable Output and apply skipdomains logic
#	3: Enable Output and apply allowdomains logic
#	4: Enable Output and apply both skip and allow domains logic
--zincoutputtype=0

# index used to save data in Zinc
--zincoutputindex=dnsmonster

# zinc endpoint address, example: http://127.0.0.1:9200/api/default/_bulk. Used if zincOutputType is not none
--zincoutputendpoint=

# zinc username, example: [email protected]. Used if zincOutputType is not none
--zincoutputusername=

# zinc password, example: password. Used if zincOutputType is not none
--zincoutputpassword=

# Send data to Zinc in batch sizes
--zincbatchsize=1000

# Interval between sending results to Zinc if Batch size is not filled
--zincbatchdelay=1s

# Zing request timeout
--zinctimeout=10s

# [general]
# Garbage Collection interval for tcp assembly and ip defragmentation
--gctime=10s

# Duration to calculate interface stats
--capturestatsdelay=1s

# Mask IPv4s by bits. 32 means all the bits of IP is saved in DB
--masksize4=32

# Mask IPv6s by bits. 32 means all the bits of IP is saved in DB
--masksize6=128

# Name of the server used to index the metrics.
--servername=default

# Set debug Log format
--logformat=text

# Set debug Log level, 0:PANIC, 1:ERROR, 2:WARN, 3:INFO, 4:DEBUG
--loglevel=3

# Size of the result processor channel size
--resultchannelsize=100000

# write cpu profile to file
--cpuprofile=

# write memory profile to file
--memprofile=

# GOMAXPROCS variable
--gomaxprocs=-1

# Limit of packets logged to clickhouse every iteration. Default 0 (disabled)
--packetlimit=0

# Skip outputing domains matching items in the CSV file path. Can accept a URL (http:// or https://) or path
--skipdomainsfile=

# Hot-Reload skipdomainsfile interval
--skipdomainsrefreshinterval=1m0s

# Allow Domains logic input file. Can accept a URL (http:// or https://) or path
--allowdomainsfile=

# Hot-Reload allowdomainsfile file interval
--allowdomainsrefreshinterval=1m0s

# Skip TLS verification when making HTTPS connections
--skiptlsverification

# [metric]
# Metric Endpoint Service
--metricendpointtype=

# Statsd endpoint. Example: 127.0.0.1:8125 
--metricstatsdagent=

# Prometheus Registry endpoint. Example: http://0.0.0.0:2112/metric
--metricprometheusendpoint=

# Format for  output.
--metricformat=json

# Interval between sending results to Metric Endpoint
--metricflushinterval=10s

## 配置文件
你可以使用以下命令运行 `dnsmonster` 来使用配置文件：```shell
$ sudo dnsmonster --config=dnsmonster.ini

# Or you can use environment variables to set the configuration file path
$ export DNSMONSTER_CONFIG=dnsmonster.ini
$ sudo -E dnsmonster

注意：以上命令仅更改原始DNS日志数据的TTL，这是您大部分容量消耗的来源。为了确保调整每个聚合表的TTL，您可以运行以下命令：```sql
ALTER TABLE DNS_LOG MODIFY TTL DnsDate + INTERVAL 90 DAY;
ALTER TABLE `.inner.DNS_DOMAIN_COUNT` MODIFY TTL DnsDate + INTERVAL 90 DAY;
ALTER TABLE `.inner.DNS_DOMAIN_UNIQUE` MODIFY TTL DnsDate + INTERVAL 90 DAY;
ALTER TABLE `.inner.DNS_PROTOCOL` MODIFY TTL DnsDate + INTERVAL 90 DAY;
ALTER TABLE `.inner.DNS_GENERAL_AGGREGATIONS` MODIFY TTL DnsDate + INTERVAL 90 DAY;
ALTER TABLE `.inner.DNS_EDNS` MODIFY TTL DnsDate + INTERVAL 90 DAY;
ALTER TABLE `.inner.DNS_OPCODE` MODIFY TTL DnsDate + INTERVAL 90 DAY;
ALTER TABLE `.inner.DNS_TYPE` MODIFY TTL DnsDate + INTERVAL 90 DAY;
ALTER TABLE `.inner.DNS_CLASS` MODIFY TTL DnsDate + INTERVAL 90 DAY;
ALTER TABLE `.inner.DNS_RESPONSECODE` MODIFY TTL DnsDate + INTERVAL 90 DAY;
ALTER TABLE `.inner.DNS_SRCIP_MASK` MODIFY TTL DnsDate + INTERVAL 90 DAY;