
分布式混沌工程平台,用于对视频会议系统进行负载测试。模拟超过1500个WebRTC参与者,传输H.264/Opus流,并注入网络混沌尖峰,以验证系统在降级条件下的韧性
媒体处理管道:
控制平面:
参与者池:
participant_id % total_partitions = partition_id 自动跨 Pod 分区Kubernetes 自动配置:
orchestrator-3 → PARTITION_ID=3base_port + (partition_id × 10000) + participant_indexUDP 中继链(仅 Kubernetes):
Orchestrator Pods (10×) → UDP :5000 → udp-relay Pod (Python)
→ Length-Prefixed TCP :5001 → kubectl port-forward 15001:5001
→ tools/udp-relay (Go) → UDP :5002 → Your Receiver
WebRTC 基础设施:
客户端集成:
可观测性栈(可选):
/metrics 端点抓取数据prometheus.io/scrape: "true"每个虚拟参与者生成真实的媒体流:
五种尖峰类型模拟现实网络条件:
尖峰按可配置策略分布在测试持续时间内:
Kubernetes 部署使用参与者分区进行水平扩展:
participant_id % total_partitions == partition_idbase_port + (partition_id * 10000) + participant_index最佳用途:开发、调试、小规模测试(1-100 参与者)
# 启动编排器
go run cmd/main.go
# 另一个终端:启动 UDP 接收器
go run examples/go/udp_receiver.go 5002
# 编辑 config/config.json 设置 num_participants: 10
# 运行混沌测试
go run tools/chaos-test/main.go -config config/config.json
执行过程:
:8080127.0.0.1:5002 发送 UDP配置(config/config.json):
{
"base_url": "http://localhost:8080",
"media_path": "public/rick-roll.mp4",
"num_participants": 10,
"duration_seconds": 300,
"spikes": {
"count": 20,
"interval_seconds": 5,
"types": { "rtp_packet_loss": {...}, "network_jitter": {...} }
},
"spike_distribution": {
"strategy": "random",
"min_spacing_seconds": 5,
"jitter_percent": 15
}
}
最佳用途:隔离测试、CI/CD、中规模测试(100-500 参与者)
前提条件:
docker-compose# 构建并启动编排器容器
./scripts/start_everything.sh build
# 另一个终端:启动 UDP 接收器
go run examples/go/udp_receiver.go 5002
# 编辑 config/config.json 设置 num_participants: 100
# 运行混沌测试(目标为容器)
go run tools/chaos-test/main.go -config config/config.json
资源限制(编辑 docker-compose.yaml):
services:
orchestrator:
deploy:
resources:
limits:
cpus: "14.0"
memory: 6G # 增加以容纳更多参与者
扩缩指南:
| Docker 内存 | 最大参与者数 | CPU 核心数 |
|---|---|---|
| 8 GB | ~100 | 4 |
| 16 GB | ~250 | 8 |
| 24 GB | ~400 | 12 |
| 32 GB | ~500 | 14 |
最佳用途:大规模测试(500-1500 参与者)、水平扩展、生产验证
前提条件:
# Nix 提供:Go、Docker、kubectl、kind、ffmpeg
nix develop
# 或使用 direnv 自动激活
echo "use flake" > .envrc
direnv allow
# 自动部署并优化设置(检测系统资源)
./scripts/start_everything.sh run -config config/config.json
# 或指定自定义媒体文件
./scripts/start_everything.sh run --media=path/to/video.mp4 -config config/config.json
执行过程:
kubectl port-forward选项 A:UDP 接收器(推荐用于 Kubernetes)
# 接收来自所有 1500 个参与者的聚合流
go run ./examples/go/udp_receiver.go 5002
选项 B:WebRTC 接收器(多个参与者)
# 通过 WebRTC 连接最多 150 个参与者
go run ./examples/go/webrtc_receiver.go http://localhost:8080 <test_id> 150
架构流程:
1500 个参与者分布在 10 个 Pod 上
→ 每个 Pod:150 个参与者
→ 按 participant_id % 10 分区
→ 全部向 udp-relay:5000 发送 UDP
→ UDP 中继聚合 → TCP :5001
→ kubectl port-forward 15001:5001
→ 本地中继转换 TCP → UDP :5002
→ 你的接收器获取全部 1500 个流
注意:start_everything.sh 脚本会自动设置:
# 构建并加载镜像
docker build -t chaos-monkey-orchestrator:latest .
kind load docker-image chaos-monkey-orchestrator:latest
# 部署
kubectl apply -f k8s/orchestrator/orchestrator.yaml
kubectl apply -f k8s/udp-relay/udp-relay.yaml
# 等待 Pod 就绪
kubectl wait --for=condition=ready pod -l app=orchestrator --timeout=300s
# 端口转发 UDP 中继
kubectl port-forward udp-relay 15001:5001 &
# 启动本地 TCP→UDP 中继
go run tools/udp-relay/main.go &
# 另一个终端:启动接收器
go run ./examples/go/udp_receiver.go 5002
# 另一个终端:运行混沌测试
go run tools/chaos-test/main.go -config config/config.json
# 删除 Kubernetes 资源
./scripts/cleanup.sh
# 或删除整个集群
kind delete cluster --name av-chaos-monkey
# 为 Linux x86_64 构建(最常见)
nix build .#packages.x86_64-linux.av-chaos-monkey
# 为 ARM64 构建(Raspberry Pi、AWS Graviton)
nix build .#packages.aarch64-linux.av-chaos-monkey
# 为 macOS Intel 构建
nix build .#packages.x86_64-darwin.av-chaos-monkey
# 为 macOS Apple Silicon 构建
nix build .#packages.aarch64-darwin.av-chaos-monkey
# 二进制文件位置
./result/bin/main
# 创建测试
POST /api/v1/test/create
{
"test_id": "optional_id",
"num_participants": 100,
"video": {...},
"audio": {...},
"duration_seconds": 600,
"spikes": [...],
"spike_distribution": {
"strategy": "even",
"min_spacing_seconds": 5,
"jitter_percent": 15
}
}
# 启动测试
POST /api/v1/test/{test_id}/start
# 获取指标
GET /api/v1/test/{test_id}/metrics
# 停止测试
POST /api/v1/test/{test_id}/stop
# 获取 SDP 提议
GET /api/v1/test/{test_id}/sdp/{participant_id}
# 设置 SDP 应答
POST /api/v1/test/{test_id}/sdp/{participant_id}
{"sdp_answer": "v=0..."}
# 注入尖峰
POST /api/v1/test/{test_id}/spike
{
"spike_id": "unique_id",
"type": "rtp_packet_loss",
"duration_seconds": 30,
"participant_ids": [1001, 1002],
"params": {"loss_percentage": "15"}
}
| 类型 | 参数 | 效果 |
|---|---|---|
rtp_packet_loss | loss_percentage (0-100) | 在 RTP 层丢弃数据包 |
network_jitter | base_latency_ms, jitter_std_dev_ms | 添加延迟变化 |
bitrate_reduce | new_bitrate_kbps | 限制视频编码码率 |
frame_drop | drop_percentage (0-100) | 跳过视频帧 |
bandwidth_limit | bandwidth_kbps | 限制总吞吐量 |
{
"spike_distribution": {
"strategy": "even",
"min_spacing_seconds": 5,
"jitter_percent": 15,
"respect_min_offset": true
}
}
# 提供的接收器,包含 RTP 解析
go run examples/go/udp_receiver.go 5002
输出:
Listening for RTP packets on UDP port 0.0.0.0:5002
Packet #100 from 127.0.0.1:xxxxx:
Participant ID: 1001
Payload Type: 96 (H.264 video)
Sequence: 1234
Timestamp: 90000
SSRC: 1001000
Payload Size: 1200 bytes
═══════════════════════════════════════════════════════════
PACKET STATISTICS
═══════════════════════════════════════════════════════════
Duration: 60s
Total Packets: 180000 (3000 pkt/s)
Total Bytes: 450 MB (60 Mbps)
Media Type Breakdown:
Video (H.264): 120000 packets (66.7%)
Audio (Opus): 60000 packets (33.3%)
Unique Streams (SSRCs): 1500
Unique Participants: 1500
# 单个参与者
go run ./examples/go/webrtc_receiver.go http://localhost:8080 <test_id>
# 多个参与者(最多 150)
go run ./examples/go/webrtc_receiver.go http://localhost:8080 <test_id> 150
# 示例(使用实际测试 ID)
go run ./examples/go/webrtc_receiver.go http://localhost:8080 chaos_test_1770831684 150
注意:WebRTC 需要 1:1 连接。对于 Kubernetes,建议使用 UDP 接收器,它会自动聚合所有参与者。
RTP 数据包格式:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC |M| PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Extension ID=1 | Length=4 | Participant ID (uint32) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| H.264/Opus Payload |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
负载类型:
96: H.264 视频(RFC 6184)111: Opus 音频(RFC 7587)参与者 ID 提取:
// Extension bit set?
if (packet[0] & 0x10) != 0 {
offset := 12 + int(packet[0]&0x0F)*4 // Skip CSRC
extID := binary.BigEndian.Uint16(packet[offset:])
if extID == 1 {
participantID := binary.LittleEndian.Uint32(packet[offset+4:])
}
}
| 参与者数 | 内存 | CPU | 带宽 |
|---|---|---|---|
| 100 | 2GB | 2 核 | 250 Mbps |
| 500 | 6GB | 8 核 | 1.2 Gbps |
| 1000 | 12GB | 16 核 | 2.5 Gbps |
| 1500 | 18GB | 24 核 | 3.7 Gbps |
每个参与者(1280x720@30fps + Opus):
# 暴露在 /metrics 端点
av_chaos_monkey_participants_total
av_chaos_monkey_packets_sent_total
av_chaos_monkey_bytes_sent_total
av_chaos_monkey_spikes_active
av_chaos_monkey_packet_loss_percent
av_chaos_monkey_jitter_ms
# Docker 模式:启动监控栈
docker-compose --profile monitoring up
# Kubernetes 模式:部署监控
kubectl apply -f k8s/monitoring/prometheus-rbac.yaml
kubectl apply -f k8s/monitoring/prometheus.yaml
kubectl apply -f k8s/monitoring/grafana.yaml
# 访问 Grafana
# Docker:http://localhost:3000
# Kubernetes:http://localhost:30030(NodePort)
# 默认凭据:admin/admin
# 访问 Prometheus
# Docker:http://localhost:9091
# Kubernetes:http://localhost:30090(NodePort)
Kubernetes 自动发现:
prometheus.io/scrape: "true"/metrics# 获取测试指标
curl http://localhost:8080/api/v1/test/{test_id}/metrics | jq
# 输出
{
"aggregate": {
"total_frames_sent": 45000,
"total_packets_sent": 180000,
"total_bitrate_kbps": 250000,
"avg_jitter_ms": 12.5,
"avg_packet_loss": 2.3,
"avg_mos_score": 4.1
}
}
# 检查 UDP 目标配置
kubectl logs orchestrator-0 | grep "UDP transmission enabled"
# 验证 UDP 中继是否运行
kubectl get pod udp-relay
# 检查端口转发
ps aux | grep "kubectl port-forward"
# 测试 UDP 连通性
nc -u -z localhost 5002
# 检查 TURN 服务器
kubectl get svc coturn-lb
# 验证 ICE 候选
kubectl logs orchestrator-0 | grep "ICE"
# 测试 TURN 连通性
turnutils_uclient -v -u webrtc -w webrtc123 <turn-server>:3478
# 检查每个 Pod 的参与者数量
kubectl exec orchestrator-0 -- curl -s http://localhost:8080/api/v1/test/{test_id}/metrics | jq '.participants | length'
# 减少参与者数量或增加 Pod 数量
go run tools/k8s-start/main.go -replicas 10 -participants 1000
# 增加 Docker 内存(Docker Desktop)
# 设置 → 资源 → 内存 → 16GB
单个 UDP 套接字无法处理 3000+ 并发流而不导致内核缓冲区溢出。解决方案:
setsockopt(SO_RCVBUF, 8MB)BSD 3-Clause License
欢迎贡献!关键领域: