blacksky-algorithms/atprotoFork of the AT Protocol reference implementation with performance-optimized AppView, Rust-based firehose indexer, Redis caching, and community features for self-hosted social networking at scale.
This is Blacksky's fork of the AT Protocol reference implementation by Bluesky Social PBC. It powers the AppView at api.blacksky.community.
We're publishing this for transparency and so other communities can benefit from the work. This repository is not accepting contributions, issues, or PRs. If you want the canonical atproto implementation, use bluesky-social/atproto.
All changes are in packages/bsky (appview logic), services/bsky (runtime config), and one custom migration. Everything else is upstream.
The upstream dataplane includes a TypeScript firehose consumer (subscription.ts) that indexes events directly. We replaced it with rsky-wintermute, a Rust indexer, for several reasons:
The dataplane and appview from this repo still run as-is. They read from the PostgreSQL database that wintermute writes to. We just don't start the built-in firehose subscription.
These are broadly useful to anyone self-hosting an AppView at scale.
LATERAL JOIN query optimization (packages/bsky/src/data-plane/server/routes/feeds.ts)
getTimeline and getListFeed rewritten with PostgreSQL LATERAL JOINs to force per-user index usage instead of full table scans. Major improvement for users following thousands of accounts.Redis caching layer (packages/bsky/src/data-plane/server/cache/)
Timestamp objects lose their .toDate() method after JSON round-tripping through Redis, causing incomplete profile hydration on cache hits. We currently run with Redis caching disabled. The fix is to serialize timestamps as ISO strings on cache write and reconstruct on read.Notification preferences server-side enforcement (packages/bsky/src/api/app/bsky/notification/listNotifications.ts)
reasons, the server applies the user's saved notification preferences. Without this, preferences are only enforced client-side and have no effect.Auth verifier stale signing key fix (packages/bsky/src/auth-verifier.ts)
forceRefresh), bypasses the dataplane's in-memory identity cache and resolves the DID document directly from PLC directory. Fixes authentication failures after account migration where the signing key rotates but the cache holds the old key.JSON sanitization (packages/bsky/src/data-plane/server/routes/records.ts)
\u0000) and control characters from stored records before JSON parsing. These are valid per RFC 8259 but rejected by Node.js JSON.parse(), causing silent rowToRecord parse failures in the dataplane that surface as missing posts.Infrastructure for private community posts that live on the AppView rather than individual PDSes. Specific to how Blacksky works, but could serve as a reference for other communities.
community.blacksky.feed.* with endpoints for submit, get, delete, timeline, and thread viewscommunity_post table (migration: 20260202T120000000Z-add-community-post.ts)getPostThreadV2 for mixed standard/community post threadsBLACKSKY_MEMBERSHIP_DB_URL)Bluesky Relay (bsky.network)
|
v
rsky-wintermute -----> PostgreSQL 17 <----- Palomar
(Rust indexer) | (Go search)
- firehose consumer | |
- backfiller | v
- label indexer | OpenSearch
- direct indexer |
v
bsky-dataplane (gRPC :2585) <--- Redis (optional)
|
v
bsky-appview (HTTP :2584)
|
v
Reverse proxy (Caddy/nginx)
| Component | Source | Purpose |
|---|---|---|
| rsky-wintermute | blacksky-algorithms/rsky | Rust firehose indexer: consumes events, backfills repos, indexes records into PostgreSQL |
| rsky-relay | blacksky-algorithms/rsky | AT Protocol relay for receiving moderation labels from labeler services |
| rsky-video | blacksky-algorithms/rsky | Video upload service: transcodes via Bunny Stream CDN, uploads blob refs to user PDSes |
| bsky-dataplane | This repo (services/bsky) | gRPC data layer over PostgreSQL |
| bsky-appview | This repo (services/bsky) | HTTP API server for app.bsky.* XRPC endpoints |
| Palomar | blacksky-algorithms/indigo | Full-text search: indexes profiles and posts into OpenSearch with follower count boosting |
| palomar-sync | blacksky-algorithms/rsky | Syncs follower counts and PageRank scores from PostgreSQL to OpenSearch |
Wintermute is a monolithic Rust service with four parallel processing paths:
bsky.network firehose via WebSocket, writes events to Fjall (embedded key-value store) queuesON CONFLICT for idempotencyAdditional CLI tools included in the rsky repo:
queue_backfill -- queue DIDs for backfill from CSV, PDS discovery, or direct DID listsdirect_index -- fetch and index specific repos bypassing queues (useful for fixing individual accounts)label_sync -- replay label streams from cursor 0 to catch up on missed negationsplc_import -- bulk import handle/DID mappings from PLC directorypalomar-sync -- sync follower counts and PageRank to OpenSearchVideo upload service for users whose PDS doesn't support Bluesky's video.bsky.app. Uses its own DID (did:web:video.blacksky.community) to authenticate to user PDSes via service auth JWTs. Flow: