ATHEROS SEARCH COMPONENT / INGESTION COORDINATOR
Octopus coordinates ingestion, document preparation, and PostgreSQL projections for Atheros Search. It persists events, jobs, and load results so processing can resume after a restart. This page includes current metrics and recorded history.
Read events from configured streams.
Claim jobs, build batches, and track loads.
Resume from saved progress and handle replay.
PIPELINE OBSERVABILITY
Check processing, queued work, and recorded production history to see where investigation should begin.
Pipeline readings refresh every 30 seconds. When live collection is delayed, the latest recorded totals and hourly history remain visible with their original measurement times. The page retries automatically and keeps a validated public snapshot in this browser for later visits. No example values or build-time snapshots are used.
The processing rate covers successfully committed broker deliveries at the responding coordinator, averaged over five minutes, including replays and parked records. Pending ledger rows and broker backlog are separate measurements. Live gauges appear once the rate window and collection are ready.
Historical peaks count rows in the production ingestion ledger across all paths and outcomes. They are not unique business events or a capacity benchmark. Days use UTC. Weeks run Monday to Sunday. Current periods are counted so far. History is refreshed separately, with its own measurement time.
Lifetime totals and hourly throughput windows count rows in the same ingestion ledger. The last-24-hours window holds 24 measured hourly totals and the last-7-days window holds 168. A measured hour may honestly be zero. A window that was never computed stays unavailable instead of showing zeros.
Each stage records the state needed by the next.
For platform and data teams running event pipelines that need to recover from failures and account for repeated delivery.
Engineering traces delivery and recovery state. Operations checks throughput, backlog, and the freshness of reported activity.
FOR ENGINEERS
FOR OPERATORS
Built in Scala with Cats Effect and FS2, using Redpanda streams and PostgreSQL for persistent state.
Start incident triage with the consumer group, topic, partition, and offset stored on each ingestion record. Replayed records keep their original first-seen time so you can trace the original delivery.
Recover after a restart using committed consumer offsets and durable deduplication for repeated delivery. New groups begin at the earliest records still retained.
Time-limited claims coordinate workers competing for queued work. Inspect stored jobs, batches, and outgoing messages to see the state of each load when processing stalls.
Octopus updates PostgreSQL data views and derives alerts. Atheros Search runs the separate workers that create embeddings.
DELIVERY AND RECOVERY
Delivery can repeat. Octopus records progress and uses durable deduplication to handle replay. Records outside the configured streams are not covered.
Consumer offsets, job state, and load results give operators a place to start when work stalls.
A stored record of consumed events and their outcomes, identified by consumer group, topic, partition, and offset.
A time-bounded claim on work used to coordinate processing.
Durable records of messages waiting for dispatch.
A delivery model in which records may be delivered again. Durable deduplication handles repeated work.
The highest historical ledger-row count in a UTC calendar day or Monday-to-Sunday ISO week, including the current period so far.