Queue Infrastructure Observability — Daily Report 2026-06-11

Queue Observability Snapshot — 2026-06-11

Queue Depth

State	Count	Trend (vs 2026-06-07)
done	604	≈
active	64	↓
blocked	3	≈
queued	2	↓

Stuck Tasks (>4h idle in active/)

• 57 stuck tasks detected (threshold: 4h)
• Oldest: (114.4h idle) — CRITICAL if >72h

Task	Age (hrs)	Severity
schema-drift-20260530.md	216.6	CRITICAL
cluster-fix-calliope-kanban-bug.md	179.4	CRITICAL
cluster-fix-chat-shim-gosu.md	179.3	CRITICAL
4a346f5d-vt-004-mastodon-public-timeline-unauthenticated.md	178.9	CRITICAL
fix-test-assertion-failed-20260604.md	173.5	CRITICAL
crashloop-calliope-kanban-2026-06-04.md	150.1	CRITICAL
crashloop-chat-shim-gosu-2026-06-04.md	149.7	CRITICAL
cluster-janitor-long-jobs-2026-06-04.md	135.7	CRITICAL
2010f0a5-long-running-job-analysis.md	117.7	CRITICAL
44c52e92-pods-pending-10-min.md	115.9	CRITICAL

Failure Tag Triage (active/ tasks modified in last 24h)

Category	Count	Severity
Unknown tag	0	✅ CLEAN
CrashLoopBackOff	1	LOW
Timeout	3	LOW
API connection fail	0	NORMAL

Active Task Recency

• Most recent modification in active/: 0.6h ago
• Baseline snapshots on disk: 11 (last: 2026-06-07)

Assessment

Metric	Status	Notes
Throughput	✅ Healthy	604 total done — consistent growth
Active load	✅ Normal	64 active tasks
Blocked	✅ Clean	3 blocked — 3 items pending unblock
Unknown tags	✅ Fixed	0 in 24h (was 47 before fix-recurring-unknown)
Stuck tasks	⚠️ High	57 idle >4h — 45 stale >72h

Key Findings

1. 57 tasks have been idle >4 hours. 42 are older than 4 days — candidates for archival or re-pickup.
2. Unknown failure tag is zero — fix-recurring-unknown remains effective (47 → 0 since May 6).
3. Blocked queue has 3 items — awaiting pvs sign-off on autopilot-db secret rotation (stale >10 days), stuck guard triggered 5+ days ago, and 1 calliope crashloop fix.
4. Baseline snapshots stopped on Jun 7 — 4-day gap in daily collection. This run restores continuity.

Recommendations for pvs

1. Archive stale active tasks: 45 tasks >72h idle with no progress — archive to done/ or failed/ based on task content review.
2. Unblock coaching note: is a meta-task stuck since May 30. Either unblock for autopilot re-processing or archive as completed (tag taxonomy drift was partially fixed, schema updated).
3. Secret rotation: has a fully-diagnosed root cause (autopilot-db password mismatch) needing pvs sign-off for secret update. Blocked since June 1 (>10 days).