Quartz 4

Smart Groceries Session 2026-05-25

2 min read

Smart Groceries — Session 2026-05-25

Start: 04:09 AEST
Duration: ~8 minutes
Outcome: Root cause identified for CronJob failures

Task: Investigate why CronJob smart-groceries-catalogue-scrape has been failing

Findings

The daily scrape CronJob (smart-groceries-catalogue-scrape in ai-agents namespace) has never successfully populated the database since it was restored on 2026-05-12. The DB remains at 0 products (schema intact, stores: 2, categories: 21).

Root Cause: GitLab authentication failure in init container

The CronJob’s init container (clone-and-install) fails with:

remote: HTTP Basic: Access denied. If a password was provided for Git authentication, the password was incorrect or you're required to use a token instead of a password. If a token was provided, it was either incorrect, expired, or improperly scoped.
fatal: Authentication failed for 'https://gitlab.paralla.org/hermes/smart-groceries.git/'

The .git-credentials file contains hermes:***@gitlab.paralla.org — the password/token is literally *** (three asterisks), not a valid PAT.

This was introduced when the CronJob was restored on May 12. The original scrape runs (May 7) used a different authentication method that worked, but after the CronJob re-creation/fix cycle, the token became invalid.

Evidence

DateEventResult
2026-05-07Manual scrape + local run~5741 products imported (bakery only)
2026-05-12CronJob restored, init container fix applied (apt-get install -y git)Token already broken — never verified
2026-05-19pvs confirmed git install fixToken still broken, nobody noticed
2026-05-24 07:32 AESTScheduled run attemptedFailed (BackoffLimitExceeded)
2026-05-25 04:16 UTCManual test triggeredSame auth failure confirmed

What I tried

  1. ✅ Confirmed DB state — schema OK, 0 products
  2. ✅ Verified SSH auth works (git ls-remote git@gitlab.paralla.org:hermes/smart-groceries.git returns HEAD)
  3. ✅ Triggered manual CronJob run, captured init container logs — auth failure confirmed
  4. ✅ Checked .git-credentials — token is literally ***, not a real PAT

Option A (easiest): Generate a new GitLab personal access token for hermes user with read_repository scope and update:

  • /opt/data/.git-credentials on the hermes-data PVC
  • The CronJob init container’s git clone URL (embed new token or use env var)

Option B (more robust): Mount an SSH key into the init container via K8s secret instead of relying on HTTPS basic auth. SSH already works for the hermes user (id_ed25519 at ~/.ssh/).

Status: BLOCKED

Cannot proceed with data import without resolving GitLab authentication. This is a credentials issue, not a code or infrastructure problem.

@pvs: Please either provide a valid PAT for the hermes user, or approve switching the init container to SSH-based git clone.

Graph View