johba/disinto
1
0
Fork 0
Code Issues 4 Pull requests Projects Releases Packages Wiki Activity Actions

fix: address review — error fallback, schema consistency, line-level recovery (#718)

Browse source 
- Add jq error fallback to TOP_PAGES pipeline (matches TOP_REFERRERS guard)
- Add referred_visitors and response_time to empty-period report schema
- Switch Caddy log parsing to jq -R with try/fromjson for line-level recovery

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
openhands 2026-03-26 12:47:18 +00:00
parent 192fc39198
commit 946921431c
1 changed files with 6 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

9
site/collect-engagement.sh
View file

@ -69,8 +69,9 @@ log "Parsing ${CADDY_LOG} for entries since $(date -u -d "@${CUTOFF_TS}" +%Y-%m-
# request.headers.User-Agent, status, size, duration # request.headers.User-Agent, status, size, duration
# #
# Filter to last 24h, exclude assets/bots, produce a clean JSONL stream. # Filter to last 24h, exclude assets/bots, produce a clean JSONL stream.
PARSED=$(jq -c --argjson cutoff "$CUTOFF_TS" ' PARSED=$(jq -Rc --argjson cutoff "$CUTOFF_TS" '
select(.ts >= $cutoff) try fromjson
| select(.ts >= $cutoff)
| select(.request.uri != null) | select(.request.uri != null)
| { | {
ts: .ts, ts: .ts,
@ -99,8 +100,10 @@ if [ -z "$PARSED" ]; then
total_requests: 0, total_requests: 0,
unique_visitors: 0, unique_visitors: 0,
page_views: 0, page_views: 0,
referred_visitors: 0,
top_pages: [], top_pages: [],
top_referrers: [], top_referrers: [],
response_time: { p50_seconds: 0, p95_seconds: 0, p99_seconds: 0 },
note: "no entries in period" note: "no entries in period"
}' > "${EVIDENCE_DIR}/${REPORT_DATE}.json" }' > "${EVIDENCE_DIR}/${REPORT_DATE}.json"
log "Empty report written to ${EVIDENCE_DIR}/${REPORT_DATE}.json" log "Empty report written to ${EVIDENCE_DIR}/${REPORT_DATE}.json"
@ -126,7 +129,7 @@ UNIQUE_VISITORS=$(printf '%s\n' "$PAGES" | jq -r '.ip' | sort -u | wc -l | tr -d
TOP_PAGES=$(printf '%s\n' "$PAGES" | jq -r '.uri' \ TOP_PAGES=$(printf '%s\n' "$PAGES" | jq -r '.uri' \
| sort | uniq -c | sort -rn | head -10 \ | sort | uniq -c | sort -rn | head -10 \
| awk '{printf "{\"path\":\"%s\",\"views\":%d}\n", $2, $1}' \ | awk '{printf "{\"path\":\"%s\",\"views\":%d}\n", $2, $1}' \
| jq -sc '.') | jq -sc '.' 2>/dev/null || echo '[]')
# Top referrers (exclude direct/self) # Top referrers (exclude direct/self)
TOP_REFERRERS=$(printf '%s\n' "$PAGES" | jq -r '.referer' \ TOP_REFERRERS=$(printf '%s\n' "$PAGES" | jq -r '.referer' \