Runbooks

These runbooks are aligned to the current deployment/runtime model (Render API, Vercel web, Supabase, Square).

Runbook 1: Square OAuth Connect Fails

Trigger

Merchant sees error on POS connect flow (/pos?error=...).

Immediate checks

1. Verify API env vars:
   • SQUARE_APP_ID
   • SQUARE_APP_SECRET
   • SQUARE_ENVIRONMENT
   • API_BASE_URL
   • FRONTEND_URL
   • ENCRYPTION_KEY
2. Call debug endpoint with authenticated client/admin token:

curl -H "Authorization: Bearer <token>" \
  https://<api-origin>/api/v1/client/debug/square

3. Confirm generated OAuth host is connect.squareupsandbox.com (sandbox) or connect.squareup.com (prod).

Resolution

• Fix env mismatch and redeploy API.
• Ensure Square dashboard redirect URI exactly matches https://<api-origin>/api/v1/oauth/square/callback.
• Re-attempt connect flow from POS page.

Runbook 2: Webhooks Received but No Wallet Credits

Trigger

Square webhooks appear accepted but points are not being credited.

Immediate checks

1. Ensure webhook signature key is configured in production.
2. Verify active connection exists for merchant account id in pos_connections.
3. Inspect recent raw webhook rows (pos_webhook_events_raw) and processed state.
4. Check ingestion logs for outcomes: stored_anonymous, duplicate, no_rule_no_credit.

SQL triage snippets

-- Recent webhook events
SELECT id, tenant_id, provider, event_type, processed, received_at
FROM pos_webhook_events_raw
ORDER BY received_at DESC
LIMIT 50;

-- Active reward rule for tenant
SELECT id, tenant_id, version, is_active
FROM reward_rules
WHERE tenant_id = '<tenant-id>' AND is_active = true;

Resolution

• If no_rule_no_credit: create/activate reward rule for tenant.
• If stored_anonymous: customer identity could not be matched; verify consumer email/phone consistency.
• If repeated async failures: inspect API logs and fix normalization/enrichment error path.

Runbook 3: Square Poll Endpoint Unauthorized or Failing

Trigger

Scheduled poll does not process payments.

Immediate checks

1. CRON_SECRET is set on API runtime.
2. Scheduler sends X-Cron-Secret header with exact value.
3. Endpoint health:

curl -X POST \
  -H "X-Cron-Secret: <CRON_SECRET>" \
  https://<api-origin>/api/v1/internal/cron/square-poll-payments

Resolution

• Fix scheduler secret/header mismatch.
• Ensure active Square connections and decryptable tokens exist.
• Validate SQUARE_APP_SECRET and token refresh path if refresh errors appear.

Runbook 4: Merchant Stuck in Pending Approval

Trigger

Merchant can log in but remains blocked from app usage.

Immediate checks

1. Check user status via admin search endpoint:

curl -H "Authorization: Bearer <admin-token>" \
  "https://<api-origin>/api/v1/admin/users/search?q=<email>"

2. Confirm user role/status in DB (users.role, users.status).

Resolution

Approve user:

curl -X PATCH \
  -H "Authorization: Bearer <admin-token>" \
  https://<api-origin>/api/v1/admin/users/<user-id>/approve

This also provisions merchant workspace for client role when needed.

Runbook 5: Wallet Balance Discrepancy

Trigger

Consumer or admin reports wallet balance mismatch.

Immediate checks

1. Compare wallet snapshot vs ledger sum.
2. Identify recent ledger writes and corresponding transactions.

SQL triage snippet

SELECT
  w.id,
  w.points_balance AS snapshot,
  COALESCE(SUM(CASE WHEN l.entry_type = 'credit' THEN l.points ELSE -l.points END), 0) AS ledger_sum
FROM wallets w
LEFT JOIN wallet_ledger l ON l.wallet_id = w.id
WHERE w.id = '<wallet-id>'
GROUP BY w.id, w.points_balance;

Resolution

• Trace ingestion/redemption path for duplicate or missing writes.
• If administrative correction is required, use:

POST /api/v1/admin/wallets/:walletId/adjust

Do not patch balances directly without audit.

Escalation

1. On-call engineer triages with logs + endpoint checks.
2. Service owner resolves code/config root cause.
3. For data integrity incidents, involve engineering lead and document postmortem with corrective actions.