Observability and Error Handling

Observability is implemented through shared package @salesarc/observability and API middleware.

Correlation IDs

Generation

packages/observability/src/correlation.ts generates IDs in format:

• req_<16 hex chars>

Request propagation

apps/api/src/middleware/correlation.ts:

1. Reads incoming x-correlation-id if present
2. Otherwise generates new id
3. Stores it on Hono context
4. Echoes it in response header X-Correlation-Id

This ensures traceability across logs and API responses.

Structured Logging

packages/observability/src/logger.ts uses Pino.

Logger properties

• JSON output
• Level from LOG_LEVEL environment variable
• ISO timestamps
• Standardized level formatting

Child loggers

Services and routes create child loggers with context:

• logger.child({ route: "webhooks" })
• logger.child({ service: "ingestion" })
• logger.child({ service: "email" })

PII redaction

Sensitive paths are redacted centrally, including:

• email, mobile
• Auth headers
• Common nested user/customer fields

Redaction is process-wide, reducing accidental leakage risk.

Sentry

packages/observability/src/sentry.ts initializes optional Sentry capture.

Behavior:

• Disabled if DSN missing
• Environment-aware traces sample rate
• Defensive beforeSend strips user PII fields

API startup (apps/api/src/index.ts) initializes Sentry before route registration.

Error Translation Layer

apps/api/src/middleware/error-handler.ts provides unified API error formatting.

Handled categories:

Error Class	Source	HTTP Status
AppError	apps/api/src/lib/errors.ts	Custom (set per error)
AuthError	@salesarc/auth	401
RbacError	@salesarc/rbac	403
TenantError	@salesarc/tenant	403/404
HTTPException	Hono framework	Custom
ZodError	Zod validation	400
Unknown	Any unhandled error	500

All responses include requestId from correlation context.

Standard response shape

{
  "code": "ERROR_CODE",
  "message": "Human-readable message",
  "requestId": "req_abc123..."
}

For validation failures:

{
  "code": "VALIDATION_ERROR",
  "message": "...",
  "issues": [...],
  "requestId": "req_..."
}

Unhandled exceptions are logged (with stack trace) and returned as INTERNAL_ERROR (500) without stack leakage.

Error Code Registry

packages/types/src/api.ts centralizes error codes across domains:

Domain	Example Codes
Auth	TOKEN_INVALID, UNAUTHORIZED, TOKEN_EXPIRED
RBAC	FORBIDDEN, INSUFFICIENT_ROLE
Tenant	TENANT_NOT_FOUND, TENANT_NOT_MEMBER
Wallet	WALLET_FROZEN, INSUFFICIENT_BALANCE
POS	POS_CONNECTION_FAILED
General	INTERNAL_ERROR, VALIDATION_ERROR, NOT_FOUND

Using shared constants prevents frontend/backend divergence in error semantics.

Recommended Dashboards

Based on current runtime behavior, priority observability views are:

1. API status code rate by route group.
2. Webhook ingestion outcome distribution (credited, duplicate, stored_anonymous, no_rule_no_credit).
3. Cron polling summary and failures by connection id.
4. Auth/profile failures by code and status.
5. Admin action volume by action type.
6. Email send success/failure rate.

Operational Logging Guidelines

1. Include tenantId, userId, and correlationId where available.
2. Never log decrypted tokens or raw authorization headers.
3. Keep webhook logs high-signal (signature result, tenant resolution, raw event id).
4. Treat async errors after fast-ack as first-class incidents; they do not surface to caller.
5. Email failures should be logged but never escalated (fire-and-forget pattern).