CANDOR

Decision 1

Chose: FastAPI

Over: Django REST Framework, Flask

Because: Native async support for LLM polling chains, automatic OpenAPI generation, and Pydantic validation that maps cleanly to the 89-field extraction schema. Django's ORM would have fought the flexible document-to-profile mapping.

Decision 2

Chose: 150+ skill nodes, 300+ aliases — weighted graph with co-occurrence scoring

Over: Vector embeddings, BM25 full-text search

Because: Skill matching requires deterministic, auditable scoring. Embedding distance is opaque when a hiring manager asks "why did this job score 83?". The taxonomy explicitly handles "Go" → "Golang" → "Go language" synonym chains and seniority deltas.

Decision 3

Chose: PostgreSQL JSONB for flexible job metadata

Over: Separate tables per attribute type

Because: Job posting schemas vary wildly by source — Greenhouse adds greenhouse_id, Adzuna adds category, HackerNews adds no standard fields. JSONB absorbs the variance without migrations.

Decision 4

Chose: 5+ job sources — Adzuna, RemoteOK, HackerNews, Greenhouse, Lever

Over: LinkedIn API or single premium provider

Because: No single source has enough coverage for long-tail specialist roles (platform engineering, ML infrastructure, Go-primary). Diversity of sources is itself a feature.

Decision 5

Chose: asyncio.to_thread wrapping synchronous Bedrock SDK calls; asyncio.gather for parallel extraction and assessment

Over: SQS-based worker decoupling

Because: Full SQS decoupling (S3 staging + queue + worker + SSE push) is a multi-session effort. asyncio.to_thread fixes the event-loop blocking issue and passes all tests while Phase 5 designs the proper async architecture. The debt is documented and deferred.

Decision 6

Chose: AWS Cognito User Pools (Essentials tier)

Over: Custom JWT with PyJWT + bcrypt, Auth0, Keycloak

Because: Free below 10,000 MAU — Auth0 starts billing at 7,500 MAU. JWKS rotation, hosted email verification, SRP password exchange, Forgot Password, and MFA are all included at zero incremental cost. FastAPI validates Cognito-issued tokens locally — zero per-request Cognito API calls in the hot path.

Decision 7

Chose: AWS Bedrock (claude-sonnet-4-6 via cross-region inference profile)

Over: Anthropic API with a static API key

Because: IAM credential chain from ECS task role — no static key in the codebase or Secrets Manager. CRIS pools quota across three regions for burst resilience at zero per-token price premium. Surprise: granting bedrock:InvokeModel on the inference profile ARN alone is insufficient — IAM must also explicitly list each regional foundation model ARN because Bedrock fires an IAM check at each regional hop. Not in the Getting Started docs.

Decision 8

Chose: EventBridge (RotationSucceeded event) → Lambda → ECS force-redeploy

Over: RDS Proxy ($36/month), longer credential TTL, manual monitoring

Because: RDS rotates the master password every 7 days. ECS tasks read credentials at container startup and cache them for the task lifetime. When rotation fires mid-run, every DB call returns a password authentication failure while /health still returns 200 — no standard alarm fires. This caused an 18-hour outage. The EventBridge pattern detects the rotation event and force-redeploys automatically.

The pipeline succeeded. The data was silently discarded.

During early testing, the extraction pipeline showed success in CloudWatch — profile extracted, assessment complete, HTTP 200. When a real test user ran the full flow, the response came back with extraction_runs: 0. The pipeline had run. The DB write was located after the career assessment LLM call. The assessment had a 60-second botocore default timeout. It ran for 106 seconds. Timeout fired. The in-memory extraction data — already complete — was thrown away. Fix: raised the timeout to 300s, moved the DB write to after extraction rather than after assessment, added asyncio.to_thread so the LLM call no longer freezes the FastAPI event loop during status polling.

Scoring weights were meaningless until 222 real jobs were in the corpus.

Scoring logic was written with reasonable weights. But until 222 jobs were fetched and scored against a real user profile, there was no way to know whether those weights produced useful signal or noise. The first real run produced 115 jobs above the threshold — a workable number. The current calibration (skills 40%, role 25%, company 20%, growth 15%) was validated against a senior platform engineer profile with Go + Kubernetes + ML infrastructure as primary signals.

Auth guard tests passed. Twelve production routes were unprotected.

Several page routes were returning 200 for unauthenticated requests. The auth dependency was correct but not applied uniformly. Unit tests for the auth layer all passed. The failure was structural: routes missing the current_user dependency injection in the handler signature were not protected at all, but no test asserted the route boundary. Fix: audited all 12 authenticated routes to verify the dependency was wired at the handler level. Added auth guard assertion to the functional test plan.

Lever jobs were fetching correctly and deduplicating to zero.

Lever was one of the 5 job sources. Results were being fetched but all records were discarded at deduplication. Lever job URLs are https://jobs.lever.co/{company}/{uuid} — the UUID is stable per posting, but the company slug sometimes varies across regions or roles. The deduplication logic matched on the full URL including the company slug, causing valid new postings to hash-collide with existing records under variant slugs. Fix: normalised Lever URLs to UUID-only for deduplication keying.

42% of jobs scored above 85. The score was not discriminating.

After the first end-to-end scoring run, the match summary showed 42% of jobs at 85 or above. An 85/100 score is supposed to mean strong match. When nearly half the corpus scores at that level, the score is not useful. Root cause: skill overlap was being measured against the user's full skill list, not against the skills the job actually requires. A platform engineer with 41 skills scores highly against almost any tech job. Fix: added inverse skill weight — skills shared by more than 70% of postings in the corpus are downweighted. Go, Kubernetes, and Terraform became higher-weight signals because they appeared in fewer postings as requirements.

11 production deploys failed before the first successful Cognito activation.

Migration 018_drop_local_auth_columns was written with a pre-check-and-raise pattern: if any row in users.password_hash was non-null, raise an exception and abort. The dev database had legacy password_hash rows from accounts created during Cognito parallel testing. The migration blocked. The deploy failed. ECS crashed. The health check failed. The circuit breaker kicked in. This happened across multiple iterations because the pattern was wrong for the intent — decommissioning an auth subsystem is not a schema bug. The correct pattern: log the count of rows being cleared, null them out, drop the column. After the fix, the deploy succeeded.

AWS denied SES production access. Built a cross-account email relay instead.

After verifying the SES domain identity, DKIM, SPF, and DMARC — all verified successfully — a production access request was submitted to AWS Support. Denied with no technical explanation. First workaround: cross-account delegate sender — grant Candor's task role permission to send via Crucible's SES identity. This failed. AWS evaluates sandbox restrictions against the calling account, not the identity-owner account. Final solution: Candor's ECS task signs a request with IAM SigV4 and posts it to an API Gateway endpoint in Crucible's account. Crucible's Lambda validates the request and calls SES. Crucible became core infrastructure.

RDS rotated its password. Production was broken for 18 hours.

On 2026-05-26 at 20:20 PT, RDS rotated the master password on its 7-day schedule. The running ECS task had the old password cached from container startup. Every authenticated request returned 500. The /health endpoint returned 200 — it has no DB dependency. The ALB alarm stayed OK. No automated alert fired. The asymmetry that cracked it: correct-password login returned 500, not 401. Wrong password returned 401. That pointed at the database layer, not auth. Fix: force-new-deployment restored service in 90 seconds. Structural fix: EventBridge rule on RotationSucceeded → Lambda → ECS force-redeploy, running automatically every 7 days.

The resume email was being used as the user's identity.

The original endpoint called get_or_create_user_by_email() with the email extracted from the resume file. This created an orphaned user row with no cognito_sub and no link to the authenticated account holder. After two CloudWatch-confirmed successful pipeline runs, the data came back empty — written to an orphaned row with no connection to the Cognito account. The flaw: resume email and account email are different things. A professional contact email on a resume may differ from a signup email. Fix: users.id is the immutable account owner, tied to cognito_sub. Resume email is metadata. get_or_create_user_by_email was prohibited from the codebase.