Skip to content

feat: self-service data upload for course and PDP/AR files (#86)#92

Open
William-Hill wants to merge 14 commits intomainfrom
feat/issue-86-self-service-upload
Open

feat: self-service data upload for course and PDP/AR files (#86)#92
William-Hill wants to merge 14 commits intomainfrom
feat/issue-86-self-service-upload

Conversation

@William-Hill
Copy link
Collaborator

Closes #86

Summary

  • /admin/upload page (admin/ir only) with drag-drop file selector, 10-row preview, and commit
  • Course enrollment CSVs stream-parsed and batch-upserted (truncate + reload) into course_enrollments Postgres table via pg
  • PDP cohort CSVs and AR .xlsx files uploaded to Supabase Storage pdp-uploads bucket, then GitHub Actions ml-pipeline.yml triggered via repository_dispatch
  • "Upload Data" nav link added — visible to admin/ir roles only

New files

  • app/admin/upload/page.tsx — upload UI
  • app/api/admin/upload/preview/route.ts — parse first 50 rows, validate columns
  • app/api/admin/upload/commit/route.ts — course enrollment upsert + PDP/AR Storage + Actions dispatch
  • .github/workflows/ml-pipeline.yml — downloads from Supabase Storage, runs ML pipeline, uploads artifact

New env vars required (see env.example)

  • SUPABASE_SERVICE_ROLE_KEY — for Storage writes
  • GITHUB_PAT — for repository_dispatch (scope: Actions write)
  • GITHUB_REPO — full repo path e.g. devcolor/codebenders-datathon

GitHub Actions secrets required (before workflow can run)

SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY, DB_HOST, DB_USER, DB_PASSWORD, DB_PORT, DB_NAME, DB_SSL

Supabase setup required

Create a private Storage bucket named pdp-uploads in the Supabase dashboard

Test plan

  • Admin/IR user can reach /admin/upload; other roles are redirected to /
  • "Upload Data" nav link visible to admin/IR only
  • Course enrollment CSV: preview shows first 10 rows with correct column names
  • Course enrollment CSV: commit truncates and reloads course_enrollments table, returns { inserted, skipped, errors }
  • Switching file type clears the selected file and the hidden input (no stale re-selection bug)
  • Missing required columns shows warning; Confirm & Upload button is disabled
  • PDP cohort CSV: commit uploads to Supabase Storage and returns { status: "processing", actionsUrl }
  • GitHub Actions link in result phase opens the correct repo Actions page
  • npx tsc --noEmit passes with 0 errors

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: self-service data upload for PDP, AR files, student and course data

1 participant