Skip to Main Content
Back to All Work

Course Quality

SCOUT Omega

Course QA Pipeline

I built an automated first pass that prepares each course for human QA review.

EdPlus at ASU

Measured

44 criteria

15 settled by rules alone, 28 rules first with AI only for what rules cannot settle, 1 AI-only (grammar). Yes/No verdict rules cover 43; the last goes to the reviewer.

Measured

100 courses, 50 min

Runtime of the automated first pass on a full production cohort, before any human review.

Measured

56.4 min

Mean assisted human review per course (median 49.2) across 42 reviews timed in Airtable. An upper bound.

The Friction

One instructional designer's full manual QA audit, which I observed, took 6 to 8 hours for a single course. ID assistants run a narrower review at about 1.8 hours per course, so 100 courses still take about 180 reviewer hours. Reviewers were spending their expertise on checks a machine could make.

What I Did First

I shadowed a QA initiate end to end before writing any code, then interviewed the reviewer about the full process. The goal was to find which checks were mechanical and which needed judgment, because only the first kind should be automated.

What I Built

A prototype that completes a QA initiate form from an .imscc package in 3 to 5 minutes, using Python and Google Gemini. It grew into a batch pipeline that grades courses against WCAG 2.1 AA and ASU design standards, with adaptive rate limiting that keeps requests inside the Canvas API quota.

What Came of It

The internal demo led to my leading the technical side of the QA workstream under the AI taskforce. Reviewers now start from a prefilled report and spend their time on judgment calls.

Scope and Limits

Still a 100-course pilot. The manual review step is estimated at about 1.8 hours per course from ID assistants' own reported pace, so the comparison with 56.4 minutes is not a controlled one. The pipeline surfaces issues and leaves the fixing to a person.

More systems, and the learning experiences behind them.

Back to All WorkEmail Me

or write directly to brent.michael670@gmail.com