GovEd e-Learning System

Purpose

Academic

HCI 520 Learning Centered Design Project

Project

Design a GovEd e-Learning System and conduct an experimental comparison to measure its instructional effectiveness

Overview

Research Question

Does the GovEd e-learning pretest, pretraining, and posttest cause learning?

Scope

City of Chicago government and the roles and responsibilities that support local government

Context of Use

Self-paced e-learning

e-Learning System Elements

• Pretest
• Pretraining
• Posttest

Target Users

• Adults who live, work, or attend school in the City of Chicago and want to understand how local government functions
• High school students who live or attend school in the City of Chicago and want to build their civic literacy

Framework

Intended learning outcomes based on the Revised Bloom's Taxonomy Table (RBT) and Knowledge dimension

• Learning Objective 1 (LO1): Remember (1.2 Recall) knowledge of civic education concepts and terminology (Aa Knowledge of terminology)
• Learning Objective 2 (LO2): Remember (1.2 Recall) knowledge of government offices and responsibilities (Ba Knowledge of classifications and categories)
• Learning Objective 3 (LO3): Understand (2.6 Compare) different government offices and responsibilities (Ba Knowledge of classifications and categories)
• Learning Objective 4 (LO4): Analyze (4.1 Differentiate) criteria to identify appropriate government offices and obtain information or complete a process (Cc Knowledge of criteria for determining when to use appropriate procedures)
• Learning Objective 5 (LO5): Apply (3.1 Execute) knowledge of government offices and responsibilities (Db Knowledge about cognitive tasks, including appropriate contextual and conditional knowledge)

Instructional Design Principles

Design principles based on e-Learning and the Science of Instruction by Ruth Colvin Clark and Richard E. Mayer

• Multimedia Principle
• Contiguity Principle
• Pretraining Principle
• Segmenting Principle
• Personalization Principle
• Learner Control Principle
• Worked Examples Principle

Challenges

Limited participant pool

• Participant recruitment was limited to individuals who live, work, or attend school in Chicago or the Chicago suburbs, which does not fully represent the target user population.

Limited participant documentation

• The research documentation does not include a complete list of the 16 participants who received the Qualtrics study link.
• The research documentation records nine of the 16 participants.
• The research documentation does not clearly record the seven participants who completed the Qualtrics study.

Limited access to participant demographics

• The Qualtrics study link included a name question only.
• The research documentation identifies the age range and education of nine participants, but not their residency (See the Data Collection section).

Limited learning objective-level data

• The research documentation records overall correct answers and scores per participant, but does not indicate which specific questions were answered correctly. As a result, only aggregate improvement can be analyzed.

Project Value: UX Perspective

The GovEd e-learning process shows a statistically significant and practically meaningful improvement in learner performance

• Participants showed a clear, substantial improvement in scores from pretest to posttest, a result unlikely to be due to chance and large enough to matter in practice.

Task flows support successful task completion

• Seven participants successfully completed the pretest, pretraining, and posttest.

Goals span multiple levels of RBT

• The Qualtrics study didn't just test memorization, it tested whether learners could reason with and act on the material.

Project Value: PM Perspective

The GovEd e-Learning System provides a defensible business case

• The Data Analysis section demonstrates statistically significant, quantifiable evidence of training effectiveness. This initial value justifies continued investment and iterations.

Research establishes a baseline for iteration

• The before and after benchmarks mean future iterations can be measured against a consistent standard, turning a one-time validation into a repeatable evaluation process.

Research reduces the risk in decision-making

• Statistically grounded evidence provides a clear, actionable signal for go/no-go decisions.

Process

Instructional Design

Designed a pretest, pretraining session, and posttest sequence to measure knowledge gain. Used Qualtrics to conduct the experimental comparison and developed a website for the pretraining session.

Participants received a Qualtrics study link and entered the following task flows:

• Clicked on the Qualtrics study link
• Completed the 11 pretest questions
• Clicked on the pretraining session link to navigate to the pretraining website
• Walked through the three lessons
• Clicked on the posttest link
• Completed the 11 posttest questions

Data Collection

All responses were recorded in Qualtrics. None of the participants received pretest or posttest results, but three participants requested verbal feedback to compare their before and after results. Any information collected was used only for the purposes of this study.

Data Analysis

Seven participants (n = 7) completed the pretest, pretraining session, and postest. The pretest and posttest consisted of the same 11 questions. The data table within this section (See also data analysis spreadsheet) reflects the correct answers (raw number of items) and scores (number as a percentage).

A paired-samples t-test and Cohen's d_z were used to assess statistical and practical significance, since these methods account for the correlation between each participant's pretest and posttest scores. The results were evaluated against conventional thresholds: p < 0.05 for statistical significance and d > .8 for a large practical effect.

Results

The seven participants completed the task flows for the pretest, pretraining session, and posttest. Their average scores improved from 60% to 82%. This shows that the pretraining session significantly improved learner performance (p = 0.007), with a large practical effect (Cohen's d_z = 1.50). Note: One participant's pretest and posttest scores did not change and remained at 36%.

Findings

The GovEd e-learning pretest, pretraining, and posttest indicate improved learner performance. Participants showed measurable gains in overall scores, reflecting progress toward learning outcomes from remember/recall to apply/execute, though item-level data is needed to confirm which specific Learning Objectives were achieved.

Opportunities

Future iterations can address the four challenges listed above and expand on measuring design effectiveness:

• Recruit a larger set of individuals who reflect the target users
• Document all participants
• Capture and anonymize demongraphics for research integrity
• Measure learning objective-level data
• Measure the effectiveness of each design principle with methods like A/B testing

Together, these adjustments can better answer the research question of whether the GovEd e-Learning System causes learning.

AI disclaimer: This case study uses Claude.ai to proofread copy, validate the data analysis, and collaborate to break down the calculation processes.