Reporting platform
to help schools understand assessment results

Problem

Schoology is a virtual learning environment for K-12 schools and higher education institutions. It provides a platform for creating, managing, and sharing academic content. In 2017, Schoology invested in a new product that allows users to collaboratively create and manage question-based assessments and distribute them to multiple course sections across a district.

However, quickly after launch, the team realized that understanding the results of the assessments and the growth trajectory of the students is as crucial as having a great experience building assessments. To address this need, we created an assessment reporting platform. This case study explains the research and go-to-market strategy for the reporting suite.

Research & Discovery

Initial research goals:

  • What are some of the questions they are trying to answer when assessing students?

  • What does 'student growth' mean for district administrators?

  • How are administrators currently measuring growth?

  • What actions do school districts take after analyzing assessment results?

Remote and in-person discovery interviews (12+ interviews).

Interviews with school districts

Concept exploration and iterations

Hypotheses

Explorations

  • Providing both high-level summaries and granular drill-down capabilities (e.g., click-through assessments to item level) will increase educator satisfaction and usage of the reporting suite.

  • Supplementing basic measures like average scores with visualizations and interactive elements will improve data accessibility and user engagement.

  • Offering comparison capabilities across assessments and timeframes will empower educators to track growth and program effectiveness, improving student outcomes.

  • Pre-built "out of the box" reports on learning objectives, student progress, and demographics will provide valuable insights and save educators time compared to custom analysis.

  • Allowing users to personalize and interact with pre-built reports (e.g., filter by student group and adjust visualizations) will increase report usage and relevance to specific needs.

Validation & testing

  • Remote moderated usability testing (5 rounds, 7+ sessions).

  • Unmoderated concept testing and open questions via Optimal Workshop (2 rounds of feedback, 150+ responses).

Remote moderated usability testing

Strategic Plan

Our initial strategy prioritized establishing a robust foundation for our reporting suite. This introductory product served as a starting point, demonstrating the vision and core functionality of our product. Our priority was to acquire beta testers and early customers to secure additional funding and sponsorship for the product.

  • Phase 1: Building the foundation and showcasing value. We launched a dashboard that could be accessed by all districts, regardless of their size, which provided information about one assessment at a time. The dashboard included question-level reporting, breakdowns by demographics, results by schools and grades, and other important metrics.

  • Phase 2: Deeper analysis and comparisons. This phase allowed districts to compare the average scores of different assessments across various timeframes. This functionality helped reveal growth trends, identified areas for improvement and allowed benchmarking against past performance.

  • Phase 3: Expanding insights with pre-built reports.

    • Learning objectives report: Analyzing district-wide performance against established learning objectives, helping educators identify areas where instruction needs adjustment.

    • Student progress reports: Tracking individual student progress over time, allowing educators to monitor performance and personalize learning paths.

    • Demographics Reports: Analyzing student demographics within assessments, enabling the identification of potential achievement gaps and informing equity-focused initiatives.

As expected, each phase resulted in many new design system components, including new tables, filters, progress indicators, and data selectors, among others.

Benefits of a phased approach:

  • Smooth User Adoption: Users gain familiarity with core functionalities before progressing to advanced features.

  • Clear Value Demonstration: Each phase offers tangible benefits, ensuring user engagement and product investment.

  • Staged Complexity: Users acquire skills progressively, maximizing platform utilization and insights gained.

Product Launch

Release #1: Dashboard for a single assessment

Dashboard per assessment

Release #2: Multiple assessments Comparison report

Comparison creation flow

Live product walkthrough

Release #3: Pre-build reports

Example of a pre-build report we launched: Distribution report

Outcomes

  • Increased Investment: The success of the assessment management platform attracted double the investment for the following year, demonstrating its value proposition and market demand.

  • Competitiveness: The focus on out-of-the-box reporting and expanded assessment types made Schoology more competitive against other platforms by providing educators with targeted insights and ease of use.

  • Acquisition and Retention: Schoology was acquired by PowerSchool in 2019. Despite PowerSchool having dedicated assessment tools, they recognized Schoology's strength in managed assessment reporting and retained it for its user-friendliness and data robustness.