Learning Management System Review Summary Report and Recommendation

Northern Illinois University (NIU) conducted a comprehensive Learning Management System (LMS) review during the 2020-2021 academic year. While faculty usage and overall satisfaction with the university’s current LMS (Blackboard) remains high, the university hadn’t conducted a formal evaluation of other LMS systems in a number of years.

The review was a thorough, comprehensive effort to evaluate the primary learning management systems in the higher education market. This evaluation was broad in scope and sought input from faculty, staff and students on not only their current satisfaction with NIU's current LMS but also what features are desired in a LMS moving forward. Through NIU’s partnership with Internet2, the top three leading LMS platforms in Higher Education – Blackboard Learn, D2L Brightspace, and Instructure Canvas – were reviewed to ensure that we select a platform that will serve the core interests of the university for years to come. The evaluation supplied the data to prepare a recommendation on whether an alternative LMS should be considered at this time.

Overview and History of Blackboard at NIU

Blackboard Learn was originally licensed at NIU in 1999 to facilitate the delivery of web-based courses, but it has been used much more broadly as a critical system for facilitating teaching and learning.

Over time, other components have been added (initially individually priced but now bundled in our institutional license):

• Blackboard Community System in 2003 – facilitates non-academic use of Blackboard
• Blackboard Content System in 2007 – file repository for storage and sharing, within and outside of courses
• Blackboard Collaborate in 2011 – two-way audio/video web conferencing (upgraded to Collaborate Ultra in 2016)
• Blackboard Mobile Learn in 2011 – mobile app for student and faculty access to Blackboard on smartphones and tablets

NIU hosted Blackboard locally from 1999 to 2018, when it was migrated to the cloud to reduce costs, increase performance, and provide a path to the Ultra Experience, Blackboard’s modern and completely redesigned interface. The Ultra Base Navigation and Course View were introduced in Spring 2019. Blackboard was relied upon heavily in the pivot to emergency remote instruction in response to the pandemic.

Usage of Blackboard

Academic Courses

Blackboard is used by 95% of faculty and instructors, 99% of students, and for 73% of all course sections (fall 2020). A majority of NIU courses each semester make extensive use of the capabilities of the system. Blackboard integrates with other cutting-edge tools for teaching and learning including video applications, publisher platforms, exam proctoring, and student engagement systems.

Figure 1: NIU Students & Faculty with at Least 1 Course in Blackboard, Fall Semester, 2002-2020

Figure 2: Use of Blackboard by Individual Students, 2002-2020

Figure 3: Use of Blackboard by Individual Faculty/Staff/TAs to Teach, 2002-2020

Figure 4: Course Sections on Blackboard, 2002-2020

Figure 5: Percentage of Course Sections Taught Using Blackboard (since 2010)

Figure 6: Top 10 Tools Used in Blackboard Courses

Additional Blackboard use stats are available.

Other Uses

In addition to scheduled academic courses, Blackboard is used in a variety of ways to support the mission of the university:

• Departmental communication to students by major/minor on events and program deadlines
• Development and storage of course materials for upcoming and previous semesters
• Student portfolios for program assessment and supporting job searches
• Department and programmatic assessment across courses, including general education assessment
• Storage and delivery of electronic reserves using a process custom-developed by the University Libraries and Division of Information Technology
• Annual compliance training for students, faculty, and staff, including state-mandated ethics and Title IX training
• Delivery and tracking of training completion for employees including Card Services, Division of Information Technology, Recreation, and Holmes Student Center

Satisfaction with Blackboard

NIU faculty and instructors were surveyed in 2016 and 2019 to gauge satisfaction and comfort with Blackboard. In 2016, 83% of faculty strongly agreed or agreed that they were satisfied with Blackboard and 82% responded that they were confident using Blackboard (these questions were not repeated in 2019). From 2016 to 2019, other measures of confidence and satisfaction were relatively consistent or increased, including finding that Blackboard enhanced teaching effectiveness and helped students to be more successful.

For more details on reported faculty use and satisfaction with Blackboard, see complete reports of past Blackboard usage surveys.

Review Task Force

In October 2020 Executive Vice President and Provost Ingram authorized the LMS review and for it to be jointly coordinated by the Center for Innovative Teaching and Learning (CITL) and the Division of Information Technology (DOIT). Given the widespread and longstanding use of the LMS, a task force with broad and inclusive representation was formed with members nominated by colleges and stakeholder units from across the university, consisting of the following:

Executive Committee

Responsible for execution of the review, including the solicitation of feedback, development of a communications plan, and presentation of a final report. The executive committee met weekly and consisted of the project co-chairs, project manager, application services senior director, one member from the technical committee, two members from the advisory committee, and procurement representative.

• Project Co-chair – Matthew Parks, Associate Vice President and Chief Information Officer, DOIT
• Project Co-chair – Jason Rhode, Executive Director, CITL
• Project Manager – Susan Smith, Projects Coordinator, Institutional Effectiveness
• Application Services Representative – Nick Choban, Senior Director, Application Services, DOIT
• Technical Committee Representative – Daniel Ihm, Director, Application Development and Support, DOIT
• Co-chair of Advisory Committee – Therese Arado, Acting Director of the Law Library and Professor
• Co-chair of Advisory Committee – Stephanie Richter, Director of Teaching Excellence and Support, CITL
• Procurement Representative – Antoinette Bridges, Director, Procurement Services and Contract Management

Advisory Committee

Representatives from across university stakeholder groups who provided input into the evaluation process and establishment of evaluation review criteria. The advisory committee met on an as-needed basis and was responsible for seeking and incorporating campus feedback on the process, responding to draft reports, and supporting the communications plan. The technical committee served as a subgroup of the advisory committee.

• College of Business Faculty Representative – Gina Nicolosi, Professor and Chair, Department of Finance
• College of Education Faculty Representative – Wei-Chen Hung, Professor and Chair, Department of Educational Technology, Research and Assessment
• College of Engineering and Engineering Technology Faculty Representative – Gary Chen, Professor, Department of Industrial and Systems Engineering
• College of Health and Human Sciences Faculty Representative – Hamid Bateni, Associate Professor, School of Allied Health and Communicative Disorders
• College of Liberal Arts and Sciences Faculty Representative – Artemus Ward, Professor, Department of Political Science
• College of Law Faculty Representative – +Therese Arado, Acting Director of the Law Library and Professor
• College of Visual and Performing Arts Faculty Representative – Kelly Gross, Assistant Professor, School of Art and Design
• University Libraries Faculty Representative – Alissa Droog, Assistant Professor, Education and Social Sciences Librarian
• Instructor Representative – Federico Bassetti, Instructor, Department of Operations Management and Information Systems
• Instructor Representative – Joyce Laben, Instructor, Department of Leadership, Educational Psychology and Foundations
• Instructor Representative – Candace Medina, Instructor, Department of World Languages and Cultures
• Student Success Representative – LaBrian Carrington, Academic Advisor/Counselor, Academic Advising Center
• Student Affairs Representative – Amanda Newman, Associate Director, Disability Resource Center
• Staff Representative – Kelsey Johnson, Senior Director, Operations and Financial Strategy, College of Business
• Staff Representative – Deirdre Young, Acting Associate Director, Registration and Records
• Student Representative – Antonio Johnson, President, Student Government Association
• Student Representative – Marissa Dordick
• Student Representative – Gaylen Rivers
• Graduate Student Teaching Assistant – Bailey Zo Kreager, Department of Geology and Environmental Geosciences
• Communications Representative – Holly Nicholson, Web Team Manager, Division of Enrollment Management, Marketing and Communications
• Division of Outreach, Engagement and Regional Development Representative – Amy Jo Clemens, Director, Center for P-20 Engagement

+ = Advisory Committee Co-chair

Technical Committee

Responsible for operationalizing the review of systems, engagement with key stakeholders, and collection of evaluation data.

• DOIT Representative – Daniel Ihm, Director, Application Development and Support
• DOIT Representative – Shawn Staggs, Applications Programmer Analyst
• CITL Representative – +Stephanie Richter, Director of Teaching Excellence and Support
• CITL Representative – Jason Underwood, Director of Instructional Design and Development
• CITL Representative – Tracy Miller, Assistant Director

Additional Internal Engagement

Beyond the aforementioned individuals who served collectively as the LMS review task force, the following internal stakeholder groups were also engaged throughout the review process and provided with status updates along with opportunities for feedback.

• Office of the Executive Vice President and Provost
• Council of Deans
• Associate Deans
• Department Chairs/Directors
• Faculty Senate
• Academic Planning Council (APC)
• Innovative Teaching and Learning Advisory Committee (ITLAC)
• Online Learning Advisory Council (OLAC)
• IT Planning Council (ITPC)
• Computing Facilities Advisory Committee (CFAC)
• Student Government Association (SGA)

Review Criteria and Scoring

Review Scoring

The following scoring system was established, with scores weighted according to the following maximum scores:

Technical Review Criteria

The LMS Review Task Force identified and established technical review criteria, organized into the following categories:

1. General Usability
2. Accessibility
3. Content Management
   1. Content Creation
   2. Content Reuse
4. Communication
   1. Profiles
   2. Email/Messaging
   3. Announcements
   4. Discussion Board
   5. Groups
   6. Notifications and Calendar
5. Assessment
   1. Assignments
   2. Tests
   3. Portfolios
   4. Gradebook
   5. Other Assessment

6. Analytics and Reporting
7. Technical Requirements
   1. Hosting
   2. Security
   3. Disaster Recovery
   4. Customization
   5. Integration
   6. Data Access and Reporting
   7. Technical Support

Criteria for support of non-credit activities and non-matriculated students were optional but could increase technical review score.

Additional Criteria and Scoring

The following additional criteria and scoring elements were similarly developed for use during the review and shared:

• Technical Review Weighted Scoring
• Vendor Technical Questionnaire
• Roadmap Scoring
• Vendor Support for Migration Scoring

Review Process and Timeline

The review process was comprehensive and transparent, with detailed criteria and metrics for evaluation and scoring. The executive committee met with each vendor to provide an overview of NIU’s review process and topics for inclusion in campus demonstration presentations. Subsequent steps in the review included:

1. Each vendor met with the executive committee to provide background into their history as a company, general overview of solution, and roadmap ahead -- information that provided additional context in advance of campus demonstrations.
2. Each vendor provided a test instance of their platform for technical review and access to existing documentation; technical committee then reviewed the systems and rated each according to the full list of technical criteria.
3. Executive committee and technical committee requested written responses to additional questions based on the technical review.
4. Each vendor provided two virtual demonstrations for faculty, staff, and students to attend on functionality for teaching and learning. Demonstrations were recorded and made available for NIU users to view on the LMS review website. Participants completed follow-up surveys to rate each system’s functionality.
5. Each vendor provided separate virtual demonstrations of shopping cart features for outside learners and external programming. Participants similarly completed follow-up surveys to rate the system’s functionality.
6. Technical committee built sample courses in each system with identical content and conducted focus group sessions with advisory committee and other NIU constituent groups. Focus groups compared the courses from each system and rated the systems’ functionality.
7. Each vendor provided references of client institutions comparable to NIU for the executive committee to reach out to for their institutional experience and feedback.

Recommendation

Frequently Asked Questions