Massive open online courses (MOOCs) have captured the attention of higher education and the political community in a way seldom seen. To many, the idea of accessing instruction at world class institutions for free is the answer to a prayer. As a new form of instructional delivery, MOOCs could take some of the heat off of higher education, with its ever rising tuition, and the politicians as well, whose cuts to state support have made necessary those increases. Hopefully, MOOCs will offer ways for state governors, legislators and college administrators to promote universal access to higher education without having to worry about cost.
On the other hand, many within higher education question MOOCs’ effectiveness as vehicles for real learning. They view any rush to recommend credit for use in satisfying degree requirements as premature. Partly as a result of this growing concern, only five MOOCs have been deemed credit-worthy, so far.
Even with a MOOC that has been recommended for credit, questions remain as to the extent participants actually learned the material. While few question the capabilities of the sponsoring institutions or their faculty, the degree to which reputation translates into learning is not clear, and to date, very little attention has been given to the measurement of learning outcomes by MOOC providers. In addition, astounding attrition rates have called into question whether MOOCs are more akin to entertainment, or a 21st century text book offering than a serious learning experience. According to outcomes from one recent course offering by Coursera, the MOOC initially enrolled more than one hundred thousand students, had but just seven (sic) take the final exam. With so few reaching the finish line, one must question the quality of the journey.
Historically, a recommendation of credit for learning outside of an accredited institution, and the oversight of its faculty, has been held to higher standards than we see with MOOCs. Whether it be The College Board and its College Level Examination Program (CLEP), Educational Testing Service and its DSST portfolio, or UExcel exams of Excelsior’s Center for Educational Measurement (CEM) , college-level credit comes from passage of a psychometrically valid assessment, administered in a secure facility, operated by a third party. Those colleges and universities that accept such credit recommendations have come to expect these rigorous measures.
The three national providers have created a sophisticated assessment development process where, for example, a typical subject assessment (e.g., college algebra) is created with input from subject matter experts at multiple institutions. In conjunction with doctoral level psychometricians, exams are prepared for field testing to assure statistical reliability, predictability, and overall validity. The quality and integrity of these exams are superior to those drawn from a text book publishers "test bank," or an instructor’s notes.
In addition to the validity of an assessment, student identity and security of the instrument itself are critical during administration. While some MOOC providers do verify the identity of those taking their course and proctor their end-of-course examinations, more attention must be paid to the security of the assessment instruments.
With the potential for thousands of exam takers from around the world, question banks must be more comprehensive than those of a homegrown instrument, or for a smaller population of potential test takers. ETS and Pearson discovered this key consideration some years ago when the Test of English as a Foreign Language (TOEFL) was compromised by a well-organized group of students with near photographic memories. If stringent measures are not put in place to ensure assessments are comprehensive, fair and randomized, students can navigate exams without truly learning the material.
To be clear, for an assessment’s results to be worthy of transcription and acceptance toward degree requirements, it should be:
This contrasts with what is seen from the major MOOC providers. A final examination, or assessment of learning, if provided, is typically prepared by the offering instructor. (As one edX executive stated, "The fact that the exam comes from an MIT faculty member is the only validity we need.") This instructor is usually untrained in the preparation of statistically valid assessments and, in the eyes of some, has an inherent conflict of interest in evaluating his or her own work. Administration of those exams now offered is understood to involve remote proctoring, relying on an internet camera and/or various forms of identity assuring software. This may not be sufficient to provide the needed assurance of accurate student identification and assessment security.
At a time when the federal government, regional accreditors and much of the academic world has shifted from the measurement of inputs (i.e., number of PhD faculty, ratio of students to instructors, books in the library, etc.) to the assessment of outputs, MOOC providers and their sponsors must increase their attention to the learning side of the ledger. The higher ed community no longer can nor should assume that a sufficient level of learning takes place merely because the instruction comes from a brand name institution, with a credentialed instructor.
Today, there is an expectation that we measure actual learning. This is, admittedly, new, both for the instructional provider and for those who are asked to determine possible equivalency – the American Council on Education (ACE) and the National College Credit Recommendation Service (NCCRS). As they navigate these changes, both organizations will need to place greater emphasis on learning outcomes and their measurement if their evaluations are to meet the expectations of accreditors and regulators.
The good news is that this will not be a particularly difficult problem to resolve, assuming that learning is occurring and that we just need to do a better job of measurement. While most MOOCs have so far featured topics for which there isn’t an appropriate exam, the process to create one is not substantially longer or more complex than what is required to create and offer a course in MOOC format. In return for paying more attention to learning outcomes, and how they are measured, MOOC providers will not only enhance the validity of this pioneering effort, but also gain credibility in the eyes of critics. The ultimate goal is to evolve MOOCs to the point where acceptable learning can be validated, and credit awarded.
We aren’t quite there, yet.