Evaluating the Open Educational Resource

It is important to recognize that course materials are evaluated and adopted by the instructors themselves, who care first and foremost about the quality of the instruction they offer.

Posted on November 1, 2018 in Publisher Editorial

Our recent foray into the evaluation of open educational resources (OER) has been something of an eye-opener to those of us accustomed to what now seems the almost Newtonian (by comparison) world of traditional academic publishing. Even granting that Choice reviews are brief—no more than semaphores, really, compared to those of The New York Review of Books—up to this point our basic criteria for the evaluation of a work have been those of every reputable review organ, indeed, of scholarly peer review generally.

Provenance, accuracy, lack of hidden bias, cultural relevance, internal consistency, comprehensiveness, acknowledgement of sources…these and other criteria are the cornerstones of scholarly review. Certainly open educational resources can and should be evaluated according to these same criteria. After all, the qualifier open signifies nothing more than the existence of certain rights pertaining to the ownership and use of the work. It has no bearing whatsoever on the quality of its content. Yet if we are to take the most oft-employed definition of an OER as a standard, David Wiley’s now ubiquitous 5Rs, the responsibilities of the reviewer go well beyond those items mentioned above, if for no other reason than that the reviewer, in addition to commenting on the quality of the work, has the added task of evaluating to what extent the work is actually “open,” how well it satisfies the requirements of openness.

In its fullest sense, the ability to retain, reuse, revise, remix, and redistribute content imposes rigorous criteria against which to evaluate an OER, but these criteria, while necessary, are not by themselves sufficient. Pedagogical concerns and, of course, content quality are no less important. And so in thinking about how to review these works for Open Choice (OC), our forthcoming review service for OER, we realized that neither the freestyle format of Choice reviews nor the relatively more structured format of ccAdvisor were adequate to the complexity of the task. Ultimately we decided that only a highly structured review format based on multiple, overlapping criteria would be sufficient to ensure uniform and comprehensive evaluations of the works in question. The OC review template evaluates OER in nine areas: provenance, licensing, accessibility, adaptability, content quality, pedagogy, interface design, ancillary materials, and competing works. In conjunction with these, reviewers are asked to comment on and score a variety of factors, including the licensing profile under which it is published, conformity with WACG accessibility requirements, ease of alteration and export, screen design and navigation, and dozens of other topics.

But our chief concern is with the quality of the work, which remains, in our view, the single greatest determinant of a work’s suitability. Of what use are adaptability or the existence of a CC-BY license if the quality of the source material is deficient? In light of this, we ask our reviewers to rate and discuss content under ten separate criteria, to comment on the qualifications of the author(s), and, as OER are created under a variety of conditions (not all of them providing the quality-control mechanisms that are part of the commercial publication process), to assess rigorously how the work compares to its competitors, both open and commercial.

It is important to recognize that course materials are evaluated and adopted by the instructors themselves, who care first and foremost about the quality of the instruction they offer. If they are advocates of open education, they have become so secondarily, subsequent to their assessment of its value for their students. For OER to become accepted as alternatives to commercial works, it is essential that instructors have confidence in them. So we make no apologies for insisting on rigorous review standards, if by doing so we have created quality benchmarks that will help the enormous creative energies liberated by the open education movement become long-term features of our educational landscape.

However, all of this begs the question as to just how open a work needs to be to accomplish its task of educating the reader. Away from the intellectual centers of the movement, the definition of open becomes very much a matter of perspective. From the student’s point of view, after all, affordability and accessibility are the criteria that best define the term, perhaps the only ones. Without these, education cannot take place. Accordingly, affordability and accessibility constitute what we shall call first-order openness, the ability of the user to discover, access, and use a work free from significant licensing or technical impediments. As our recent faculty survey amply demonstrates, some even believe that included in this order should be resources purchased or licensed by the library, which, if they do not qualify as open in the strict sense of the word, and certainly not as OER, at least meet the needs of a majority of students for free, readily accessible resources, particularly if they can be accessed digitally from any device.

Then there is open as defined by the instructor. Here the criteria become more complex and can include the instructor’s desire to modify the text to her personal instructional style, to her pedagogical goals, or in accordance with her understanding of the subject matter and the needs of her students. This might include combining texts in new and creative ways or sharing the combined texts, the mash-up, with her colleagues. This is second-order openness, second in the sense that it is largely invisible to the student, who views the “modified” text as every bit as canonical (or otherwise!) as a “straight” one. Works in this category, particularly those distributed under a CC-BY license, come close to satisfying the criteria of OER.

Finally, for many in the open educational community, the first two orders are way points toward a broader goal, that of making education itself truly open. Open education, of which, taxonomically speaking, OER are a part, advocates for the removal of barriers to education universally, whether these barriers are technical, economic, social, or political. The movement includes among its goals free or low-cost education for all; open access to courses, programs, and resources; and the sharing of information as freely and as widely as possible, aided in many cases by technology. Significantly, open education also entails a collaborative component in which both students and faculty, working together, create their own learning materials and share them with others. So this openness, of the third order, is much more programmatic and intentional than the other two, and certainly broader in scope. It entails no new properties of the text but rather is associated with the actions of those who use it, and as such is beyond the purview of the review itself.

In the reviews we are creating for Open Choice we have evaluated the review “target” against a strict definition of open as it is defined in the first two orders. That is, we have attempted to evaluate a work in terms of its conformity to the 5Rs in their most extended sense, and beyond that to certain pedagogical concerns, such as the extent to which it aligns with a course’s learning objectives, the ease of integration with an LMS, and so forth. But we do so in the realization that for many, including many instructors, the formal requirements of OER are only imperfectly understood, and that for some, open simply means free. We make no judgment about the degree of openness required by an individual instructor or course. It is up to the reader of the review to decide whether any of the criteria we score are irrelevant to his or her needs.—MC

About the author:

Mark Cummings is the editor and publisher at Choice

Related Posts