Free and Open Resource Curation Rubric
Knovation’s Free and Open Resource Curation Rubric is meant to be applied to a variety of learning resource types, such as videos, articles, activities, interactive media, and whiteboard resources. While it can be applied across large scale websites and lesson plans, it’s primarily meant to evaluate self-contained learning resources where the student is the primary audience.
It is composed of collections of rubrics, each grouped within one of three categories:
- Publisher Rubrics
- Resource Rubrics
- Resource Type Rubrics
Often, OER rubrics favor rigor over usability. While this tool was designed to apply rigor to the evaluation process, it also places a strong emphasis on practical application and ease of use, particularly in situations where resource curation is performed at scale.
This is accomplished by performing the first round of evaluation on all resources created by a single Publisher. When scoring one or more resources using the Publisher rubrics, scores are awarded according to the actual characteristics of all resources that originate from a single source.
Resource rubrics examine both the assumed effectiveness (e.g., Ease of Use) and the actual effectiveness (e.g., Age of Information) for all resources, independent of format or type.
The third category of rubrics, Resource Type rubrics, evaluates the assumed effectiveness of specific types of resources by examining characteristics that make them unique. Rarely will all rubrics in this category apply to a single resource.
As a resource is evaluated, a score is awarded based on whether or not it meets “Superior” or “Strong” criteria. If the resource does not meet or exceed “Strong” criteria, it is rejected.
Amount and Nature of Advertisements
Resources from Superior Publishers do not include advertisements, pop-ups, or content that distracts from the topic, concepts, or skills taught within the resource.
It is recognized that publishers must often include advertisements in order to fund the creation of content. Therefore, a resource can be considered strong if it includes 2 or fewer advertisements. Advertisements must be appropriate, and may not obstruct the use of the resource, or become an intrusion within its overall flow. Resources with advertisements that are not clearly marked, distract the user, or influence the user to focus on unrelated content are not considered Strong resources.
Required Authentication for Access
Resources from Superior Publishers do not require authentication for access, thus improving their accessibility for students and educators. Resources can be accessed from a publicly available URL.
Resources from Publishers can be considered strong if authentication is not required to access the page's contents, but it is used in order to store progress or save user created work. While rare, there are exceptions that may influence a curator to harvest resources that require free login (e.g., limited resources available on a topic).
Resources are considered Strong if student data practices are not clearly stated, but login is not supported, and users are not prompted to enter information that could be used to track their identity.
Level of Author and/or Publisher Authority
Superior Resources originate from a source with a strong reputation, are properly cited, and whose author(s) demonstrate no clear signs of bias. Superior Publishers are often institutions of higher learning, non-profit organizations, or government agencies.
Resources are considered Strong if the author’s qualifications are unclear, but the source has demonstrated a reputation for creating accurate, educationally-relevant content.
Accuracy of Information
Superior Resources include information that is properly cited and can be verified as containing no substantial errors or inaccuracies.
Strong resources lack proper citation, but can be verified as containing no substantial errors or inaccuracies.
Opportunity for Student Engagement (Interest)
Superior Resources are designed to provide an engaging experience for students, often making use of visual elements, sounds, animation, and multiple forms of media that capture students’ interest and attention.
Resources can be considered strong if they express information primarily through text, but are written in a way that provides accurate and relevant information.
Age of Information
Superior Resources are created using technology that makes use of modern web standards, and presents information in ways that recognize observed usability and user experience best practice. Information is up to date and reflects the current understanding of the topics or concepts.
Resources can be considered strong if the website shows visible or functional signs of age, but information is up to date and reflects the current understanding of the topics or concepts.
Alignment to Curriculum Concepts
Superior Resources are primarily focused on topics or concepts typically expressed within curriculum or delivered during instruction. They support standards or skills that are addressed by the majority of educators in classroom settings.
Resources can be considered strong if they can be applied to the typical curriculum with some modifications, or address topics that do not receive direct focus within instruction, but are commonly of interest to students.
Ease of Use
Superior Resources require little up-front explanation for use by teachers and students, and no training is required to support immediate use by students.
Resources can be considered strong if use of the resource requires that students or teachers read and understand a short set of directions.
Degree of Alignment to Standards
A resource has superior alignment only if both of the following are true:
- All of the content and performance expectations in the identified standard are completely addressed by the resource.
- The content and performance expectations of the identified standard are the focus of the resource. While some resources may cover a range of standards that could potentially be aligned, for a superior alignment the content and performance expectations must not be a peripheral part of the resource.
A resource has strong alignment for either one of two reasons:
- Minor elements of the standard are not addressed in the resource.
- The content and performance expectations of the standard align to a minor part of the resource.
This rubric is only applied to learning resources that have suggested alignments to standards.
This rubric follows the Achieve Rubric for Evaluating OER Objects: Rubric 1: Degree of Alignment to Standards
Utility of Materials Designed to Support Teaching
A resource is rated superior for the utility of materials designed to support teaching only if all of the following are true:
- The resource provides materials that are comprehensive and easy to understand and use.
- The resource includes suggestions for ways to use the materials with a variety of learners. These suggestions include materials such as “common error analysis tips” and “precursor skills and knowledge” that go beyond the basic lesson or unit elements.
- All resources and all components are provided and function as intended and described. For example, the time needed for lesson planning appears accurately estimated, materials lists are complete, and explanations make sense.
- For larger resources like units, materials facilitate the use of a mix of instructional approaches (direct instruction, group work, investigations, etc.).
A resource is rated strong for the utility of materials designed to support teaching if it offers materials that are comprehensive and easy to understand and use but falls short of “superior” for either one of two reasons:
- The resource does not include suggestions for ways to use the materials with a variety of learners (e.g., error analysis tips).
- Some core components (e.g., directions) are underdeveloped in the resource.
This rubric is only applied to resources designed to support teachers in planning or presenting subject matter.
This rubric follows the Achieve Rubric for Evaluating OER Objects: Rubric 3: Utility of Materials Designed to Support Teaching
Quality of Assessments
A resource is rated superior for the quality of its assessments only if all of the following are true:
- All of the skills and knowledge assessed align clearly to the content and performance expectations intended, as stated or implied in the resource.
- Nothing is assessed that is not included in the scope of intended material unless it is differentiated as extension material.
- The most important aspects of the expectations are targeted and are given appropriate weight/attention in the assessment.
- The assessment modes used in the resource, such as selected response, long and short constructed response, or group work require the student to demonstrate proficiency in the intended concept/skill.
- The level of difficulty is a result of the complexity of the subject-area content and performance expectations and of the degree of cognitive demand, rather than a result of unrelated issues (e.g. overly complex vocabulary used in math word problems).
A resource is rated strong for the quality of its assessments if it assesses all of the content and performance expectations intended, but the assessment modes used do not consistently offer the student opportunities to demonstrate proficiency in the intended concept/skill.
This rubric is applied to those resources designed to determine what a student knows before, during, or after a topic is taught.
This rubric follows the Achieve Rubric for Evaluating OER Objects: Rubric 4: Quality of Assessments
Quality of Instructional and Practice Exercises
A resource is rated superior for the quality of its instructional and practice exercises only if all of the following are true:
- The resource offers more exercises than needed for the average student to facilitate mastery of the targeted skills, as stated or implied in the resource. For complex tasks, one or two rich practice exercises may be considered more than enough.
- The exercises are clearly written and supported by accurate answer keys or scoring guidelines as applicable.
- There are a variety of exercise types and/or the exercises are available in a variety of formats, as appropriate to the targeted concepts and skills. For more complex practice exercises the formats used provide an opportunity for the learner to integrate a variety of skills.
A resource is rated strong for the quality of its instructional and practice exercises if it offers only a sufficient number of well-written exercises to facilitate mastery of targeted skills, which are supported by accurate answer keys or scoring guidelines, but there is little variety of exercise types or formats.
This rubric is applied to resources that contain exercises designed to provide an opportunity to practice and strengthen specific skills and knowledge.
This rubric follows the Achieve Rubric for Evaluating OER Objects: Rubric 6: Quality of Instructional and Practice Exercises