Quality Matters: Inter-Institutional Quality Assurance in Online Learning

PEER COURSE REVIEW RUBRIC (Pilot Version)
Alignment: Critical course components – Course Objectives (III), Learner Interactions and Activities (V), Resources, Materials, and Technology (IV and VI), Assessment and Measurement (III) – work together to ensure that students achieve the desired learning outcomes. Specific standards included in Alignment are indicated in the rubric annotations.
I. COURSE OVERVIEW AND INTRODUCTION ↑
General Review Standard: The overall design of the course, navigational information, as well as course, instructor and student information are made transparent to the student at the beginning of the course.
Specific Review Standards: | Points | Annotation: What’s the idea? |
---|---|---|
I.1 Navigational instructions make the organization of the course easy to understand. |
3 |
Instructions provide a general course overview, guide the new student to explore the course website, and indicate what to do first, rather than list detailed navigational instructions for the whole course. Instructors may choose to incorporate some of this information in the course syllabus. If so, students should be directed to the syllabus at the beginning of the course. A useful idea is a “Read Me First” or “Start Here” button or icon on the course home page, linking students to start-up information. Examples:
|
I.2 A statement introduces the student to the course and to the structure of the student learning. |
3 |
The instructor’s statement gives the new student an idea of how the learning process is structured including schedule, communications modes, types of activities, and assessments. These features are often found in the course syllabus. Look for some or all of the following:
|
I.3 Netiquette expectations with regard to discussions and email communication are clearly stated. |
2 |
Expectations of student conduct online are clearly stated, however brief or elaborate they may be. The expectations themselves are not evaluated. Examples:
|
I.4 The self-introduction by the instructor is appropriate. |
1 |
The initial introduction creates a sense of connection between the instructor and the students. It should present the instructor as professional as well as approachable, and include more than the essentials, such as the instructor’s name, title, field of expertise, email address and phone. The self introduction helps students get to know the instructor. It could include:
|
I.5 Students are requested to introduce themselves to the class. |
1 |
The student introduction helps to create a supportive learning environment and a sense of community. Students are asked to introduce themselves and given guidance on where and how they should do so. Student introductions themselves are not evaluated. Instructors may ask students to answer specific questions (such as why they are taking the course, what concerns they have, what they expect to learn, etc.) or may choose to let the student decide. Instructors may provide an example of an introduction and/or start the process by introducing themselves. |
I.6 Minimum technology requirements, minimum student skills, and, if applicable, prerequisite knowledge in the discipline, are clearly stated. |
1 |
Explanations of technical requirements and skills, and prerequisite knowledge and skills may be found within the course, in documents linked to the course, or in supporting material not on the course site. Look for a link to that content and/or a reminder of it for the entering student. Technology requirements may include information on:
Examples of technology skills may include the capability to:
Discipline knowledge prerequisites should include academic course prerequisites. |
II. LEARNING OBJECTIVES (COMPETENCIES) ↑
General Review Standard: Learning objectives are clearly defined and explained. They assist the student to focus learning activities.
Specific Review Standards: | Points | Annotation: What’s the idea? |
---|---|---|
II.1 The learning objectives of the course describe outcomes that are measurable. |
3 |
Measurable learning objectives ensure instructors precisely describe what students are to gain from instruction, and then guide instructors to accurately assess student accomplishment. Objectives should describe student performance in specific, observable terms. If this is not possible, (e.g., internal cognition, affective changes), check for clear indications that the learning objective is meaningfully assessed. Special situations: In some cases (check the Instructor Worksheet), objectives for the course are institutionally mandated and the individual instructor does not have the authority to change them. For such cases, review Standard II.1 in connection with Standard II.5 below. If the specific learning outcomes on the module/unit level are measurable but the institutionally mandated learning objectives for the course are not, Standard II.1 should be considered as met for scoring purposes. Please note in the “comments” box that the institutionally mandated learning objectives for the course are not measurable. Examples of measurable objectives:
Alignment: This standard is included in Alignment. (Critical course components work together to ensure that students achieve the desired learning outcomes.) |
II.2 The learning objectives address content mastery, critical thinking skills, and core learning skills. |
3 |
Examine the learning objectives (course and unit level) as a whole for all three types of skill. Every single objective may not contain all three components. Content mastery should be appropriate for the type and level of the course. Ask the Subject Matter Expert (SME) for guidance. Critical thinking skills may include the ability to:
Core learning skills may include:
Alignment: This standard is included in Alignment. (Critical course components work together to ensure that students achieve the desired learning outcomes.) |
II.3 The learning objectives of the course are clearly stated and understandable to the student. |
2 |
Students can easily grasp the meaning of the learning objectives. Use of jargon, confusing terms, unnecessarily complex language, and puzzling syntax are avoided. Special situations: In some cases (check the Instructor Worksheet), objectives for the course are institutionally mandated and the individual instructor does not have the authority to change them. For such courses, review Standard II.4 in connection with Standard II.5 below. If the specific learning outcomes on the module/unit level are clearly stated and understandable to the student but the institutionally mandated learning objectives for the course are not, Standard II.4 should be considered as met for scoring purposes. Please note in the “comments” box that the institutionally mandated learning objectives for the course are not clearly stated and understandable. |
II.4 Instructions to students on how to meet the learning objectives are adequate and easy to understand |
2 |
Instructions may take various forms (e.g. narratives, bulleted lists, charts) and may appear at different levels within the course, such as module-based or weekly assignment sheets. Instructions are clear and complete. Examples:
|
II.5 The learning objectives of the course are articulated and specified on the module/unit level. |
2 |
Module or unit level objectives may be written by the instructor or come from the textbook. If no module or unit level objectives are found or referenced on the website, reviewers should communicate with the instructor to gather more information. |
III. ASSESSMENT AND MEASUREMENT ↑
General Review Standard: Assessment strategies use established ways to measure effective learning, assess student progress by reference to stated learning objectives, and are designed as essential to the learning process.
Specific Review Standards: | Points | Annotation: What’s the idea? |
---|---|---|
III.1 The types of assessments selected measure the stated learning objectives and are consistent with course activities and resources. |
3 |
Assessments, learning objectives, and learning activities align in a clear and directway. The assessment formats provide a reasonable way to measure the statedlearning objectives. Examples of inconsistency:
Examples of objective/assessment alignment:
Some assessments may be geared towards meeting objectives other than those stated in the course; for example, a course may have a writing component as part of a college-wide “Writing Across the Curriculum” requirement. In that case suggest including appropriate objectives in the course. (Note: Learning Objectives may be called Learning Outcomes.) Alignment: This standard is included in Alignment (Critical course components work together to ensure that students achieve the desired learning outcomes.) |
III.3 Assessment and measurement strategies provide feedback to the student. |
3 |
Students learn more effectively if they receive frequent, meaningful, and rapid feedback. This feedback may come from the instructor directly, from assignments and assessments that have feedback built into them, or even from other students. Examples:
|
III.4 The types of assessments selected and the methods used for submitting assessments are appropriate for the distance learning environment. |
2 |
Assessments make use of the technologies and security typically found in an online classroom. Examples that DO meet the standard:
Examples that do NOT meet the standard:
|
III.5 “Self-check” or practice types of assignments are provided for quick student feedback. |
1 |
Students have ample opportunity to measure their own learning progress. Look for examples of “self-check” quizzes and activities, as well as other types of practice opportunities that provide rapid feedback. These types of assignments should be voluntary or allow multiple attempts. Examples:
|
IV. RESOURCES AND MATERIALS ↑
General Review Standard: Instructional materials are sufficiently comprehensive to achieve announced objectives and learning outcomes and are prepared by qualified persons competent in their fields. (Materials, other than standard textbooks produced by recognized publishers, are prepared by the instructor or distance educators skilled in preparing materials for distance learning.)
Specific Review Standards: | Points | Annotation: What’s the idea? |
---|---|---|
IV.1 The instructional materials support the stated learning objectives and have sufficient breadth and depth for the student to learn the subject. |
3 |
Instructions should provide meaningful content in a variety of ways, including the textbook, PowerPoint presentations, websites, lecture notes, outlines, and multimedia. Decisions on this standard may be particularly difficult for individual reviewers whose expertise is not in the course discipline. Reviewers should consult with the SME (subject matter expert) and use common sense to determine if the content is robust enough to support the course. Alignment: This standard is included in Alignment. (Critical course components work together to ensure that students achieve the desired learning outcomes.) |
IV.2 Instructional materials are presented in a format appropriate to the online environment, and are easily accessible to and usable by the student. |
3 |
Students who have the required technical equipment and software can view the materials online. If some of the course resources, including textbooks, videos, CD-ROMs, etc., are unavailable within the framework of the course website, determine how students would gain access to them and examine their ease of use. Examples:
Examples of some visual format problems:
|
IV.3 The purpose of the course elements (content, instructional methods, technologies, and course materials) is evident. |
2 |
Students can easily determine the purpose of all materials, technologies and methods used in the course and know which materials are required and which are recommended resources. For example, a course may be richly garnished with external links to Internet resources, but it is not clear whether those resources are for background information, additional personal enrichment, or required for an assignment. Examples:
|
IV.4 The instructional materials, including supporting materials – such as manuals, videos, CD ROMs, and computer software are consistent in organization. |
1 |
Online courses often use multiple types of instructional materials. Students can easily understand how the materials relate to each other. The level of detail in supporting materials is appropriate for the level of the course. For example, a course requires students to use the following materials: a textbook divided into chapters, video segments ordered by topics, a website organized around specific skills, and a tutorial CD-ROM that has an opening menu consisting of “practice quizzes,” “images,” and “audio examples.” Reviewers would need to determine if such diversely formatted course materials are integrated well enough to be useful to the uninitiated student. Example:
Decisions on this standard may be particularly difficult for individual reviewers whose expertise is not in the course discipline. Reviewers should consult with the SME (subject matter expert) and use common sense to determine if the materials are appropriate to this course. |
IV.5 All resources and materials used in the online course are appropriately cited. |
1 |
Materials created by the instructor and those borrowed from other sources are distinctly identified. Text, images, graphic materials, tables, videos, audios, websites, and other forms of multimedia are appropriately referenced according to the institution’s copyright and intellectual property policy. Courses that use an e-pack or course cartridge may provide a blanket statement acknowledging that a significant portion of the course materials came from the publisher rather than include individual citations for each instance of publisher materials. |
V. LEARNER INTERACTION ↑
General Review Standard: The effective design of instructor-student interaction, meaningful student cooperation, and student-content interaction is essential to student motivation, intellectual commitment and personal development.
Specific Review Standards: | Points | Annotation: What’s the idea? |
---|---|---|
V.1 The learning activities promote the achievement of stated objectives and learning outcomes. |
3 |
Learning activities are various including class discussions, case studies, simulation exercise, practice quizzes, tests, etc. Activities align with and support the learning objectives. Most of the objectives can reasonably be achieved by students completing the learning activities. Examples of mismatches between activities and objectives:
Alignment: This standard is included in Alignment. (Critical course components work together to ensure that students achieve the desired learning outcomes.) |
V.2 Learning activities foster instructorstudent, contentstudent, and if appropriate to this course, student-student interaction. |
3 |
All online courses should include interaction between the instructor and the students and between the students and the content. The degree and type of student-to-student interaction may vary with the discipline and the level of the course. Examples of learning activities that foster the following types of interaction:
|
V.3 Clear standards are set for instructor response and availability (turnaround time for email, grade posting, etc.) |
3 |
Information clearly indicates how quickly the instructor will respond, when feedback will be provided, and when the instructor is available to meet. Information clearly indicates instructor response time for key events and interactions, including e-mail turnaround time, time required for grade postings, discussion postings, etc. Standards also include instructor availability, including email response time, degree of participation in discussions, and availability via other media (phone, in-person) if applicable. This standard does not prescribe what that response time and availability ought to be. |
V.4 The requirements for course interaction are clearly articulated. |
2 |
A clear statement of requirements should indicate the criteria for interaction. For example, students required to participate in discussions are told how many times each week they must post original comments, how many times they must post responses to other’s comments, what the quality of the comments must be, how the comments will be evaluated, what grade credit they can expect for various levels of performance, and whether the interaction is required or optional. |
V.5 The course design prompts the instructor to be present, active, and engaged with the students. |
2 |
Students know that the instructor is approachable and will regularly interact with them. Opportunities for interaction will vary with the discipline of the course. Examples:
|
VI. COURSE TECHNOLOGY ↑
General Review Standard: To enhance student learning, course technology enriches instruction and fosters student interactivity.
Specific Review Standards: | Points | Annotation: What’s the idea? |
---|---|---|
VI.1 The tools and media support the learning objectives of the course and are integrated with texts and lesson assignments. |
3 |
Tools and media used in the course support related learning objectives, and are integrated with texts and lesson assignments. Students know how the tools and media support the assignments and how they support the learning objectives. Technology is not used simply for the sake of using technology. For example, a course might require viewing video materials, but it may not be clear how the video materials illustrate or support any learning objective. Alignment: This standard is included in Alignment. (Critical course components work together to ensure that students achieve the desired learning outcomes.) |
VI.2 The tools and media enhance student interactivity and guide the student to become a more active learner. |
2 |
Tools and media used in the course help students actively engage in the learning process, rather than passively “absorbing” information. Examples:
|
VI.3 Technologies required for this course are either provided or easily downloadable. |
2 |
For this standard, the term “technologies” may cover a range of plug-ins such as Acrobat Reader, media players, etc. In addition, courses may require special software packages (spreadsheets, math calculators etc.). Clear instructions tell students how to obtain needed plug-ins and software packages. |
VI.4 The tools and media are compatible with existing standards of delivery modes. |
1 |
Course tools, media, and delivery modes meet current standards for widespread accessibility. For example, if most students have access to DVD players or use streaming media, use of those delivery modes in an online class meets this standard. If the typical student cannot be expected to have access to a technology at his or her out-of-the box home computer off campus, that technology should probably not be used in the course. |
VI.5 Instructions on how to access resources at a distance are sufficient and easy to understand. |
1 |
Online students need to know about and be able to obtain access to educational resources by remote access. Information on these resources is readily visible with clear instructions on how to access these resources. Examples:
|
VI.6 Course technologies take advantage of existing economies and efficiencies of delivery. |
1 |
As innovative technologies appear on the market all the time, online course technology should be current. Courses not recently developed may need to be updated. (Check the Instructor Worksheet). Examples:
|
VII. LEARNER SUPPORT ↑
General Review Standard: Courses are effectively supported for students through fully accessible modes of delivery, resources, and student support.
Specific Review Standards: | Points | Annotation: What’s the idea? |
---|---|---|
VII.1 The course instructions articulate or link to a clear description of the technical support offered. |
2 |
Technical support for online students differs from institution to institution. Technical support includes information about such topics as how to log in, how to use the software, and how to upload files. It does not include help with course content, assignments, academic or student support services (see Standards VII.2 and VII.3 below). Look for evidence that students have access to technical support services from within the course. The purpose is not to review the adequacy of those services on an institutional level. Examples:
|
VII.2 Course instructions articulate or link to an explanation of how the institution’s academic support system can assist the student in effectively using the resources provided. |
2 |
Academic support for online students, and the scope of what “academic support” entails, differs from institution to institution. For the purposes of review, academic support includes access to library resources, readiness assessment, testing services, tutoring, a writing center, a math center, supplemental instruction programs, and teaching assistants. Look for evidence that students have access to academic support services from within the course. The purpose is not to review the adequacy of those services on an institutional level. Examples:
|
VII.3 Course instructions articulate or link to an explanation of how the institution’s student support services can assist the student in effectively using the resources provided. |
1 |
Student support services for online students, and the scope of what such support entails, differs from institution to institution. For the purposes of this review, student support services include advising, registration, financial aid, student life, counseling, etc. Look for evidence that students have access to student support services from within the course. The purpose is not to review the adequacy of those services on an institutional level. Examples:
|
VII.4 Course instructions articulate or link to tutorials and resources that answer basic questions related to research, writing, technology etc. |
1 |
Online students’ access to tutorials and help files related to writing, technology, research, etc. differs from institution to institution. This item does NOT refer to:
Look for evidence that students taking the course DO have access to those support services from within the course, for example, a clear description of the tutorials available and how to get them. |
VIII. ACCESSIBILITY ↑
General Review Standard: The course is accessible to all students.
Specific Review Standards: | Points | Annotation: What’s the idea? |
---|---|---|
VIII.1 The course acknowledges the importance of ADA requirements. |
3 |
Disabilities Act (ADA) services on their campus. There should be a statement in the course that tells students how to gain access to ADA services at their institution. To meet this standard a course must have both a statement that tells students how to gain access to an institution’s ADA services AND be on an approved Course Management System (Blackboard, WebCT, or WebTycho) or documentation provided by the CMS that it is ADA compliant. Encourage faculty to consult the office on their campus that provides disability services for the wording of an ADA Statement appropriate to their institution. |
VIII.2 Web pages provide equivalent alternatives to auditory and visual content. |
1 |
Alternative means of access to course information are provided for the vision or hearing impaired student, such as, equivalent textual representations of images, audio, animations, and video in the course website. Presenting information in text format is generally acceptable because screen reader software (used by the vision impaired) can read text. Examples:
|
VIII.3 Web pages have links that are self-describing and meaningful. |
1 |
Instructors provide links to Internet content that includes useful descriptions of what students will find at those sites. These descriptions enable the vision impaired student to use screen reader software to understand links. Examples:
|
VIII.4 The course demonstrates sensitivity to readability issues. |
1 |
The course employs appropriate font, color, and spacing to facilitate readability and minimize distractions for the student. Examples:
|
Meets Expectations if: ↑
Answered ‘Yes’ to all 3-point Essential Standards: I.1, I.2, II.1, II.2, III.1, III.2, III.3, IV.1, IV.2, V.1, V.2, V.3, VI.1, VIII.1
AND
68 points or more