Education and demonstration – Question or problem eliciting response – Grading of response form
Reexamination Certificate
2001-02-12
2003-06-10
Cheng, Joe H. (Department: 3713)
Education and demonstration
Question or problem eliciting response
Grading of response form
C434S30700R, C434S350000
Reexamination Certificate
active
06577846
ABSTRACT:
BACKGROUND OF THE INVENTION
1. Field of the Invention
This invention is related to the general field of academic assessment testing, and within that field to the task of selecting test taker responses appropriate to be exemplar responses for the score points in open-ended (non-objective) assessment items, commonly referred to as “range finding”. Although the methods described herein may be useful in other related tasks, the most common application is likely to be range finding to provide exemplar responses for training the readers or scorers of the responses to the open-ended test items.
2. Description of Related Art
While multiple-choice and other objective assessment items have proven to be cost effective and generally accurate for many tests, open-ended or non-objective items are often specified for a variety of tests. Typically such open-ended items must be scored by human “readers”; that is, each response is read by one or more trained readers who assign appropriate scores to the response.
It is necessary to train multiple readers to give equivalent scores to assessment responses. Typically training is done by showing readers the rules under which the scores are given and sample responses representing all of the possible “score points”. The rules are embodied in a “rubric”, while the sample responses are included in “anchor sets”, “training sets”, and/or “qualifying sets” of items. So-called “anchor sets” typically consist of sample responses selected to represent an “exemplar” response for each score point. “Training sets” include exemplar responses selected to teach readers how to properly assign all possible scores. “Qualifying sets” consist of responses selected to verify that any reader can appropriately assign the proper score point to a student response after having been trained.
“Range finding” is a term that refers to selecting student responses appropriate to be the sample responses for training and related materials including, but not limited to, anchor sets, training sets, and qualifying sets. The term “range finding” reflects the fact that exemplar responses must be located or “found” representing the full range of scores. Range finding is typically performed on a sampling of unscored student responses. Often the sample of unscored materials is a subset of student responses “pulled” from a live administration of a test. In some cases the sample will be from a smaller “field test” administration designed to check the test before it is put into operational status.
When items are scored as credit
o credit (or other dichotomous scoring), exemplar samples can generally be found quickly. Many open-ended items, however, are scored on scales, typically with 4 to 6 score points. This significantly complicates the range finding since exemplar samples must be found for all points. In addition, many tests are designed to identify students that meet high standards, a result of which may be that high-end exemplars may be rare or even non-existent within the particular sample of student responses being examined.
Range finding has been traditionally performed by a team of content experts and scoring specialists working together with the paper responses. Members of the team go through unscored test materials looking for sample responses representing each score point. When one member “finds” what he or she believes to be an appropriate student response for a particular score point, at least one other member of the team confirms the score point for that response. This process can be tedious. Often the sample of papers consists of student responses clustered around one or two score points so that team members spend the bulk of their time looking at samples of the same score points again and again. As a result, they often resort to quickly glancing at responses “looking” for papers that might be examples of the higher score points. Sometimes they need to go through each and every student response in the sample, and even then may not find enough samples of each score point for training and qualifying readers.
The problem of scarce high score points is well known, and methods have been found methods to alleviate the problem. When large test administrations are the source of the sample of students, administrators will pick some student responses from traditionally high scoring schools or classrooms of honors students to include in the sample. When the sample is based on a field test, administrators will try to recruit participation by at least one or more high performing schools.
In assessment testing, answer documents may contain both pages that are processed by Optical Mark Reading (OMR), and separate pages that contain responses (or answers) to open-ended assessments, such as hand-written essays or other responses that can have a range of score points. When completed test documents are received for the scoring process, they are prepared for optical scanning and processed by an Optical Mark Scanner. In the scanning progress, the pages with OMR responses are processed and separated from the pages with open-ended responses. The pages with open-ended responses are divided into “packets” of materials to be read and scored. These packets are passed on to trained readers so that the open-ended responses can be scored.
When range finding is required to locate exemplar responses for the range of score points, some documents are diverted from the normal processing and become the sample of documents that are reviewed for range finding. When entire test documents are used for range finding, some documents are diverted to the range finding process prior to scanning and are examined by the range finding team. The responses selected as score point exemplars are then typically photocopied, and the copies are used for training materials. Once they have been reviewed for the range finding process, the diverted documents are put back into the production process for scanning.
The range finding is also sometimes done by diverting just a sample of pages with open-ended responses, rather than the entire test document. After scanning the test document, some pages containing open-ended responses are diverted to the range finding team for review as exemplars of score points. Again, the responses selected as exemplars are typically copied and the copies are used for training. Once they have been through the range finding process, the diverted sheets are put into packets and the packets are forwarded to readers for scoring.
When the entire test documents are large, the process of diverting only the open-ended response pages has the advantage of providing the range finding team with less paper. However, because the sheets containing student identification are typically separated during the scanning process, it would be difficult to locate student responses from students in specific schools or to locate responses from specific students, if it were desired to use that information to target sets of responses to examine for particular score points.
SUMMARY OF THE INVENTION
The current invention consists of methods to expedite and simplify the range finding process. First, the invention utilizes digitized representations, such as scanned images, of test taker answers to review for exemplars of score points, rather than review of the paper documents containing the student answers. Secondly, the invention uses a method of identifying targeted subsets of individual student answers which are likely to be exemplary of any given score point. In the preferred implementation of this invention, digital representations of the answers are linked to the student identity and some student performance indicator so that a computer system can determine a targeted subset. Alternatively, the digital representations could be linked to a performance indicator without utilization of the student identity. The invention also allows for expanding the sample of answers available to range finding, storing evaluation data, and sequencing answers.
Use of Digitized Representation of Open-Ended Responses
Range finding is significantly enhanc
Cheng Joe H.
Christman Kathleen M
CTB-McGraw Hill, LLC
Rothwell Figg Ernst & Manbeck P.C.
LandOfFree
Methods for range finding of open-ended assessments does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Methods for range finding of open-ended assessments, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Methods for range finding of open-ended assessments will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3100926