Our online automatic marking of student open or free text short
answers overcomes the narrow constraints of multiple choice testing and allows
for open-ended student responses. Marks are awarded for sentences, phrases or
words, that match the key ideas of the mark scheme. There are no additional
marks for the repetition of key ideas. Test elements force an improvement in
student essay skills by insisting, for example, upon explicit reference being
made to the question subject words. The tests also check for basic grammar and
punctuation. Extra spaces entered in error between words are ignored, as are the
capitalization/non-capitalization of letters. Correct spelling is essential, and
a spell checker is provided. Diagnostic feedback is provided at the end of the
tests, together with authenticated certificate awards.
The successful automatic marking of free text answers has
required extensive trials with geography students in Barcelona.
The example mark scheme below shows some of the available
answers for an open ended or free text input test on Plate
Tectonics : "with reference to a located example, explain how
tectonic processes at a destructive plate margin result in the formation of
landforms. (6 marks)". This is a common AS level geography examination
question in the UK.
Mark Scheme Details
After testing with students in Barcelona, the automated mark
scheme below has been adjusted to focus more on processes. This makes the test
more difficult, but stops high marks being generated simply by referring to a
range of appropriate landforms. Nine possible key ideas have been scripted for
this question, with many other alternative wording possibilities also providing
(but not duplicating) a mark. There are potentially unlimited alternative
mark-gaining responses that can be scripted in the mark scheme.
The nine key ideas (with some examples of one, two or three
word answers that are awarded marks) are:
Note that many phrase and sentence text answer inputs are also
supported within the mark scheme. They are not listed here.
There are a maximum of 6 marks for this question, so
over-scoring is possible. Key ideas are taken from Waugh's Integrated Approach
text.
Student free text input for the key ideas above gain (but do
not duplicate) marks. For example, 'Two plates converge' gains
the same mark as 'two plates collide', with no mark
duplication. Similarly, 'Japan' gains a mark within the 'accurate
location' key idea, but mentioning the Andes does not gain an additional
mark.
There is a simple alert put in place to ensure that students
refer to the subject words of the question in their answer. This itself, in
testing with students, had an immediate impact upon their performance. They were
forced to refer to the question appropriately.
The number of marks available and alternative wording are
unlimited.
Note that extra spaces entered in error between words are
ignored, as are the capitalization/non-capitalization of letters.
The key test source code is not visible, neither is it stored
in your computer's memory cache.
1. Spell checker to be replaced with improved version.
5.11.2005
2. Optional test timer to be added. 4.12.2005
3. Test certificates to be integrated into the tests.
12.11.2005
4. An online interface to be provided for school staff to enter question(s),
optional image(s) and mark scheme with alternative answer wording. The current
state of the
advanced teacher test interface is online at http://geographyfieldwork.com/auto_mark.htm
When students type their test answers online, their input will be automatically
scanned for key ideas (supported by a large synonyms/alternative wording
database), and in context, (with double-scanning of primary and secondary key
ideas). The answers will be graded for the key ideas, and for spelling and
grammar - with diagnostic feedback on all errors, and key ideas identified and
missed. A certificate at test completion will contain a copy of the student
answer, plus grade and all diagnostic data. A spelling and grammar checker will
be provided as an option for all tests. Estimated completion Easter 2007.
5. End-of-test diagnostic feedback to be provided. 12.11.2005
6. Client-side source code to be removed. 17.12.2005
7. The script currently works in Internet Explorer browsers only.
6.1.2007
8. Marking in context to be improved. 3.1.2006
9. Bug that can cause freezing of check answer button removed.
3.1.2006
