One way to use degrees of certitude

Reading Prof. Gardner-Medwin comments (here) it obviously appears that my point of view needed some clarification. The way I use degrees of certitude will be described in this post. It is only one way, not pretending to be the best, just my way. Since degrees of certitude (or degrees of certainty) do not influence the mark it is not de facto a confidence based marking system.

Students are free to associate degrees of certitude with their answers. If they don’t use degree of certitude or if they use them being wrong or being right their score will not change. They answer MCQ choosing one out of 7 to 8 propositions, 4 being always the same: none of the proposition, all propositions together, missing data to answer the question, the question doesn’t make sense. The tariff is

• for a correct answer: +1
• for an incorrect answer: -0.5
• for not answering or omission: -0.25

Training a) to answer this kind of MCQ, b) to use degrees of certitude and c) specific training regarding the topics evaluated with these tests are organised, both on line and during simulated exams. Students are free to participate. If not using CBM they receive their scores. Students using CBM for all their answers receive detailed results. They connect to the web and using their student identity number they download their full results.

On a willing base they can answer 20 questions about their performance helping them to analyse their results and regulate their learning for the next test.

Five tests are organised during the year. None is mandatory. Student are free not to participate to any and only take the final oral exam.

1. The first lesson in September is a test about prerequisite knowledge.
2. The second test in December is a formative training test with MCQ and open ended questions covering all subjects seen at the point of the formation.
3. The third test in January is a summative test that will influence the final score if the test is taken. It is the same kind of test than the previous.
4. The fourth test in Ferburay is a formative evaluation covering all subjects of the year course.
5. The fifth test in March is a summative test covering all subjects of the year course.

 

The first summative test, if taken, will count for 15% of the final score. The second summative test, if taken, will count for 35% of the final score. The oral exam in June will take the remaining percentage, meaning

• 100% if no test was taken during the year,
• 85% if the first summative test was taken,
• 65% if the second summative test was taken,
• 50% if both summative test were taken.

If a student pass both summative tests with a score above 60% he then can choose

1. to take the oral exam for 50% of the final score.
2. to report his score to the oral exam so that his grade for both summative tests will be his final grade.

Any student who has not taken both test or not passed with 60% or more is obliged to take the oral exam. Any student who has taken all 5 tests and reached a score between 45% and 60% will be evaluated on the quality of his comments. If metacognitive analysis and regulations are “good” bonus points will be added to the student actual score. The bonus points allow the student to reach the 60% needed to be able to keep his grade and not being obliged to take the oral exam.

It is a long explanation. Let’s hope it makes things easier to understand. Next post will present the tool the help student to analyse and regulated their learning using results collected with the use of CBM.

Confidence weighting or Certainty Marking?

or from confidence weighting to certainty-based marking

In a previous post Robert Ebel’s definition of confidence weighting was presented: answers are associated with certainty of the answer correctness. Ebel uses weighting which in his definition is associated with a mode of scoring. Gardner-Medwin (see the post here) uses marking, which is related to words like evaluation, grade. Because of the use of “marking” both confidence or certainty are associated with a mark scheme.

Ebel’s objectives for using confidence weighting are related to minimize score by guessing (“to reduce the chance error component on a test score”).

Gardner-Medwin’s objectives range

• from “risk losing marks if wrong” which is related to words like tariff, mark scheme, reckoner,…
• to “think about how reliable your answer” which is related to metacognitive judgment.

I’ve already mentioned that it is my opinion that in evaluation, degrees of certainty can be successfully used without being associated with reckoner or tariff. But I must add that mark scheme might interfere with honest expression of confidence using degree of certainty. In a survey on 349 students, 52% admitted that the were using one specific strategy, some always, some time to time, to choose the degree of certainty given with an answer instead of giving an honest expression of how reliable the chosen answer was. This survey, with other facts, suggests that using degree of certainty with a mark scheme might bias the honest expression of confidence or doubt associated with an answer.

Depending of the objectives linked with the use of degree of certainty a careful choice must be made regarding he fact of using or not using a mark scheme, tariff or reckoner because of possible bias using a strategy to answer instead of honest expression of confidence.

Definition of Certainty Marking

or From confidence marking to certainty marking.Tony Gardner-Mewin on his page at University College of London (UCL, LAPT page) mentions that: ” The word “certainty” seems to carry much less baggage than “confidence”. The term “Confidence-based marking” has sometimes suggested to people that confident personalities are being rewarded. This is not so. Those who are rewarded are those who can distinguish between reliable and unreliable answers. In the context of “How certain are you that this is right?” or “How confident are you that this is right?”, they are equivalent. But “Certainty-Based Marking” is perhaps less open to misinterpretation.“

LAPT’s mark scheme (compare with the scheme in this post)

Gardner-Medwin objectives for using degree of certainty are (from lapt’s website):

• To make you think about how reliable your answer is
• To encourage you to try to understand the issues, not just to react immediately to a question
• To encourage you to think laterally: other pieces of knowledge may help to validate or question your answer
• To challenge you – if you won’t risk losing marks if wrong, then you don’t really know the answer.
• If you are a careful thinker, but not very confident, you will gain in confidence
• It is more fair – a thoughtful and confident correct answer deserves more marks than a lucky hunch
• You need to pay attention if you make confident wrong answers – to think, reflect and learn!
• Efficient study requires that you constantly question how your ideas arise, and how reliable they are.

What is the difference between confidence weighting and certainty marking? Answer in the next post.