One way to use degrees of certitude

Reading Prof. Gardner-Medwin comments (here) it obviously appears that my point of view needed some clarification. The way I use degrees of certitude will be described in this post. It is only one way, not pretending to be the best, just my way. Since degrees of certitude (or degrees of certainty) do not influence the mark it is not de facto a confidence based marking system.

Students are free to associate degrees of certitude with their answers. If they don’t use degree of certitude or if they use them being wrong or being right their score will not change. They answer MCQ choosing one out of 7 to 8 propositions, 4 being always the same: none of the proposition, all propositions together, missing data to answer the question, the question doesn’t make sense. The tariff is

• for a correct answer: +1
• for an incorrect answer: -0.5
• for not answering or omission: -0.25

Training a) to answer this kind of MCQ, b) to use degrees of certitude and c) specific training regarding the topics evaluated with these tests are organised, both on line and during simulated exams. Students are free to participate. If not using CBM they receive their scores. Students using CBM for all their answers receive detailed results. They connect to the web and using their student identity number they download their full results.

On a willing base they can answer 20 questions about their performance helping them to analyse their results and regulate their learning for the next test.

Five tests are organised during the year. None is mandatory. Student are free not to participate to any and only take the final oral exam.

1. The first lesson in September is a test about prerequisite knowledge.
2. The second test in December is a formative training test with MCQ and open ended questions covering all subjects seen at the point of the formation.
3. The third test in January is a summative test that will influence the final score if the test is taken. It is the same kind of test than the previous.
4. The fourth test in Ferburay is a formative evaluation covering all subjects of the year course.
5. The fifth test in March is a summative test covering all subjects of the year course.

 

The first summative test, if taken, will count for 15% of the final score. The second summative test, if taken, will count for 35% of the final score. The oral exam in June will take the remaining percentage, meaning

• 100% if no test was taken during the year,
• 85% if the first summative test was taken,
• 65% if the second summative test was taken,
• 50% if both summative test were taken.

If a student pass both summative tests with a score above 60% he then can choose

1. to take the oral exam for 50% of the final score.
2. to report his score to the oral exam so that his grade for both summative tests will be his final grade.

Any student who has not taken both test or not passed with 60% or more is obliged to take the oral exam. Any student who has taken all 5 tests and reached a score between 45% and 60% will be evaluated on the quality of his comments. If metacognitive analysis and regulations are “good” bonus points will be added to the student actual score. The bonus points allow the student to reach the 60% needed to be able to keep his grade and not being obliged to take the oral exam.

It is a long explanation. Let’s hope it makes things easier to understand. Next post will present the tool the help student to analyse and regulated their learning using results collected with the use of CBM.

Definition of confidence

Another word from the 14th century. As a noun it covers different meanings like: 

• a feeling or consciousness of one’s powers or of reliance on one’s circumstances; faith or belief that one will act in a right, proper, or effective way 
• a relation of trust or intimacy; reliance on another’s discretion; support especially in a legislative body
• the quality or state of being certain 

 The last definition is the one related to the subject of confidence marking. The same words are found in the The American Heritage® Dictionary of the English Language, 4th Edition (retrieved October 26, 2007, from Dictionary.com website), the Merriam-Webster Online dictionary (retrieved October 26, 2007) and the FreeDictionary: which add to the state or quality of being certain the notion of freedom from doubt.

• Synonyms most often cited: confidence, assurance, aplomb, self-confidence, self-possession. These nouns denote a feeling of emotional security resulting from faith in oneself. 
• The Merriam-Webster adds: a state of mind or a manner marked by easy coolness and freedom from uncertainty, diffidence, or embarrassement.
  • confidence stresses faith in oneself and one’s powers without any suggestion of conceit or arrogance <the confidence that comes from long experience>. 
  • assurance carries a stronger implication of certainty and may suggest arrogance or lack of objectivity in assessing one’s own powers <handled the cross-examination with complete assurance>. 
  • self-possession implies an ease or coolness under stress that reflects perfect self-control and command of one’s powers <answered the insolent question with complete self-possession>. 
  • aplomb implies a manifest self-possession in trying or challenging situations <handled the reporters with great aplomb>.

Regarding confidence marking the meaning of confidence definetelly recovers the state or quality of being certain. But the difference between confidence and assurance is to be remembered because of the relation between assurance and certainty.

Confidence weighting or Certainty Marking?

or from confidence weighting to certainty-based marking

In a previous post Robert Ebel’s definition of confidence weighting was presented: answers are associated with certainty of the answer correctness. Ebel uses weighting which in his definition is associated with a mode of scoring. Gardner-Medwin (see the post here) uses marking, which is related to words like evaluation, grade. Because of the use of “marking” both confidence or certainty are associated with a mark scheme.

Ebel’s objectives for using confidence weighting are related to minimize score by guessing (“to reduce the chance error component on a test score”).

Gardner-Medwin’s objectives range

• from “risk losing marks if wrong” which is related to words like tariff, mark scheme, reckoner,…
• to “think about how reliable your answer” which is related to metacognitive judgment.

I’ve already mentioned that it is my opinion that in evaluation, degrees of certainty can be successfully used without being associated with reckoner or tariff. But I must add that mark scheme might interfere with honest expression of confidence using degree of certainty. In a survey on 349 students, 52% admitted that the were using one specific strategy, some always, some time to time, to choose the degree of certainty given with an answer instead of giving an honest expression of how reliable the chosen answer was. This survey, with other facts, suggests that using degree of certainty with a mark scheme might bias the honest expression of confidence or doubt associated with an answer.

Depending of the objectives linked with the use of degree of certainty a careful choice must be made regarding he fact of using or not using a mark scheme, tariff or reckoner because of possible bias using a strategy to answer instead of honest expression of confidence.