Level II Evaluaiton 3: Level II Evaluation: Constructing Your Own Tests

Level I I Evaluation 3Here we will consider the points to consider when you are creating your own tests.

  1. Define WHAT you want to measure: In Level II you are dealing with knowledge. It is simple to design tests that assess knowledge. However, there are several factors other that affect the validity of the test as we shall discuss here.
  2. Assessments have to relate to the learning objectives: The assessments/ tests have to directly relate to the learning objectives of the training. If they do not, it may mean that either the test is not valid, or the training design in not valid. The training design and evaluations have to follow the learning objectives of the training program.
  3. Design two or more formats of the test: You want to test before the training and after the training. Since you do not want to confound your results because of memory effects, it is advisable that you do not repeat the exact same test. Sometimes you may have to ask the same questions, especially if it has to deal with learning of a basic concept, but in other cases, you need to design two or more forms of the same test
  4. Choosing the Test Items: The items on the two formats have to be of the same difficulty level, and need to measure the same content. You cannot have a more easy/difficult pre-test than the post test and vice versa. Both the tests need to be designed at the same level of difficulty. One way that I create tests is to write successive items with the same content/ difficulty level, and then choose the alternate items to create the alternate form.
  5. Difficulty level of the test: The tests have to be a mix of difficult and easy items. Ideally the items should start off easy and then progress to more difficult questions. If everyone fails most of the items, the test is said to have a “floor effect” where scores are near the bottom- and if everyone can get through most items, the test is said to have a ceiling effect- with scores at the near top.
  6. Test Format: If these are assessing knowledge, you can create tests in different formats:
    1. Objective Type: These include tests like True or False/ Match the correct words/ Multiple Choice, etc
    2. Subjective Type: Long Answer, Essay types

Choosing the Objective Type Tests: 

Testing Considerations: Some condition when objective tests are suitable would be when you want to:

  • Measure recall items as in fill in the blanks
  • Measure if the participant can delineate small differences between or discriminate between the concepts such as in multiple choice items/ true or false items
  • Assess if the participant can recognize the correct answers as in multiple choice of any format

Time: If you have to correct the answers in a short time, then objective type tests are better than subjective type tests

Machine Correction: The objective type tests can be administered and scored using scantron and similar gadgets and therefore the answering is easy and scoring is very fast.

These tests can also be created as online tests, and administered as computer based tests.

Objective items can test if the participant has understood the basic concepts and applications of the concepts

Some limitations of Objective Testing: Objective tests do not assess the higher order thinking and analysis of the concepts. If you want to assess this using objective type, as a test designer you have to create detailed questions like case-lets and then create questions related to these, as well as the choices for answers. It takes a much longer time to design such objective tests when you want to assess higher order thinking.

Precautions while using objective tests:

  • Since the response sheets generally are scantron based, the respondent has to be very careful while answering and see that he/she is answering the right question number. I have seen respondents failing a test because inadvertently they have filled a wrong item number
  • The test designer who creates the key also has to be extremely careful, as even one error would mark an error on all the respondents’ answer sheets, which needs rescoring, or manual scoring
  • The scantron or other machines need to be in order so that they can be effectively used

Choosing the Subjective Type Tests: Some of the conditions in which subjective type questions are suitable when you want to:

  • Test if the participant can analyze a concept
  • Assess if participant can synthesize concepts and relate them to other concepts
  • So to measure higher levels of cognitive functions, use subjective type tests.

Some limitations of Subjective Tests:

One of the limitations of the subjective tests is that the participants should have the ability to write and paraphrase. Language ability is the key to answering these well

They take a long time to be completed, and some participants may not be motivated to answer, unless there is going to be some feedback or other reinforcement for completion

Precautions: Some precautions while administering the subjective type tests:

One seemingly simple precaution is the answer sheet! Sometimes the test designer provides a little space between questions. Unless otherwise specified, respondents tend to limit their answers in that short space. They are not able to explain their answers in detail. So it would be advisable to give a word limit and instruct the participants to restrict their answers to particular number of lines, or words, to use extra paper if required and so on

The answers have to be corrected by subject matter experts or by trained personnel. The criteria for correction have to be spelt out clearly, so there is uniformity in the correction of answers by different evaluators, and less chance of subjectivity

Language and other considerations have to be taken into account. If the assessment is critical, help with translation, writing help, and so on should be provided to the respondent.

General Guidelines for Test Creation and Administration:

  • Tests, whether subjective or objective, have to be related to the learning objectives of the training
  • They have to be administered right after the training, or within a reasonable period of the training. A follow up test may be conducted after a longer gap like a month
  • Respondents have to be clearly told the objectives of the testing and its repercussions if any- what would happen if the respondent passes or fails on the test
  • If the testing is done for specific reason of standardizing, or improving training, the participants have to be told that as well
  • The tests have to be designed by Subject Matter Experts along with the facilitators (if these are two different individuals)- this way, there will not be a gap between what the SME thinks the facilitator taught, and what the facilitator thinks he/ she is teaching!
  • Tests have to be designed keeping in mind the rules of good test designing:
    • They have to discriminate between good and poor performers
    • Items have to be selected carefully so that the test is consistent, valid and reliable
    • Items should not be ambiguous
    • In items such as True/ False, the statements have to be only in one direction- there should not be two choices in the same statement. For example, consider this item that is on a test after a conflict handling training: Say “True” or “False”
      • Each person has different conflict handling Styles, and each person has a unique way to handle conflicts (It would be very difficult to answer this question!)
  • Tests have to be administered in an environment that is comfortable for the participants- well lit rooms, quiet surroundings with minimum distractions
  • Pre and Post tests have to be similar – they cannot be of different difficulty levels- otherwise you cannot assess the validity of the Training. Remember, the test has to be related to the training. We are not assessing the validity of the TEST in Level II Assessment; we are assessing the validity of the TRAINING.

In this blog, we have discussed just the creation of tests for gaining knowledge, which is Level II.

Other tests, such as Performance Tests could be considered a part of Level III Evaluation, where changes in skills and behaviors is tested. We shall consider different evaluation techniques for Level III in the next blog.

blog comments powered by Disqus
Crimes in Design Webinar
Subscribe to our Monthly Newsletter