Remote testing: discussing options, advantages and disadvantages

Now that the final test in primary education and the central exams in secondary education have been cancelled, we are missing important information about the learning process of pupils and what they learn at home. Information that is not only important for the closing years and, for example, a good transfer to secondary and higher education, respectively. But also to know which subjects and which students will have some catching up to now, now and later. In this contribution, Melanie Ehren and Judith Conijn discuss the possibilities of remote testing, and the advantages and disadvantages of the various options.

The importance of testing

Now that the final test in primary education and the central exams in secondary education have been cancelled, we are missing important information about the learning process of pupils and whether and what they learn at home. Information that is not only important for the closing years and, for example, a good transfer to secondary and higher education, respectively. But also to know which subjects and which students will have some catching up to do, now and later. CITO, the organisation responsible for final testing in primary schools, rightly notes that tests are not intended as a “final reckoning” and has decided against having pupils sit tests at home now that schools will be closed for an extended period of time. At the same time, however, it also notes that tests are necessary to monitor the development of pupils and therefore to tailor the education to pupils in the best way possible.

How can we test students remotely?

What are the pros and cons of the various options?

Six essential features of testing

A good test satisfies the following six essential characteristics (see Drenth & Sijtsma, 2005):

1. Efficiency: a test must be efficient. After all, by taking a test you want to quickly gain insight into the knowledge and skills of a student, rather than making an estimate based on all available data and observations of a student.
2. Standardisation of the decrease: standardisation is important when we want to compare the performance of a student with that of others. Standardisation means that how the tests are administered (e.g. test material, time limits) is the same for all pupils. Standardisation prevents differences in test results between students attributable to differences in how the tests are administered or the instructions given.
3. Frame of reference for interpretation: it is also desirable that a frame of reference is available to interpret test results. Is a given result above or below average compared to what one would expect? Does the result correspond to the minimum level of required knowledge? A frame of reference ensures that figures also have a relative meaning: an “8” means that the learning material has been better absorbed than a “7”.
4. Objectivity: the test score must be independent of the person who assigns the score. This requirement is irrelevant for automatic online scoring, but very relevant if students are graded by different assessors (e.g. for oral tests or essays) and their test result is interpreted using the same frame of reference.
5. Validity of the result: the validity of a test can be described as the extent to which the test fulfils its purpose. For tests in education, validity means that the test result gives a good picture of the extent to which a student has mastered the material tested.
6. Reliability: high reliability is achieved if the repeatability is good. The question then is: would the test result yield the same score if these were taken again – under similar circumstances?

These characteristics are particularly important for summative tests used for decision-making, such as failing or passing, selection for a specific follow-up course or whether to offer extra support.

However, remote testing of pupils at home creates specific challenges around each of these characteristics. Especially in summative testing, fraud (getting help from parents, looking up answers online, etc.) has consequences for important decisions and it is therefore important to ensure that tests are properly administered in line with these six characteristics. Below we discuss the options for taking such summative tests online.

Advantages and disadvantages of different forms of online (summative) testing

• Written tests in small groups on location. In this form, students are tested at multiple locations (for example, in schools or in a community centre), whereby the guidelines of the health authorities are followed concerning the maximum number of people per meeting and a minimum distance between pupils and assessor. If all students are tested at the same time, one version of the test can be used. If that is not possible, several test versions must be developed. Here, most essential characteristics for good testing can be met, except efficiency; this form is expensive and requires the necessary national organisation if many students take a test at different locations at the same time. If several versions of a test have to be produced, ensuring the comparability of the different test versions will come at the expense of efficiency.
• Take written tests at home, with invigilation by means of a webcam and using special software for ‘online proctoring’. Such software makes it possible to monitor the online movements of a student and whether he or she searches for answers online or collaborates with someone who provides prompts with the right answers. Here, however, careful thought must be given to privacy, whereby parents and pupils must, for example, give permission for the use of a camera. Other options are testing with multiple choice questions in which a question bank is used, and students are offered different versions. Students would therefore not all see the same questions and cannot pass on their answers.
• Online oral tests. In this form, a teacher administers an individual test with a pupil, for example, by using Skype, Google Hangouts or MS Teams. The test largely meets the above characteristics if the condition under which the oral test is taken is properly regulated. The room where the student is located must be free from disturbing influences (e.g. noise) and the pupil must be informed in advance about what aids may or may not be used. The comparability of scores of different students can be further increased by starting from a standardised set of questions, an assessment section, a fixed amount of time for the test, whereby the test is administered by a teacher who preferably does not also teach the pupil in question. However, a major drawback of this form is its inefficiency: it is very expensive and time consuming to organise a form of individual oral assessment for all students. In addition, this type of test should also consider possible fraud. Camera surveillance could help here, as could removing reproduction questions from the test.
• Assignments as a replacement for a test, for example giving a presentation or writing an argument. This form of assessment has the advantage that different learning objectives can be tested simultaneously and that students can show creative skills. On the other hand, a relatively large number of measures must be taken to guarantee the objectivity of the assigned mark, for example by drawing up a proper assessment section for the assessor. Standardising the assessment is also a challenge: to what extent have the pupils received help from parents or fellow pupils, for example? Collaboration can be limited by specifying a fixed start and end time and limiting the duration. Another possibility is to observe students ‘live’ while working on a text in Google Docs. For short assignments, camera surveillance could also be used to determine whether a student is working alone.

An important condition for the validity of the various innovative forms of online assessment is that pupils have a good online connection at home, are familiar with the form of the assessment and have already practiced this. Where this is not the case, or where it varies for different groups of students, their score would be influenced by factors that are not related to their understanding of the material.

The motivational effect of tests

And finally, and perhaps most importantly … we must also remember that testing is not an end in itself, but is always a (small) step in the educational career of a student. A test therefore has a (de) motivational function for the future. With each choice, we must therefore also consider how the chosen form is perceived by pupils and parents, and motivates them to continue to learn. Doing an assignment or an online test may lend itself better to this than a high-pressure written exam with online proctoring.