SQA has adapted requirements from E-assessment: Guide to Effective Practice for e-portfolios and for e-testing.
Standards exist that govern the delivery of online testing – BS ISO/IEC 23988:2007 Information technology – and the following information in this section is drawn from those requirements and is intended to relate more to timed, supervised e-assessments.
Technical capacity & infrastructure of institution
There are different types of e-assessment, some more demanding on centres’ infrastructure than others. It may be that not every institution is capable of applying and supporting the infrastructure needed for high stakes assessment using e-testing software, unless the institution can guarantee the levels of IT support and backup facilities required.
Similarly e-testing and other e-assessment approaches that require all candidates to undertake an assessment at the same time will make heavy demands on the infrastructure of an institution – computers, rooms, networks, invigilators - and may not be the right choice of strategy for every centre.
However e-portfolios, and other approaches to e-assessment which allow candidates to present their evidence for assessment when they are ready, should be possible for most centres if the centres are willing to invest in the necessary technology, staff development and learner support.
The supplier of the assessment engine will specify the minimum requirements for the memory and speed of computers being used by students for e-assessments, as well as the operating system needed. You should check the screen resolution and colour depth, as well as the keyboard and mouse for functionality, of all computers that will be used by students. No student should be put at any disadvantage, or given an unfair advantage, by the specification or the set-up of the equipment that they are using. Sufficient workstations are required to allow one networked computer per student, with some spare capacity. We suggest that one additional workstation for every ten students sitting the assessment would be sufficient to cater for most unforeseen difficulties.
A typical workstation needed for e-assessment would include:
It is advisable if possible to mirror the server in a separate location so that e-assessments can still go ahead if the primary server breaks down.
Servers should be backed-up daily, and it is particularly important to back up students' responses immediately after the assessment has taken place.
JISC RSC E-Responsibility Online Resource
JISC Regional Support Centres have brought together the best resources for raising awareness and planning e-safety which will support a range of roles. This toolkit has been put together as a national resource designed to be of use to develop strategy.
RSC UK - National e-Safety toolkit
BS / ISO
BS ISO/IEC 23988:2007 Information technology - A code of practice for the use of information technology (IT) in the delivery of assessments - recommends that no one should spend longer than 1½ hours working continuously at a computer. This should not pose any real problem where NAB or college assessments are involved, although it is as well to be aware that the time any student spends on the computer may include:
The infrastructure requirements for large scale delivery of summative e-assessments, particularly high-stakes assessments, are considerable, and before an institution commits to such a delivery, adequate thought must be given to contingency planning. This can cover a host of contingencies – server problems, network problems, backup and data recovery issues, desktop issues, personal profile problems, logging problems, interference from other network services, and so forth.
When running an e-assessment session It is advisable to have spare workstations in case equipment fails (approximately one spare workstation per ten students). Students must be able to restart the assessment on another computer without loss of working time, and if possible to access any responses they have already made. In the event of system failure or network outage, responses should be saved so that students can restart from where they left off when the system has been restored. Timing should be reset accordingly.
In the event of fire alarms or other emergencies, it should be possible to vacate the room with browsers still open. When they return to the assessment, students should be given the full time for completion. Details of all emergencies and the implications for the assessment should be logged.
Invigilation is the supervision of an examination for the purpose of maintaining a fair and consistent testing environment. Even if a centre has a room designed for e-assessment, it may still be necessary to invigilate summative e-assessment sessions, and more than one member of staff may be needed to ensure the assessment is securely monitored and technical support is at hand.
The publication e-Assessment: Guide to effective practice includes information on the roles and responsibilities of those involved in e-assessment, including for invigilators monitoring e-testing.
Key areas of responsibility could include:
Some subject specialists feel the balance of time spent in staff development on e-assessment should be weighted in favour of pedagogy over software, because the skills needed to use e-assessment systems are relatively easy to develop, compared with developing skills and experience in the pedagogy of assessment writing.
Both staff and students need adequate preparation to use e-assessment. For students, this will largely focus on the need to familiarise themselves with the software interface, although if e-assessment is to be used within the context of a virtual learning environment, the initial induction may have to be more substantial and ongoing. This will also be the case with deploying software such as e-portfolios.
For staff, however, a similar process of familiarisation will be necessary, but in conjunction with further training in deployment, tracking and reporting processes associated with e-assessment. An entirely different and more complex kind of induction will be necessary where staff will be required to create e-assessments. The focus here will be as much on pedagogy as on technology.
While members of a Senior Management Team may find a briefing on e-assessment adequate, teaching staff who will be using the technologies are likely to need training and support. Support staff in libraries, and in administration and technical areas, may need specialised training in how to support those using the systems.
Technical assistance is needed both before and during e-assessment sessions at a number of levels. Maintenance of computer labs, networks and servers becomes much more critical when they are being used for an invigilated assessment. Also when running e-assessment sessions and when using practice e-assessments, it may be necessary to involve the librarian or other support staff. It is advisable that:
Key areas of responsibility within e-testing and e-portfolios contexts will vary. Responsibility for e-assessment may be taken forward by one individual or indeed by different people depending on the resources available.
Guidance on roles and responsibilities in e-assessment is available by using this link.
Although e-assessment benefits many candidates, the use of an e-assessment option can create an added demand on the technical skills of learners. In some circumstances, this can create an unnecessary barrier to the assessment of knowledge and learning. This will vary from individual to individual and from group to group, and with different software interfaces.
In general, requiring learners with low levels of IT skills and confidence to use an e-assessment interface, particularly one which is not user-friendly, may be inappropriate. One strategy to help remove this barrier is to use the interface initially for formative assessment, to build learner confidence and familiarity.
Learners who are required to use an e-assessment system will need to familiarise themselves with the software interface. If e-assessment is to be used within the context of a virtual learning environment, and/or e-portfolio, the induction may have to be more substantial and ongoing.
In order to ensure that the technology doesn’t affect the outcome of any assessments, students must be familiar with the system, and comfortable with using it, before they use it for summative assessment. They should practise using the system, trying out the navigational aspects of the software and most of the question types - questions with a similar style, level of difficulty and method of feedback to those to be used in the actual assessment.
Students should be given Usernames and Passwords as required to carry out the practice assessments. Practice assessments can be conducted in the centre, either in timetabled subject time, for example during personal development time (PDT), or outwith timetabled time, such as in a library during a break. Alternatively, it may be possible for students to carry out practice assessments at home, if you feel this is appropriate, and the software supports it.
It’s important that learners are made aware, at an early stage in the use of e-testing, that the marks provided in a summative online test will be automatically assigned by the computer. You may need to modify these marks and, once this has happened, the results can be returned to the learners. You should also make learners aware that, as happens with paper-based assessments, the results will be provisional at this stage, and could be subject to internal and external verification.
Some e-assessment is self-paced, particularly in the area of formative assessment. Where it is not, the planning of e-assessment, especially planning the timing and pacing of the e-assessment delivery, must be considered.
Largely, consideration of timing issues for e-assessment will be similar to considering timing for conventional assessment delivery, but with the caveat that learners may need to be given earlier opportunities to familiarise themselves with the software interface. Pacing can be a more particular consideration for certain assessment software packages, ie those which allow you to set time constraints on the assessments at the global or individual question level.
Although online assessment, particularly objective test type questioning, may appear to be less time consuming than conventional deliveries, learners need to be given adequate time to read and check questions and answers, and it is a common fault to underestimate the amount of time needed to complete such assessments.
Research has shown that learners taking assessments on computer require on average ten percent more time to complete the assessments compared to traditional methods. You should allow for this when you are planning the assessment.
JISC TechDis is a leading UK advisory service on technologies for inclusion.
They explore and promote inclusive practices, resources and advice for learning and teaching in UK higher education, further education and skills sector, and independent and specialist colleges.
The following link will take you to an overview of the resources and publications from JISC Techdis complete with signposting to their location on the new JISC Techdis website.
A Guide to JISC Techdis Resources
TechDis Tuesdays are online fortnightly workshops organised by JISC where they:
For more information about TechDis Tuesdays, including topics for future sessions and to access recordings of previous sessions, visit the TechDis Tuesday page of the JISC website.
SQA has worked with CALL (Communication, Access, Literacy and Learning) Scotland to design digital question papers for learners who've been identified as having reading and/or writing difficulties, including learners who may sometimes require human readers or scribes. The digital papers are supplied in PDF format on CD. Learners can read the question paper 'on screen', using speech technology, where appropriate. Where the format of the question paper is a question/answer booklet, learners can use the computer to write their responses, or can ‘speak’ their responses.
Digital question papers from previous years are available for download from SQA by using this link.
CALL Scotland provides services and resources to assist with communication difficulties, particularly difficulties experienced in learning contexts. CALL Scotland can provide advice and training on creating digital question papers for use as prelims or for assessment generally.
Copies of the Profound and Complex Needs Newsletter are available in Latest Publications at Scotland's Colleges website by using the following link:
Publications from Scotland's Colleges
Some e-assessment activities will require supervision, such as taking an on-screen, closed-book test in which the learner is not allowed to access any additional resources. Other e-assessment activities may not need supervision; for example, creating a report for uploading to an e-portfolio for assessment.
See the following section on Authentication.
Authentication involves introducing and maintaining systems to establish the identities of learners and the ownership of the evidence they submit for assessment. Centres are responsible for ensuring the authenticity of this evidence.
Authentication is linked to assessors’ knowledge of learners and their work; if assessors are familiar with a learner’s level and pattern of work they should be able to spot differences or anything unusual. Assessors will often observe activities being carried out and know that evidence produced is learners’ own work. However if evidence is produced without supervision, especially using resources acquired online, assessors may need to confirm that the evidence was generated by the learner, to ensure malpractice doesn’t occur.
Centres should have authentication strategies incorporating combinations of tools and approaches such as:
When running e-assessment sessions designed to be held in controlled conditions, the following procedures are recommended:
Internal verification is the process used by centres to ensure that internal assessment has been carried out consistently within the centre and that national standards have been applied.
This involves checking assessment evidence and other assessment resources. Internal verification is checked as part of the External verification process.
If you use an online internal verification process to sample the assessment resources and record internal verification activity, this will allow more flexible approaches to external verification, such as E-verification.
You may find the information on the Education Scotland website about Quality assurance and moderation useful.
Curriculum for Excellence
The following publication in the Curriculum for Excellence Building the Curriculum 5 - A Framework for Assessment series contains useful information about quality assurance and moderation.
Understanding, Applying and Sharing Standards in Assessment for Curriculum for Excellence: Quality Assurance and Moderation
External verification is the process used by SQA to ensure that internal assessment within and across centres has been carried out consistently and that national standards have been applied. It involves checking assessment evidence and other assessment resources, which can be in paper or electronic form; and also checking centres’ Internal verification processes.
External verification is carried out by SQA External Verifiers by visit, by post or through central verification. Where a centre is assessing learner performance using electronic media, such as e-assessment, External Verifiers could verify either by visiting the centre or, in some circumstances, by accessing the electronic assessment resources remotely. This is sometimes called E-verification.
As with external verification of traditional assessments, verification of e-assessment may also require the centre to provide information about the assessment event, such as the location; conditions and arrangements; authentication of learners and their evidence; results and reporting.
SQA has produced a Guide to External Verification, which can be accessed by using the following link:
External Verification: A Guide for Centres
External Verifiers can in some circumstances verify by accessing a centre’s electronic assessment systems and resources remotely. This is sometimes called e-verification. With a centre’s agreement, External Verifiers could gain access to its systems online, from the External Verifier's home, awarding body premises or another mutually agreed location, for the purposes of carrying out e-verification.
Centres should have documented procedures detailing access arrangements for External Verifiers, including access arrangements to allow e-verification, where the centre has agreed to this. If access to the centre’s system is not possible remotely, traditional external verification will be used.
SQA has produced guidelines for FE staff and external moderators on e-moderation. SQA e-Moderation: guidelines for FE staff and external moderators (2004)
They have also published Case Studies on the moderation of e-assessments. Moderation of e-assessment : Case studies (2006)
Verification of an e-assessment involves the inspection of various electronic documents, and possibly also paper documents. If the e-assessment is an online, automatically-marked test, these would include:
An external verifier carrying out an e-verification may need access to other documentation covering: the assessment event(s) - information relating to the location and conditions and arrangements used; the authentication of learners and the learners’ evidence; results and reporting. To be valid as evidence for external verification, these documents should be date and time stamped.