E-assessment

The website should be of interest to senior and operational managers in centres; practitioners who wish to create or deliver e-assessments; centre administrators; invigilators and SQA verifiers and other appointees...Find out more about the website

It’s arranged in four sections:

In Suitability you'll find information on the benefits of e-assessment and the different types of e-assessment available. This should help you think about the most suitable approaches for you, the centre and your learners.

In Preparation you'll find out about existing e-assessment resources and there’s advice on creating your own e-assessments, on quality assurance and on supporting staff and learners.

In Delivery there’s advice on policies that could help you deliver e-assessment, and information on delivery platforms and systems.

In Evaluation there’s advice and tools to help you measure the outcomes of e-assessment and suggestions for informing future practice.

What benefits will e-assessment bring?

Supporting learning and teaching

Time available for teaching

E-testing increases the time available for teaching. This is mainly because most questions are automatically marked by the system and the results are returned immediately to learners. The time spent in marking and returning learners' work is greatly reduced. Re-sits can often be organised on an individual basis, freeing the teacher or lecturer to concentrate on teaching the rest of the class.

Time for teaching 

Analysis of learners’ responses

E-testing systems have the ability to collate and instantly analyse learners’ responses to online questions. This means that reports can be generated about the performance of a single learner, a class, or a number of classes.

Report 

 

Access & inclusion

E-assessment -v- paper assessment

Some learners with additional support needs find e-assessment preferable to traditional pencil and paper assessments. It should be possible for learners to modify the screen display to suit their particular needs.

Hand writing on paper

Assistive technologies

Most e-assessment systems are capable of supporting assistive technologies commonly used, such as the screen reader Jaws.

Supporting collaboration and networking

Learners can use social software technologies, such as wikis, to collaboratively generate and edit content on a central website. This could include learners collaborating on research or to develop strategies for different tasks or to solve problems. Social networking tools, such as blogs and forums, allow learners to share views and information and to communicate easily and regularly. Some ideas on using these tools for assessment can be found in the Section Supporting assessment with Web 2.0.

Feedback

Feedback through e-testing

In online testing, it is possible to analyse an individual learner's response to individual questions, and to use this to give feedback.

If a lot of the responses to a particular question from learners in the same teaching group show the same misunderstanding, this can be used to amend teaching. Also you’ll be able to identify the learners who need help.

Someone giving feedback

Instant feedback

Online testing which uses automated marking can be incorporated into the delivery of a Unit and can present feedback to learners immediately.

Feedback through e-portfolios and social software

Other e-assessment tools such as e-portfolios, blogs and wikis can be set up to notify teachers and lecturers when learners have created content which they may wish to review, or have submitted work for formal assessment. These systems allow you to create and deliver feedback to learners in a secure environment.

Feedback being given                Web image

Supporting assessment and marking

Streamlines marking load

In conventional assessment, marking can vary between markers; however with e-testing, computers will mark the same responses in the same way every time. This also streamlines the marking load for teachers and lecturers.

Demented marker 

Free text questions

Although free-text / extended response or essay questions can be handled by most assessment engines, they are not usually marked automatically. Instead learners type answers in text boxes and the assessment system stores the responses for scrutiny by a teacher or lecturer. Answers can be printed out if necessary for marking.

Hands typing 

Supporting assessment with Web 2.0

Web 2.0 describes the range of web services that collectively represent the second ‘version’ of the world wide web. These include collaborative and social networking facilities. These tools can be used to aid and measure learning and to find, capture, generate, store and present evidence for assessment.

‘Assessment 2.0’ is the use of Web 2.0 services to generate assessment evidence. Some of the ways in which Web 2.0 technologies can be used in the assessment process can be seen by using Link opens in new windowthis link.

New assessment techniques

E-assessment can measure things that we have not been able to assess before. For example, by using social software (wikis and blogs) the contribution made by individuals to a group assessment task can be assessed with much more confidence.

Security, cost and administrative benefits

Storage and transfer of data

Security will be improved though the gathering and storing of assessment data and results online.

Results can also be collated easily and provided to learners online or by e-mail.

Documents flying about        An email

In addition, e-assessment systems often offer secure, automatic transfer of assessment results directly into management information systems, saving time and reducing clerical errors.

Computers networked

Online -v- paper

There is a significant reduction in printing of paper assessments leading to a reduction in printing costs.

Printer

The associated administration involved in storing papers securely, as well as distributing papers and collecting responses is also eased. With robust e-assessment systems, the chances of a learner's responses going missing should be reduced.

Marking exam paper

Software security features

There are many measures that can be taken to improve security during e-assessments. Some possible security breaches that can be prevented with the help of the software are:

  • Loss of learners' responses
  • Opportunities for copying
  • Communication with others and access to other information sources

 

securitysmall.jpg

Validity and authentication

Validity is about the effectiveness of an assessment to measure what it is supposed to measure. An important aspect of validity is the realism of the activities learners are asked to carry out; the more realistic the activity, the more valid the assessment. Social software and Web 2.0 approaches generally can improve realism by encouraging learners to use the technologies they use in their everyday lives, to capture and generate authentic items for assessment.

Authenticity relates to the ownership of the evidence; whether it is actually produced by the learner or someone else. A learner’s e-mail messages, forum contributions and blog posts can provide an assessor with information on the learner’s knowledge and skills, and so alert the assessor if the learner’s work changes in quality. Technologies such as VoIP permit remote oral questioning of learners to verify that they understand the evidence they have submitted for assessment, which gives a good indication that the evidence is the learner’s own work.


 

Although written in 2002, the paper Principles of Assessment published by the CAA Centre at the University of Luton defines validity and considers the different types of validity related to assessment.


 

Many online tools, such as social software and other Web 2.0 tools, can be used to aid validity and authentication.

What is Web 2.0?

The learning potential of Web 2.0 services is well known. This paper explores their assessment potential.


 

Although the use of technology can increase opportunities for plagiarism, it can also provide ways to detect plagiarism. Essays submitted electronically can be analysed using tools which will highlight inconsistencies between different sections of the essay. This means that if other peoples’ work has been incorporated into the essay this can be detected automatically.

At the University of the West of Scotland, academics from a number of disciplines have implemented the use of Turnitin originality checking and plagiarism prevention software.

Quality enhancement

E-verification

With e-assessment, e-verification (or remote verification) may sometimes be possible. Benefits of e-verification include:

  • Centres can arrange for External Verifiers to securely access their systems to view assessment evidence and quality assurance data, no matter the distance between the External Verifier and the centre.
  • With e-verification there is less likelihood of lost data than with central/postal verification and potentially less disruption to centre activities than there is with verification visits.
  • It may also be possible for External Verifiers to request some items prior to the verification ‘event’, allowing the event to focus on other items or issues.
  • For an awarding body, e-verification offers potential savings in External Verifier travel and subsistence costs, but the approach would be used to complement, not replace, verification visits and central verification.

 


 

From 2008-2010, SQA worked with centres and External Verifiers to pilot e-verification. This project was initiated because SQA was aware that centres and learners were increasingly using e-portfolios and other electronic means to manage assessment evidence and they wanted to explore how SQA could make use of e-verification as part of its quality enhancement process.

The report on this pilot is available by using the following link:

Link opens in new windowPiloting e-verification 2008-10

Standardisation of assessment

E-assessment, and electronic evidence in general, facilitates the online sharing of assessment evidence and resources and so facilitates professional discussion on standards and assessment decisions. This can benefit centres by supporting new models of internal verification – cross-centre; authority-wide; company-wide. It could also benefit awarding bodies by supporting new approaches to external verification – e-verification; regional verification; online central verification.

e-verifiers at computers and on the telephone 

Linking formative & summative assessment

Formative assessment used as practice

Online formative e-tests can be used for practice and to reinforce learning while preparing learners for the summative assessment. At the same time, the learners become familiar with the assessment engine.

Formative assessment at computers

Randomisation

Assessments that incorporate randomisation and selection from a substantial bank of questions can be used repeatedly by learners for formative assessment without losing the confidentiality of the questions for use in summative assessment.

Formative & summative assessment in technology-supported environments

When learners are working with e-portfolios, or in technology-supported environments (such as VLEs), the distinction between formative and summative assessment may be less clear than it is traditionally. Online learning environments provide access to learning resources, and communication tools and forums. Learners are able to work largely independently, or in collaboration with other learners, benefitting from online feedback on their formative work from peers, teachers/tutors and Assessors. This feedback is likely to become an integral part of the assessment process, encouraging learners to re-draft some of their formative work and promoting reflection and self assessment.

This means that formative work can evolve to become summative evidence, as learners work incrementally towards the point when they judge that their e-portfolios meet the assessment criteria. Once items are submitted for summative assessment they should be ‘locked’, preventing learners from making further amendments.


 

JISC

Use this link to find out how JISC shows that e-portfolios can be used to support assessment.

Link opens in new windowE-portfolios supporting assessment


 

Higher Education Academy

Use this link to find out what the Higher Education Academy says about the role of e-portfolios in formative and summative assessment.

Link opens in new windowThe role of e-portfolios in formative and summative assessment


 

The PassIT Research Project

Use this link to read about the PassIT Research Project on formative and summative assessment.

Link opens in new windowPassIT Research Project

Motivating learners

Positive impact on learning

E-assessment can have a positive impact on learning. By increasing the frequency of assessment, and improving feedback, we can improve learners’ motivation. Learners who prefer to do coursework could be motivated by the increased level of feedback that online assessment can facilitate.

Hooded male at computer


 

PassIT Research Project

You may find this link to the PassIT Research Project useful.

Link opens in new windowPassIT Research Project

Media-rich elements

E-assessments can incorporate media-rich elements that include video, sound, animations, simulations and interactive graphics. These elements are designed to be user-friendly and encourage learner participation.

Surfing the internet

What types of e-assessment could I use?

Types of assessment

Assessment-creation software for e-testing

There is a vast range of assessment-creation software offering a huge variety of different functions, so it is difficult to find much that is common to all. Grouping such software, it is possible to organise them into two categories. Firstly, quiz authoring tools and, secondly, assessment engines.

Quiz creation tools create one-off quizzes, or groups of quizzes, often in html, which can then be used as stand-alone assessments in a browser environment, or through a learning delivery platform such as a VLE.

Assessment engines offer the functionality to author quizzes, but in turn will also offer an online environment in which to carry out the testing, or will integrate with a learning environment to deliver the same. The key feature of assessment engines is their ability to track, record and report on learners who carry out the assessments.

E-testing

Online, on-screen tests are often made up of multiple choice and multiple response-type questions, but can also include other question types, such as sequencing and hotspots. Short answer questions, which allow learners to type in responses, could also be included.

The following is a list of the types of questions that you are likely to meet in an e-assessment.

  • Multiple choice
    With multiple choice questions (MCQs), learners are required to choose one out of several responses to a question. Multiple choice questions can also be authored as drop-down (sometimes called pull down) list questions.
     
  • True/false and yes/no
    True/false and yes/no questions are simply multiple choice questions with two possible responses. Such questions allow the rapid assessment of large amounts of material. There is very little educational benefit in using these questions since a score of 50% is statistically likely in an assessment containing only true/false or yes/no questions if the learner knows nothing about the subject. Also, it can be difficult to write stems which have two such absolute responses.
     
  • Multiple selection or multiple response
    Multiple selection or multiple response questions are similar to MCQs but they can be more difficult to answer correctly because the learner can choose one or more correct responses. The convention is to use a square checkbox rather than a round radio button where more than one response can be selected. This is a function of the software, and is not something you can normally alter.
     
  • Matching
    This question type can be used when learners are required to match two related items or concepts. Matching questions are variations of MCQs.
     
  • Sequencing/ordering
    Sometimes learners are required to know a sequence or order. They will normally be required to drag the responses into the correct order.
     
  • Hotspot
    Hotspot questions are useful when a learner should to be able to identify or interpret parts of a picture or diagram.
     
  • Judged mathematical expression
    All assessment engines can handle questions requiring a number or a mathematical expression as a response.
     
  • Short answer
    There are many types of automatically-marked questions that require a short textual response - one word, or at the most a short phrase. Fill in the blank, gap fill and word match are all varieties of this type of question.
     
  • Free text response or essay
    Free text response or essay questions are ones where learners have to enter short essay-style answers that can be marked by a teacher or lecturer later.

 


 

JISC infoNet

JISC infoNet aims to be the UK's leading advisory service for managers in the post-compulsory education sector promoting the effective strategic planning, implementation and management of information and learning technology.

In one of their online infokits, 'Effective Use of VLEs: e-Assessment', JISC notes that there are three main types of objective questions which can be used which include students making a choice (single or multiple) inputting text (words, numbers or formulae) or manipulating information on the screen (marking relevant areas, moving items or drawing graphs or diagrams).

JISC's list of assessment question types can be found by using the following link:

Assessment Questions

E-portfolios

E-portfolios are similar to paper portfolios but contain only digital items. Most are web-based so the learner is able to work online to create, store and arrange their items.

One of the key benefits of e-portfolios is that they encourage learners to take ownership of their work and to actively participate in the process of assembling their e-portfolios and using these to present their work. Although some candidates may need support in doing this initially, in the longer term it can lead to enhanced learner motivation and more rapid attainment of qualifications.

Another benefit of e-portfolios is that they allow the learner to include (or link to) evidence which spans subject boundaries, or experiences which take place outwith centres. This is important for interdisciplinary projects, and for qualifications where key pieces of evidence may come from employment or placement environments. The ability to add digital photographs, scanned images and video or audio files to e-portfolios can make these experiences even more authentic.

If used for assessment, the e-portfolio generally holds evidence for a particular unit or course. The learners, teachers/tutors and Verifiers can access the e-portfolio remotely and so the processes of review, assessment and verification are streamlined.

Sfw-sg.jpg

Social software

Social software includes communication tools for capturing, storing and presenting written, audio and video information, and also interactive tools which handle communication between users. Examples of social software include blogs and wikis.

Blogs

A blog is a type of website, usually maintained by an individual who updates it with commentary and descriptions of events. Most blogs are interactive, allowing others to leave comments. It is this interactivity that distinguishes blogs from static websites. A typical blog combines text, images and links to other relevant web pages and media. Many blogs function as online diaries.

When used for assessment, blogs can be valuable for capturing learners’ reflections and evidence such as learner personal statements; assessor observations; witness testimonies provided by others. All of this can help with authentication of evidence.

Wikis

A wiki is a web site that allows users to add and update content on the site using their own web browser. This feature means that wikis tend to be created collaboratively by the site users.

This is valuable when a wiki is used for assessment, because it can capture the contributions of different members of a group who are working collaboratively on a document related to a project.

Electronic voting systems

An Electronic Voting System (EVS) usually consists of a computer, a projector, one or more receivers and remote control handheld electronic transmitters, known as clickers. Learners are asked questions, shown as multiple-choice questions in a presentation, and they use the clickers to select their preferred options. The handsets transmit this information to the computer software which produces histograms or bar charts of the results.

When used for assessment, different types of questions can be used depending on the purpose of the assessment. Even the basic Yes/No or True/False question types can be used to test recall; to assess learners’ understanding of issues or to test if they are able to recognise solutions or appropriate approaches for solving problems. Questions can also be broken down into smaller stages.

Games-based learning and assessment

Games-based assessment uses computer gaming technology and techniques for assessment purposes. Games can be engaging for learners but can also be used to embed learning outcomes. When used for assessment, different types of games (eg quiz, action, strategy games, role-play) can be used to assess different learning outcomes such as memory/recall; behaviour;  functional competence and so on.

GamesSpace screen shot - player in virtual environment

Mobile learning and assessment

Increasing interest is being shown in delivering assessment through mobile platforms, given that many learners own one or more devices offering the capacity to deliver assessment content. Such devices will include mobile phones, wireless enabled mp4 players, tablet pcs and others.

The key advantage of authoring for and delivering to such devices is the ability to provide learners with assessment in an appropriate context – such as a workplace – and on demand. In creating assessments for mobile devices there are two key areas to be mastered, the authoring of content appropriate for a mobile device, and the delivery of that content, either by pushing it out to learners or by providing a gateway for access.

MyExam Screengrab

JISC e-assessment glossary

The JISC e-Assessment Glossary includes over 700 items relating to the processes involved during the design, presentation and recording of candidates' e-responses to assessment stimuli. It has been produced for use by awarding bodies, teachers, lecturers, practitioners and administrators across all types of educational institution and across all sectors, as well as software developers and providers.

Although this glossary is not fully up-to-date (it doesn’t include social software, electronic voting systems), it is still a very useful resource.

Pedagogy

Learning and teaching approaches

The starting point for selecting the types of e-assessment should be pedagogy - the learning and teaching approaches that are suitable for the subject. Consider which forms of assessment best support learning in conventional situations and how these approaches could be better designed or enhanced with technology. Most assessment approaches can be e-enabled.

Students and teacher in park 


 

Guidance on Pedagogy

Further guidance on pedagogy can be found in the 12 Principles of Good Assessment and Feedback which were produced by the Academic Policy Committee of The University of Strathclyde in Glasgow and agreed by Senate in 2008. These principles are intended to help academic staff who wish to promote student engagement and self-regulation in learning. They provide a guiding framework and they can be adapted to different disciplinary contexts.

Link opens in new window12 Principles of Good Assessment & Feedback (The University of Strathclyde)

Can all subjects be e-assessed?

Objective questions, such as multiple choice questions (MCQs), would not be appropriate for all subjects or assessments. They are effective in assessing factual recall. It's more difficult to use them to assess higher-order skills such as creativity, communication or synthesis.

That's not to say it can't be done at all: learners’ understanding and problem solving abilities could be assessed to some extent using these approaches but may be assessed better when the learners have to recall and enter information rather than when they are presented with a selection of possible responses.

What is important is that an e-test is comparable to a 'traditional' test, not that it is an on-screen copy of the pencil and paper test. The questions chosen should assess the same knowledge and/or skills with the same degree of difficulty. For example an e-test may use a 'matching' type of question which is not available in a traditional test environment.

Some software can be programmed to automatically mark short answer responses but complex qualitative or reflective responses would be captured and marked by human markers.

Learning styles

Some educational psychologists believe that different learners have different learning styles, such as visual, auditory or tactile approaches to learning, though in recent years learning style theories have also been challenged.

For more information on the history and theories of learning styles see Link opens in new windowhttp://en.wikipedia.org/wiki/Learning_styles

Using a variety of assessment and e-assessment approaches is likely to be the best way to suit the widest group of learners. The three categories of learning style most commonly recognized are:

  • Auditory: learners learn through listening
    Auditory learners learn best from things they hear. They benefit most from lectures, discussions and listening to others. The theory suggests that written material may have little meaning to these learners as they interpret information by listening to the characteristics of the voice. They often benefit from reading aloud, summarizing orally and making audio or video recordings.
     
  • Visual: learners learn through seeing
    Visual learners retain knowledge best by what they see. The theory suggests they benefit from observing a teacher or lecturer's body language and facial expression. They tend to prefer sitting at the front of the room and may think in pictures. They retain more knowledge from the use of charts, graphs, diagrams, spreadsheets, illustrated text books, on-screen presentations, video, flipcharts and hand-outs. Computer programs and other types of visual aids can enhance comprehension. These types of learners often prefer to take detailed notes during a lecture or discussion to absorb the information.
     
  • Tactile: learners learn through moving, doing and touching
    Tactile people learn best through hands-on activities and from moving around while learning. They may find it hard to sit still for long periods and benefit from moving around the room while reading or listening. Computers can help tactile learners by allowing them to use their sense of touch.

Purpose of assessment

It’s important to bear in mind the purpose of the assessment.

Diagnostic Assessment is carried out before or after the learning activities take place to ascertain the level of a learner’s (or group’s) knowledge or skill.

Feedback form

Formative Assessment is carried out as part of teaching and learning. This is often known as assessment for learning because the information gathered by the assessment is used to adapt the teaching to suit the learner’s needs.

Assessment for learning

Summative Assessment is carried out at the end of a learning programme or at specific points during it to judge or mark the learner’s achievement. This can affect the level of award the learner achieves and/or allow progression to the next stage of learning. 

A grade


 

JISC infoNet

JISC infoNet aims to be the UK's leading advisory service for managers in the post-compulsory education sector promoting the effective strategic planning, implementation and management of information and learning technology.

In one of their online infokits, 'Effective Use of VLEs: e-Assessment', JISC looks at the different types of assessment - diagnostic, formative and summative.

The following link will take you to JISC's Methods of Assessment:

Methods of Assessment

Pedagogy of assessment writing

Writing good quality assessments, which accurately assess learning, and which do not involve unnecessary distractions or offer unintentional clues, can be challenging. In particular, writing assessment items which test higher levels of learning requires time and thought.

You may find information in the following two SQA Academy Courses useful. If you don't have an SQA Academy account, you can login as a guest to these courses from the login page.

Link opens in new windowAn Overview of Assessment Principles

Link opens in new windowWriting Objective Test Questions


 

Levelling

Ensuring that assessment items are accurately and equally levelled requires thought and care. In particular it is possible to unintentionally introduce elements which affect the degree of difficulty of an assessment item. A simple example of this would be inappropriate distractors, which affect the difficulty of a question, or question stems with poor phrasing, which introduce ambiguity.


 

Bloom's Taxonomy
We tend to use Bloom’s Taxonomy of educational objectives to focus on the skills and abilities we want to assess.

A Link opens in new windowrevised Bloom’s was published in 2001, this time focusing on learning objectives.

Andrew Churches has developed a Digital Taxonomy, suggesting verbs, activities and technologies, appropriate to the digital age, across the different areas of Bloom’s. This Link opens in new windowPrezi by Joshua Coupal provides a summary of Bloom’s and the Bloom’s Digital Taxonomy.

You can find resources from Andrew Churches at Link opens in new windowBloom’s Digital Taxonomy.

Timing

There are two timing considerations with e-assessment.

The first relates to the demands e-assessment can make on learners who are required to use self-paced assessment. It may be possible for some learners to choose when to take the test but other learners will rely on the tutor to set the timing and pace of assessment.

Secondly, the issue of timing may be an institutional one. If assessment of learning requires large scale simultaneous testing – all candidates sitting the same test at the same time - then there are infrastructure and support requirements to be considered.

Also relevant is the section on Requirements – networks; computers; rooms in 'Preparation'; 'What other things do I need to think about?'.

Feedback

Good quality feedback is an essential element of e-assessment, in particular where it is to be used formatively. This means feedback at the general level, but also for each individual response in objective tests. If the learner is to gain value from taking e-assessment tests formatively, then there must be feedback which explains not just that a particular choice is wrong, but also why it is wrong.

Adequate feedback, and opportunities to review and respond, have to be planned into the delivery. Timing of the programme overall should make allowance for students to absorb and respond to feedback, whether individually or in a classroom situation.

What existing resources are available to me?

Online testing

SQA SOLAR assessments

SOLAR Project Banner

SOLAR is SQA’s online assessment tool that provides summative and formative e-assessments. Assessments within SOLAR are made up of questions which are generated dynamically from an itembank. The summative assessments already available have been created and packaged by SQA, and quality assured by External Verifiers. The same bank of questions can be used to create formative assessments, and these can be delivered through the SOLAR e-assessment system in the same way as the summative e-assessments. These e-assessments cover qualifications at all levels from National Certificate to Higher National Diploma.

The summative e-assessments are pre-verified — so centres know that quality is assured. This provides real benefits for both learners and learning providers.

The formative e-assessments, which are quality assured and available to support many subjects, provide detailed feedback for learners. Learners can access the assessments 24 hours a day, 7 days a week using a browser and the SOLAR Open Assess portal.

Each delivery of an e-assessment to a learner is dynamically generated by the system, based on pre-specified rules.

The picture below is a link that will take you to a web page about SQA SOLAR assessments.

openassessimage.jpg


This link will allow you to try SOLAR formative assessments which include most of the question types mentioned in Suitability; What types of e-assessment could I use?; Types of assessment; E-testing.
Link opens in new windowSOLAR formative assessments

National Assessment Resource (NAR)

NAR is Scotland’s National Assessment Resource for Curriculum for Excellence and is a joint development between Education Scotland, SQA and the Scottish Government.

It will support assessment in practice and will support practitioners in developing a shared understanding of standards and expectations for Curriculum for Excellence and how to apply these consistently.

Link opens in new windowThis link will take you to the NAR website. You will need a Glow account or SQA credentials to access the National Assessment Resource.


 

If you do not have a Glow account or SQA credentials to access the National Assessment Resource, the following links will provide you with more information about the NAR.

NAR, Scotland's National Assessment Resource for Curriculum for Excellence

About NAR: Guide to the National Assessment Resource

About the National Assessment Resource: Video Guides

Video Bank: Finding Materials on NAR

Bank on Success

Making use of e-assessments has a number of benefits for both the learner and lecturer and forms a critical part of the learning process.

Bank on Success, a project focusing on e-assessment, is currently being piloted at the JISC RSC Scotland South & West.

This link will take you to the JISC Regional Support Centre Scotland website where you can preview the assessments produced.

These resources were developed for Moodle 1.9 and the last version of Blackboard but you should still be able to view them. If there was interest, JISC may consider updating them.

As they were developed as part of Bank on Success they were not quality assured; they represent resources developed by practitioners as part of that project.

NLN materials

Funded by the Skills Funding Agency and its predecessors to support learners and organisations in the post-16 sector, the NLN materials represent one of the most substantial and wide-ranging collections of e-learning materials in the UK.

The NLN Materials are small, flexible 'bite-sized' episodes of learning. They are not whole courses but are designed to support a wide range of subject and topic areas. The materials adopt a range of approaches involving some knowledge acquisition, a chance to practice and absorb, and some form of assessment to check that the learning has been understood and the learning outcome(s) achieved.

The materials give learners things to do and a chance to think for themselves. They are designed to be fun. Learners can make mistakes but can try again. There is plenty of in-built help if they get stuck and opportunities to go back and revise. Many of the materials include games or quizzes. Accessibility options are available on many of the materials to suit learners' individual needs. The materials are also designed to appeal to the different learning styles of students.

This link will take you to the NLN website where you can access the materials.

Scotland's Colleges

Scotland’s Colleges has a range of Learning Resources which include online formative assessment items, across a range of subject areas at SCQF levels 1-8. These assessment items were designed to be incorporated into the range of VLEs in place in colleges in Scotland.

The benefits include:

  • immediate feedback and reinforcement to candidates
  • improved flexibility of access
  • savings in valuable teaching staff time.

e-Assessment Toolkit – guidance on e-testing

E-testing is a rapidly growing area of e-assessment involving the delivery of examinations and assessments on screen, either using local systems or web-based systems.

Guidance on e-testing delivery is available by using this link.

E-portfolios

Using e-portfolios for SQA qualifications

SQA is encouraging centres to provide learners with opportunities to use e-portfolios when taking SQA qualifications. E-portfolios offer benefits for learners, centres and SQA. They provide a secure environment where learners can store, organise and share their learning and assessment materials, in a range of media formats. E-portfolios allow tutors to access learners’ work, to assess and provide feedback. For SQA, e-portfolios offer a secure and flexible way to view assessment evidence and internal verification materials.

SQA has published guidance on using e-portfolios. This resource will give you advice about good practice, and highlights some of the things you’ll need to take into account when you’re supporting learners as they create e-portfolios for SQA qualifications. The guidance also provides examples of e-portfolio forms and templates which could be useful for demonstrating good practice in using e-portfolios for assessment.


 

From 2008-2010, SQA worked with centres and External Verifiers to pilot e-verification. This project was initiated because SQA was aware that centres and learners were increasingly using e-portfolios and other electronic means to manage assessment evidence and they wanted to explore how SQA could make use of e-verification as part of its quality enhancement process.

The report on this pilot is available by using the following link:

Piloting e-verification 2008-10

SQA DeskSpace

DeskSpace allows candidates to record their course evidence online, keeping everything together and accessible to teachers/tutors and assessors, even when they are at more than one centre. Work can be uploaded and submitted online and the user friendly interface helps candidates structure their learning through the use of online templates.

The DeskSpace e-portfolio is available to all SQA Skills for Work and Baccalaureate candidates and is currently available to centres free of charge until June 2012.

This link will take you to the SQA webpage: Link opens in new windowSkills for Work ePortfolios (DeskSpace).

Other e-portfolios

SQA's Endorsement Service

SQA offers an endorsement service for organisations which provide resources to support the delivery of SQA qualifications and awards. This service may also be applied to e-portfolios. The Endorsement team evaluates the e-portfolio against both the endorsement criteria and SQA's requirements for e-portfolios to ensure that it is robust and fit for purpose. You can find out about SQA's Endorsement Service by using this link.

e-Assessment Toolkit – guidance on e-portfolios

There are various types of e-portfolio systems ranging from a secure file storage function within a stand-alone system, to more sophisticated web-based products, or they may be part of a VLE. Centres may also decide to develop their own e-portfolio systems.

Guidance on e-portfolio use is available by using this link.

JISC e-portfolio resources

JISC is the UK’s expert on information and digital technologies for education and research.

Two resources from JISC offering guidance on large-scale e-portfolio implementation are available by using this link.

The e-Portfolio Implementation Toolkit has been developed from the JISC-funded e-Portfolio Implementation (ePI) study of successful institution-wide practice in the UK, Australia and New Zealand. The toolkit contains over one hundred exemplars of e-portfolio use in a wide range of further and higher education contexts and eighteen case studies of approaches to wide-scale implementation. These form the basis of the Toolkit guidance that addresses the challenges, issues and stages in wide-scale e-portfolio implementation and presents an implementation model to assist your institution or programme in embedding and sustaining e-portfolio use.

Also available are five new video case studies, Stories of e-portfolio implementation, illustrating implementation strategies taken by five of the case study institutions. These rich-media resources provide additional personal insights into the challenges and benefits each institution has encountered. The videos are embedded into the Toolkit and can be downloaded or played by using this link. Transcripts are available for users of screen readers.

Both resources are made freely available, providing practitioners and managers in further and higher education with a unique insight into current UK and overseas e-portfolio practice. Feedback, thoughts and comments are welcome on these resources, and the facility to do so provided in the Toolkit itself. The resources can be used in conjunction with Effective Practice with e-Portfolios (JISC 2008) and the JISC e-Portfolios infoKit to provide a comprehensive package of support and guidance from JISC on e-portfolios in further and higher education.

Other existing resources

Social software

SQA CLASS

CLASS (Collaborative Learning Assessed by Social Software) is a tool for learners studying for the Project-based National Course (PBNC) in Health and Safety in Care Settings. It offers them a blog and a wiki to record group activities and generate a group project.

The graphic below is a link that will take you to the CLASS homepage.

CLASS_w200


 

JISC infoNet

JISC infoNet aims to be the UK's leading advisory service for managers in the post-compulsory education sector promoting the effective strategic planning, implementation and management of information and learning technology.

JISC has produced an online Infokit  Guide to Social Software.

This infokit explains more about social software, illustrated with real-life examples, and outlining potential uses for learning and teaching, research and business and community engagement. It also aims to highlight the key issues which may need to be considered when implementing such technologies, and to provide pointers for successful use.


 

Travel and Tourism at Cardonald College

The link below will take you to the transcript of a Case Study on using a wiki to support research, planning and assessment in Travel and Tourism.

Podcast Transcript – Cardonald College

Games-based assessment

Education Scotland and game-based learning

The vibrant and dynamic world of the computer game and its impacts on learning and teaching is an area that Education Scotland is committed to exploring, promoting and developing.

You can find out what has been happening in classrooms around Scotland by using the following link: Link opens in new windowEducation Scotland and game-based learning

Mobile technologies

Mobile and wireless technologies can provide flexible and timely access to learning resources, instantaneous communication, portability, active learning experiences and the empowerment and engagement of learners, particularly those in dispersed communities. Yet, despite the almost ubiquitous ownership of mobile phones, the adoption of mobile technologies in post-16 teaching practice is still in its infancy.

This link takes you to a case study on using mobile phones for formative assessment.

This link takes you to a case study on using tablet PCs for e-assessment.


 

SQA Deskspace

The Deskspace Mobile approach has now been trialled and tested and is available for download from the Add Ons section in your Deskspace Tools menu. The application is a simplified version of the e-portfolio with ability to send pictures and record evidence straight into your e-portfolio account. We hope this will help overcome difficulty in accessing suitable ICT equipment and time available to upload material to the e-portfolio. There are three formats available to suit a range of mobile phones.

Deskspace Mobile - Basic: A basic WAP enabled interface which is visible on all web enabled phones including an upload function. Deskspace Mobile - PDA: This includes the functionality above but is designed to operate on Windows Mobile PDAs and Smartphones.

Pebblepad iPhone app: This is an approved app by Apple. However, this version is currently only available under the Pebblepad branding and costs £1.79 to download, but may be customised for Deskspace at a later stage.

The graphic below is a link that will take you to the DeskSpace homepage.

DeskSpace Homepage

Electronic voting systems

Guidance on the use of electronic voting systems technology

If using these systems for summative assessment it would be advisable to seek awarding body prior-verification of the bank of questions you will use.


 

Electronic voting systems resources

You may find the following resources about electronic voting systems useful.

JISC

JISC is the UK’s expert on information and digital technologies for education and research.
Link opens in new windowElectronic voting analysis and feedback for all

Link opens in new windowCase Study on Using Moodle easyVoter Student Response System for formative assessment

SQA

SQA offers a Prior verification service to centres who devise their own instruments of assessment or who wish to significantly change items from the National Assessment Bank to suit their particular local needs.

This link will take you to SQA's Prior Verification Service.

How do I go about creating my own e-assessments?

Guidance on pedagogy

SQA

SQA is the national accreditation and awarding body in Scotland.

You may find information in the following links about how appropriate e-assessment is for various subjects and assessments:

 


 

SQA Academy

SQA Academy has been developed to extend and enhance the training and development opportunities available to all those who contribute to the assessment processes that underpin the qualifications of the Scottish Qualifications Authority.

The SQA Academy website is divided into a number of distinct "webspaces", each of which fulfils a different purpose. The majority of these webspaces are only available to registered users. However, some spaces are open to all, without the need for registration.

 


 

Professional Development Workshops

SQA's Professional Development Workshops are subject-based. SQA is currently piloting online workshops. They hope that by introducing online workshops they are able to reach more teachers as the cost will be much less. The online workshops are delivered on the SQA Academy website (www.sqaacademy.org.uk).

SQA's programme of Professional Development Workshops can be accessed by using the following link:

Professional Development Workshops

JISC

JISC is the UK’s expert on information and digital technologies for education and research.


 

JISC has a set of e-Assessment resources in the JISC Design Studio.


 

You may find information in the following link about how appropriate e-assessment is for various subjects and assessments.
Link opens in new window
JISC


 

Assessment and Feedback programme

JISC projects funded under the Assessment and Feedback programme have just commenced and will be disseminating key findings over the next 3 years.

Assessment and Feedback

Scotland’s Colleges

Scotland’s Colleges exists to support, represent and promote the Scottish college sector.

Link opens in new windowScotland’s Colleges


 

Scotland’s Colleges maintain Subject Networks which provide opportunities for debate, discussion, collaboration and sharing information.

These Subject Networks can be accessed by using the following link:

Link opens in new windowSubject Networks

Education Scotland

Education Scotland is the national body which is responsible for supporting quality and improvement in learning and teaching from early years to adult and community learning.

Link opens in new windowEducation Scotland

e-Assessment Toolkit

The e-Assessment Toolkit exists to provide clear, current, practical information to e-assessment practitioners, or those thinking of adopting e-assessment.
The toolkit offers useful tools and resources, helping users to develop and share their knowledge of e-assessment for effective implementation.

This link will take you to the e-Assessment Toolkit.


 

The e-Assessment Toolkit is hosted on the website of the e-Assessment Association (eAA).

The e-Assessment Association is for assessment/learning professionals or educators wanting to know more about the use of technology for exams, tests and assessments, and who want to influence the way electronic assessment develops.

This link will take you to the website of the e-Assessment Association.

Quality assurance and enhancement

Awarding body validation and verification

Where e-assessments are created and used for summative assessment, they will be subject to validation and verification checks by the relevant external body.

The validation of e-assessments will focus on the form and content of the assessment questions, to check on the validity of the assessment for its intended purpose. Some awarding bodies will offer a prior-verification service for this.

The verification process will focus on the delivery and support mechanisms for the e-assessments, in order to ensure both reliability and practicality. Some awarding bodies will carry out checks on this at the prior-verification stage while others will consider it as part of their standard verification process.

In verification of e-tests which have already been delivered, it should be possible to view details of when a learner took an assessment, how long it took, and from which IP address it was taken. It should also be possible to re-create the assessment exactly as it was taken by the learner, including any randomly generated parameters and ordering used. Records of all responses to each question should be kept on the server, including the full text of free-text responses and any feedback or result given to the learner.

SQA prior verification of e-assessments

Where centres create their own assessment instruments and marking schemes, for use in summative assessment of SQA qualifications, SQA encourages them to seek prior verification of the assessments and the associated marking schemes. Prior verification is a valuable service which helps avoid issues arising with centre-devised assessment materials at external verification. It is currently a free service and if you are intending to create e-assessment items for summative assessment purposes you can also use this service.

If you intend to create a bank of assessment items, to allow learners to take online, automatically-marked summative tests for SQA qualifications, we would encourage you to tell us about that in advance to allow the bank of items and the assessment approach to be discussed.

SQA offers a Prior verification service to centres who devise their own instruments of assessment or who wish to significantly change items from the National Assessment Bank to suit their particular local needs.

This link will take you to SQA's Prior Verification Service.


 

The Link opens in new windowSOLAR summative e-assessments are pre-verified — so centres know that quality is assured. This provides real benefits for both learners and learning providers.

Internal quality enhancement

SQA has produced several publications on internal quality enhancement.

Although it was published in 2001 and uses the term 'Internal Moderation' instead of the current term 'Internal Verification', Guide to Internal Moderation for SQA Centres still contains a wealth of relevant information.

Specifically written for Ofqual qualifications, Internal Verification: A Guide for Centres Offering Ofqual Regulated Qualifications, is a brief guide on good practice in internal verification which is relevant in many other areas.

Quality enhancement via networks

Scotland’s Colleges

Scotland’s Colleges maintain Subject Networks which provide opportunities for debate, discussion, collaboration and sharing information.

These Subject Networks can be accessed by using the following link:

Link opens in new windowSubject Networks


 

SQA

SQA's Professional Development Workshops are subject-based. SQA is currently piloting online workshops. They hope that by introducing online workshops they are able to reach more teachers as the cost will be much less. The online workshops are delivered on the SQA Academy website (www.sqaacademy.org.uk).

SQA's programme of Professional Development Workshops can be accessed by using the following link:

Link opens in new windowSQA's Professional Development Workshops

What other things do I need to think about?

Requirements for delivering e-assessment

SQA requirements

SQA has adapted requirements from E-assessment: Guide to Effective Practice for e-portfolios and for e-testing.

Standards

Standards exist that govern the delivery of online testing – BS ISO/IEC 23988:2007 Information technology – and the following information in this section is drawn from those requirements and is intended to relate more to timed, supervised e-assessments.

Infrastructure & environmental

Requirements – networks; computers; rooms

Technical capacity & infrastructure of institution

There are different types of e-assessment, some more demanding on centres’ infrastructure than others. It may be that not every institution is capable of applying and supporting the infrastructure needed for high stakes assessment using e-testing software, unless the institution can guarantee the levels of IT support and backup facilities required.

Similarly e-testing and other e-assessment approaches that require all candidates to undertake an assessment at the same time will make heavy demands on the infrastructure of an institution – computers, rooms, networks, invigilators - and may not be the right choice of strategy for every centre.

However e-portfolios, and other approaches to e-assessment which allow candidates to present their evidence for assessment when they are ready, should be possible for most centres if the centres are willing to invest in the necessary technology, staff development and learner support.


 

Student workstations

The supplier of the assessment engine will specify the minimum requirements for the memory and speed of computers being used by students for e-assessments, as well as the operating system needed. You should check the screen resolution and colour depth, as well as the keyboard and mouse for functionality, of all computers that will be used by students. No student should be put at any disadvantage, or given an unfair advantage, by the specification or the set-up of the equipment that they are using. Sufficient workstations are required to allow one networked computer per student, with some spare capacity. We suggest that one additional workstation for every ten students sitting the assessment would be sufficient to cater for most unforeseen difficulties.

A typical workstation needed for e-assessment would include:

  • a web browser
  • a mouse
  • a keyboard
  • a soundcard with jack-point for headphones
  • Macromedia FlashTM plug-in installed
  • JavaScript enabled
  • space next to it to do rough working

 


 

Servers

It is advisable if possible to mirror the server in a separate location so that e-assessments can still go ahead if the primary server breaks down.

Servers should be backed-up daily, and it is particularly important to back up students' responses immediately after the assessment has taken place.

Health and safety

JISC RSC E-Responsibility Online Resource

JISC Regional Support Centres have brought together the best resources for raising awareness and planning e-safety which will support a range of roles. This toolkit has been put together as a national resource designed to be of use to develop strategy.
RSC UK - National e-Safety toolkit


 

BS / ISO

BS ISO/IEC 23988:2007 Information technology - A code of practice for the use of information technology (IT) in the delivery of assessments - recommends that no one should spend longer than 1½ hours working continuously at a computer. This should not pose any real problem where NAB or college assessments are involved, although it is as well to be aware that the time any student spends on the computer may include:

  • logging on to the local authority and/or the school/college intranet
  • logging on to the assessment site
  • taking the assessment
  • any additional time granted because of special assessment arrangements

Emergency planning

The infrastructure requirements for large scale delivery of summative e-assessments, particularly high-stakes assessments, are considerable, and before an institution commits to such a delivery, adequate thought must be given to contingency planning. This can cover a host of contingencies – server problems, network problems, backup and data recovery issues, desktop issues, personal profile problems, logging problems, interference from other network services, and so forth.

When running an e-assessment session It is advisable to have spare workstations in case equipment fails (approximately one spare workstation per ten students). Students must be able to restart the assessment on another computer without loss of working time, and if possible to access any responses they have already made. In the event of system failure or network outage, responses should be saved so that students can restart from where they left off when the system has been restored. Timing should be reset accordingly.

In the event of fire alarms or other emergencies, it should be possible to vacate the room with browsers still open. When they return to the assessment, students should be given the full time for completion. Details of all emergencies and the implications for the assessment should be logged.

Staffing issues

Invigilation

Invigilation is the supervision of an examination for the purpose of maintaining a fair and consistent testing environment. Even if a centre has a room designed for e-assessment, it may still be necessary to invigilate summative e-assessment sessions, and more than one member of staff may be needed to ensure the assessment is securely monitored and technical support is at hand.

The publication e-Assessment: Guide to effective practice includes information on the roles and responsibilities of those involved in e-assessment, including for invigilators monitoring e-testing.

Key areas of responsibility could include:

  • Verifying that the environmental conditions of the e-assessment location are suitable, including for learners with specific requirements
  • Checking that learners have been authenticated to take the e-assessment
  • Checking that learners are familiar with the procedures and regulations for the e-assessment
  • Explaining to learners what support is and is not allowed during the e-assessment
  • Checking that learners are logged on and unlocking the e-assessment
  • Supervising the e-assessment
  • Resolving any issues that arise, liaising with other staff as appropriate
  • Providing support to learners, checking that they have access to technical and other appropriate assistance
  • Reporting any emergencies, technical failures and irregularities and ensuring that these are resolved appropriately for the learner
  • Documenting any such event and notifying the awarding body of the details, if necessary
  • Supervising any planned and unplanned breaks
  • Supervising the logging-off/closing of the e-assessment in accordance with safety and security procedures
  • Ensuring learners’ responses and associated information have been submitted to the awarding body.

Staff development

Some subject specialists feel the balance of time spent in staff development on e-assessment should be weighted in favour of pedagogy over software, because the skills needed to use e-assessment systems are relatively easy to develop, compared with developing skills and experience in the pedagogy of assessment writing.

Staff induction

Both staff and students need adequate preparation to use e-assessment. For students, this will largely focus on the need to familiarise themselves with the software interface, although if e-assessment is to be used within the context of a virtual learning environment, the initial induction may have to be more substantial and ongoing. This will also be the case with deploying software such as e-portfolios.

For staff, however, a similar process of familiarisation will be necessary, but in conjunction with further training in deployment, tracking and reporting processes associated with e-assessment. An entirely different and more complex kind of induction will be necessary where staff will be required to create e-assessments. The focus here will be as much on pedagogy as on technology.

While members of a Senior Management Team may find a briefing on e-assessment adequate, teaching staff who will be using the technologies are likely to need training and support. Support staff in libraries, and in administration and technical areas, may need specialised training in how to support those using the systems.

Technical support

Technical assistance is needed both before and during e-assessment sessions at a number of levels. Maintenance of computer labs, networks and servers becomes much more critical when they are being used for an invigilated assessment. Also when running e-assessment sessions and when using practice e-assessments, it may be necessary to involve the librarian or other support staff. It is advisable that:

  • technical staff are given advance notice and made aware that such assessments should be given priority
  • dedicated technical support is available both before and during assessments

e-Assessment Toolkit – guidance on roles and responsibilities

Key areas of responsibility within e-testing and e-portfolios contexts will vary. Responsibility for e-assessment may be taken forward by one individual or indeed by different people depending on the resources available.

Guidance on roles and responsibilities in e-assessment is available by using this link.

Learner issues

Technical demands on learners

Although e-assessment benefits many candidates, the use of an e-assessment option can create an added demand on the technical skills of learners. In some circumstances, this can create an unnecessary barrier to the assessment of knowledge and learning. This will vary from individual to individual and from group to group, and with different software interfaces.

In general, requiring learners with low levels of IT skills and confidence to use an e-assessment interface, particularly one which is not user-friendly, may be inappropriate. One strategy to help remove this barrier is to use the interface initially for formative assessment, to build learner confidence and familiarity.

Learner induction

Learners who are required to use an e-assessment system will need to familiarise themselves with the software interface. If e-assessment is to be used within the context of a virtual learning environment, and/or e-portfolio, the induction may have to be more substantial and ongoing.

In order to ensure that the technology doesn’t affect the outcome of any assessments, students must be familiar with the system, and comfortable with using it, before they use it for summative assessment. They should practise using the system, trying out the navigational aspects of the software and most of the question types - questions with a similar style, level of difficulty and method of feedback to those to be used in the actual assessment.

Students should be given Usernames and Passwords as required to carry out the practice assessments. Practice assessments can be conducted in the centre, either in timetabled subject time, for example during personal development time (PDT), or outwith timetabled time, such as in a library during a break. Alternatively, it may be possible for students to carry out practice assessments at home, if you feel this is appropriate, and the software supports it.

It’s important that learners are made aware, at an early stage in the use of e-testing, that the marks provided in a summative online test will be automatically assigned by the computer. You may need to modify these marks and, once this has happened, the results can be returned to the learners. You should also make learners aware that, as happens with paper-based assessments, the results will be provisional at this stage, and could be subject to internal and external verification.

Timing and pacing

Some e-assessment is self-paced, particularly in the area of formative assessment. Where it is not, the planning of e-assessment, especially planning the timing and pacing of the e-assessment delivery, must be considered.

Largely, consideration of timing issues for e-assessment will be similar to considering timing for conventional assessment delivery, but with the caveat that learners may need to be given earlier opportunities to familiarise themselves with the software interface. Pacing can be a more particular consideration for certain assessment software packages, ie those which allow you to set time constraints on the assessments at the global or individual question level.

Although online assessment, particularly objective test type questioning, may appear to be less time consuming than conventional deliveries, learners need to be given adequate time to read and check questions and answers, and it is a common fault to underestimate the amount of time needed to complete such assessments.

Research has shown that learners taking assessments on computer require on average ten percent more time to complete the assessments compared to traditional methods. You should allow for this when you are planning the assessment.

Access & inclusion, including assistive technologies

JISC

JISC TechDis is a leading UK advisory service on technologies for inclusion.

They explore and promote inclusive practices, resources and advice for learning and teaching in UK higher education, further education and skills sector, and independent and specialist colleges.

The following link will take you to an overview of the resources and publications from JISC Techdis complete with signposting to their location on the new JISC Techdis website.

A Guide to JISC Techdis Resources

 

TechDis Tuesdays

TechDis Tuesdays are online fortnightly workshops organised by JISC where they:

  • present a short video conversation on a specific accessibility theme;
  • facilitate a live ‘accessibility clinic’ on the day’s topic, providing an opportunity for people to swap stories, compare experiences and discuss good practice;
  • signpost relevant resources from JISC TechDis and other partner organisations.

 

For more information about TechDis Tuesdays, including topics for future sessions and to access recordings of previous sessions, visit the TechDis Tuesday page of the JISC website.


 

SQA

SQA has worked with CALL (Communication, Access, Literacy and Learning) Scotland to design digital question papers for learners who've been identified as having reading and/or writing difficulties, including learners who may sometimes require human readers or scribes. The digital papers are supplied in PDF format on CD. Learners can read the question paper 'on screen', using speech technology, where appropriate. Where the format of the question paper is a question/answer booklet, learners can use the computer to write their responses, or can ‘speak’ their responses.

Digital question papers from previous years are available for download from SQA by using this link.

CALL Scotland provides services and resources to assist with communication difficulties, particularly difficulties experienced in learning contexts. CALL Scotland can provide advice and training on creating digital question papers for use as prelims or for assessment generally.


 

Scotlands Colleges

Copies of the Profound and Complex Needs Newsletter are available in Latest Publications at Scotland's Colleges website by using the following link:

Publications from Scotland's Colleges

Quality enhancement

Supervised -v- unsupervised assessments

Some e-assessment activities will require supervision, such as taking an on-screen, closed-book test in which the learner is not allowed to access any additional resources. Other e-assessment activities may not need supervision; for example, creating a report for uploading to an e-portfolio for assessment.

See the following section on Authentication.

Authentication

Authentication involves introducing and maintaining systems to establish the identities of learners and the ownership of the evidence they submit for assessment. Centres are responsible for ensuring the authenticity of this evidence.

Authentication is linked to assessors’ knowledge of learners and their work; if assessors are familiar with a learner’s level and pattern of work they should be able to spot differences or anything unusual. Assessors will often observe activities being carried out and know that evidence produced is learners’ own work. However if evidence is produced without supervision, especially using resources acquired online, assessors may need to confirm that the evidence was generated by the learner, to ensure malpractice doesn’t occur.

Centres should have authentication strategies incorporating combinations of tools and approaches such as:

  • questioning or discussion with the learner
  • oral presentations by learners on their work
  • ensuring learners know about acknowledging sources
  • arranging with learners to access their blogs or to monitor other forums used to discuss progress
  • asking learners to warrant that their work is their own and ‘sign’ and date it when submitting it for assessment, especially any scanned evidence
  • using learner reflective accounts, peer reports and/or witness testimonies, with supporting evidence such as photographs
  • keeping authentication records for each learner
  • asking learners to undertake supervised ‘write-ups’ of elements of their written work
  • using electronic tools, such as plagiarism software.

Security

When running e-assessment sessions designed to be held in controlled conditions, the following procedures are recommended:

  • Avoid unauthorised access to the e-assessment system: For a summative assessment this can largely be achieved by only issuing the login details once all learners are seated at their computers.
  • Remove access to unauthorised materials: If not permitted for the assessment, access to aids such as on-screen calculators, spell checkers and search engines should have been disabled in advance of the assessment. This can be achieved by hiding navigation toolbars, disabling shortcut keys and preventing return to the assessment if any other software has been accessed. Learners should not have access to e-mail, Messenger services , e-learning materials or mobile devices during the e-assessment.
  • Prevent unauthorised disclosure of content: Facilities to print, copy materials into another application or send files to the hard disk, removable media or other computers should be removed.

Internal verification

Internal verification is the process used by centres to ensure that internal assessment has been carried out consistently within the centre and that national standards have been applied.

This involves checking assessment evidence and other assessment resources. Internal verification is checked as part of the External verification process.

If you use an online internal verification process to sample the assessment resources and record internal verification activity, this will allow more flexible approaches to external verification, such as E-verification.


 

Education Scotland

You may find the information on the Education Scotland website about Quality assurance and moderation useful.


 

Curriculum for Excellence

The following publication in the Curriculum for Excellence Building the Curriculum 5 - A Framework for Assessment series contains useful information about quality assurance and moderation.

Understanding, Applying and Sharing Standards in Assessment for Curriculum for Excellence: Quality Assurance and Moderation

External verification

External verification is the process used by SQA to ensure that internal assessment within and across centres has been carried out consistently and that national standards have been applied. It involves checking assessment evidence and other assessment resources, which can be in paper or electronic form; and also checking centres’ Internal verification processes.

External verification is carried out by SQA External Verifiers by visit, by post or through central verification. Where a centre is assessing learner performance using electronic media, such as e-assessment, External Verifiers could verify either by visiting the centre or, in some circumstances, by accessing the electronic assessment resources remotely. This is sometimes called E-verification.

As with external verification of traditional assessments, verification of e-assessment may also require the centre to provide information about the assessment event, such as the location; conditions and arrangements; authentication of learners and their evidence; results and reporting.


 

SQA has produced a Guide to External Verification, which can be accessed by using the following link:

External Verification: A Guide for Centres

E-verification

External Verifiers can in some circumstances verify by accessing a centre’s electronic assessment systems and resources remotely. This is sometimes called e-verification. With a centre’s agreement, External Verifiers could gain access to its systems online, from the External Verifier's home, awarding body premises or another mutually agreed location, for the purposes of carrying out e-verification.

Centres should have documented procedures detailing access arrangements for External Verifiers, including access arrangements to allow e-verification, where the centre has agreed to this. If access to the centre’s system is not possible remotely, traditional external verification will be used.


 

SQA

SQA has produced guidelines for FE staff and external moderators on e-moderation. SQA e-Moderation: guidelines for FE staff and external moderators (2004)

They have also published Case Studies on the moderation of e-assessments. Moderation of e-assessment : Case studies (2006)

Verification of online tests

Verification of an e-assessment involves the inspection of various electronic documents, and possibly also paper documents. If the e-assessment is an online, automatically-marked test, these would include:

  • The instrument of assessment (IA): ie the test. If the online test includes randomised questions, it’s important that your e-assessment system is able to ‘tag’ each unique version of the test to the individual learner's name, together with the number of attempts made by the learner.
  • The learners' evidence: This will include the electronic evidence as input by the learner and also any rough working on paper which the learner has submitted as part of their evidence.
  • Evaluation of learners' evidence / evidence of internal verification: For any learner where the mark automatically awarded by the e-assessment system has been amended by an assessor, there should be evidence of the original mark and the amended mark, along with a note of the reasons for any changes. This would amount to evidence of internal verification.
  • Re-sits: If a learner has taken a second e-assessment, the same information about the re-sit assessment should be available for verification.

 

An external verifier carrying out an e-verification may need access to other documentation covering: the assessment event(s) - information relating to the location and conditions and arrangements used; the authentication of learners and the learners’ evidence; results and reporting. To be valid as evidence for external verification, these documents should be date and time stamped.

What policies will help me deliver e-assessments?

Centre policies

Centre learning & assessment policy/strategy

How centres organise and deliver e-assessment will vary considerably, and the planning processes for this delivery must take into account existing protocols for conventional assessment. These may largely be capable of being applied as they stand to e-assessment delivery, but it is likely that new procedures will also need to be created – for example the processes for appeal of or review of e-assessment decisions.

Your centre or authority may already have an e-assessment policy. Once you’ve checked this, the first things to do are to analyse your reasons for introducing e-assessment, consider the curricular areas to be targeted, and decide whether your centre has the necessary organisational and technical infrastructure. You should explain your reasons to the staff who will be involved in e-assessment. Students, and (if it is necessary) their parents/guardians, should also be told why e-assessment is being introduced, and how it links with the other course activities.

Centre authentication & security policy

Your centre or authority may already have an authentication/security policy. Ideally this should be a documented procedure on how the centre staff will check learners’ authentication and on the strategies to be used to ensure the authenticity of learners’ evidence.

You will find further information in the Authentication section of this web resource.

Awarding body requirements for e-assessment

UK Qualifications Regulators

The four UK qualifications regulators, SQA, QCA, DELLS and CCEA liaise on an ongoing basis on the issue of e-assessment. Some of the UK's awarding bodies also contribute to these discussions.

In 2007 the regulatory bodies published Regulatory Principles for e-Assessment to guide awarding bodies and centres on the use of e-assessment, with the intention of encouraging innovative practice and enhancing the quality assurance of e-assessment.

Guidance from SQA

SQA does not dictate the e-assessment systems centres use. However, as with traditional assessment approaches, we do require that the assessments are kept secure; that the storage of assessment materials is robust; that access to the assessments is restricted; that a system for authentication of candidates and their work is in place. If the external verification process determines that these requirements have not been met then the e-assessments may not be accepted.

SQA has drawn on Regulatory Principles for e-Assessment and other publications to produce criteria designed to advise SQA centres and SQA External Verifiers about the standards expected of e-assessment systems which are used to deliver SQA qualifications SQA Requirements for e-assessment.

SQA's requirements for e-portfolios provide guidance to ensure centres’ e-portfolios are robust and fit for purpose.

The e-Assessment Toolkit

The e-Assessment Toolkit was created on the basis of E-assessment: Guide to effective practice, a publication which was developed by the UK regulatory bodies.

This link will take you to the e-Assessment Toolkit.

Guidance on other awarding bodies’ requirements

Centres are advised to check with awarding bodies for guidance on their e-assessment policies.

JISC, Scotland’s Colleges and the e-Assessment Association may also be able to advise.

Access & inclusion policies & legislation

Legislation

By law, centres offering assessment for qualifications should take all reasonable steps not to discriminate against any student. If an e-assessment is not felt to be appropriate, an alternative or equivalent assessment, such as a paper version, should be provided.


 

You may find this paper from Education Scotland (formerly Learning & Teaching Scotland) to be of interest.

Focusing on inclusion and the Education (Additional Support for Learning) (Scotland) Act 2004

Access and inclusion considerations and policies

E-assessment systems should be compatible with different assistive technologies - screen readers, speech recognition software and touch screens, for example. More time should be allocated for students using such technologies, if it is needed.

For assessments where sound input or output is needed, students will have to be provided with headphones or be located in a separate room.

Images used in e-assessments should be provided with a textual equivalent. Students should be able to adjust the text size. Most systems allow the use of style sheets, enabling students to customise the appearance of the pages on the screen. Shortcut (access) keys should be made available as an alternative to the use of a mouse.


 

SQA's Assessment Arrangements

SQA has a section of its website detailing its Assessment Arrangements. This can be accessed by using the following link:

SQA's Assessment Arrangements


 

JISC TechDis

JISC TechDis has some resources relating to assessment policy. These can be accessed by using the following link:

Assessment policy resources

Accessibility and inclusion resources

You may find the following resources about accessibility and inclusion useful.

  • EduApps
    EduApps consists of eight useful software collections that are free for you to download and use.

    The EduApps Family covers a range of user requirements to support teaching and learning, so just choose the one that’s right for you.
     
  • JISC
    The Joint Information Systems Committee (JISC) is an organisation that advises the UK Further Education and Higher Education sectors on issues relating to the use of information and communication technologies.

    JISC operates the TechDis Web Accessibility and Usability Resource - a website that provides information on the practical and technical aspects of accessibility.
     
  • JISC Regional Support Centre in Scotland
    The JISC Regional Support Centre in Scotland has an eLearning Accessibility and Inclusion section of its website, dedicated to promoting accessibility and inclusion in all aspects of eLearning.
     
  • The Higher Education Academy
    The Higher Education Academy (HEA) supports the Higher Education sector by providing the best possible learning experience for all students.

    The HEA, in partnership with Scotland’s Colleges and the Equality Practitioners' Network, is running a three-year programme focusing on Embedding Equality and Diversity in the Curriculum in Scotland.
     
  • The World Wide Web Consortium
    Good design for students with additional support needs is good design for all students.

    The World Wide Web Consortium (W3C) has drawn up guidelines on how to make web content accessible. Following these guidelines when assessments are created makes the assessments more accessible to all students, whatever input they are using (desktop browser, voice browser, mobile phone, etc) and whatever constraint they are under (noisy surroundings, under- or over-illuminated room, hands-free environment, etc).

Interoperability & other standards on e-assessment delivery

Interoperability

Interoperability is the term used to describe the ability of two or more systems to exchange and use information. A typical example might be where one e-portfolio system allows a learner to transfer some or all of the contents of his or her e-portfolio to another e-portfolio system, perhaps when moving on from education to employment. Technical standards are being developed to facilitate this process.


 

JISC CETIS

JISC CETIS provides advice on educational technology and interoperability for the Higher and Further Education sectors.

You may find this JISC CETIS paper to be of interest:

The Future of Interoperability and Standards in Education – System and Process (2010)

Other standards on e-assessment delivery

International Organization for Standardization

Standards exist that govern the delivery of online testing – BS ISO/IEC 23988:2007 Information technology

World Wide Web Consortium

The World Wide Web Consortium (W3C) is an international community that develops open standards to ensure the long-term growth of the Web. W3C is working to make the Web accessible to all users (despite differences in culture, education, ability, resources, and physical limitations).

There is a W3C Community Group that focuses on e-learning. Participants discuss new and existing technologies for e-learning and M-learning. The group also talks about the reach, social change and impact of e-learning. E-learning: Evolving technologies and growing reach

Assessment policies and strategies

The Scottish Government

The report by John McClelland C.B.E. on his Review of ICT Infrastructure in the Public Sector in Scotland was published in June 2011.

The Scottish Government response to this Review includes a mention of the engagement that will be needed with all parts of the public sector and with the ICT industry  to deliver the vision described in the McClelland report, in a section entitled 'Governance'.

Amongst their achievements, the Scottish Government note that they have delivered two major services to the education sector: Glow (the national learning intranet for Schools) and Interconnect 2.0 (the supporting high speed broadband network).


 

The Scottish Government has produced an Infrastructure Action Plan which outlines its commitment to a world-class, future proofed infrastructure that will deliver digital connectivity across the whole of Scotland by 2020.

SQA

Following a consultation exercise, SQA has re-launched its Vision and Strategy for E-assessment. First published in March 2005, this document aims to set out clearly SQA's priorities for this area of work and the likely implications for SQA and its various stakeholder groups.

What delivery platforms/systems will help me deliver e-assessment?

Online testing

SOLAR

SOLAR e-assessments

SOLAR is the SQA's online assessment tool that provides summative and formative e-assessments. It’s based on banks of quality-assured question items. These e-assessments cover qualifications at all levels from National Certificate to Higher National Diploma.

The summative e-assessments are pre-verified — so centres know that quality is assured. This provides real benefits for both learners and learning providers.

The formative e-assessments are quality assured, and provide detailed feedback for learners. Learners can access the assessments 24 hours a day, 7 days a week using a browser and the Open Assess portal.

Each delivery of an e-assessment to a learner is dynamically generated by the system, based on pre-specified rules.


SOLAR Training

SQA has produced a range of interactive web-based training and support materials to enable centres to access and use the SOLAR e-assessment platform. The materials cover all of the main features of SOLAR administration and delivery.

To access these materials visit the SOLAR website and select Training Materials from the left hand menu. The materials are designed to inform on the key functions and activities within SOLAR.

If you require help with an area that is not listed, you can contact the SOLAR support team on solar@sqa.org.uk or call 0303 333 0330.


Information for New Centres

If your centre is an SQA approved centre but you are unsure if your centre has access to SOLAR, please visit the SOLAR website and check the List of approved centres trained to see if your centre already has a centre administrator for SOLAR. These administrators will be able to provide training and arrange for colleagues to access SOLARimmediately. (Some centres may have more than one person already trained).

If you think your centre does not yet have access to SOLAR, you can familiarise yourself with the SOLAR training materials and the functions of the e-assessment system before completing the training form provided. This should be submitted by email to solar@sqa.org.uk . The SOLAR team will then arrange access for your centre and email your login details.


To try a SOLAR formative assessment Link opens in new windowuse this link.


SQA guidance on e-testing systems

If you intend to offer e-assessment for SQA qualifications using an online testing system, your centre will be responsible for ensuring that the system meets the necessary standards in terms of how assessments are developed, scheduled and delivered to learners and how results are managed, internally quality assured and reported.

SQA uses quality criteria for this, which we adapted from the Regulatory Principles for e-Assessment, devised by the UK Qualifications Regulators. SQA’s version of the principles are designed to provide centres, technology providers and SQA External Verifiers with guidance on the standards required of non-SQA e-assessment systems which are used to deliver SQA qualifications.

You can download the criteria here - Link opens in new windowSQA Requirements for e-assessment (683 KB)

Other online testing systems

You may be able to find information on other online testing systems on the JISC and Scotland’s Colleges websites.

Using VLEs for online testing

VLEs typically offer both the capacity to delivery e-assessment content, created through stand-alone tools, and to create and track their own assessment items. A key feature in the VLE delivery of e-assessment is the ability to track, record and report on learners’ assessment activity. Some VLEs may also be able to integrate assessment activity with learning – for example returning students to a piece of online learning if they do not achieve the hurdle of an end assessment.

If you intend to use VLEs for online testing see Preparation: What other things do I need to think about? Quality enhancement: Verification of online tests.


 

JISC infoNet

JISC infoNet aims to be the UK's leading advisory service for managers in the post-compulsory education sector promoting the effective strategic planning, implementation and management of information and learning technology.

They have developed an infokit on the Effective Use of VLEs.

E-portfolios

E-portfolio software

Since many types of learning require an ongoing, portfolio approach to assessment it should be possible to build e-assessment opportunities using e-portfolio software. This technology is continuing to develop, and there are many types of portfolio software, which fulfil a wide range of differing functions.

When used for assessment, key things which portfolio software attempts to do are:

  • devolve some control of evidence generation/collection to learners
  • give tutors the opportunity to view and assess learner evidence and
  • allow the organisation of evidence against pre-determined assessment criteria.

SQA and e-portfolio approaches

SQA does not recommend any one e-portfolio system to its centres.

There are many different approaches and we will work with whichever system centres choose to use:

  • Some centres use Virtual Learning Environments (VLEs) to store, manage and assess learner evidence.
  • It’s possible to use commercial presentational software, or webpages, to gather and present evidence for SQA qualifications.
  • There are many commercial e-portfolio products available; SQA has endorsed several of these.
  • Centres could choose to work with a technology provider to develop a bespoke e-portfolio to meet their specific requirements.
  • There are open source e-portfolio products; one example is Mahara, which is often used along with the open source Moodle VLE.

 

SQA has developed e-portfolio requirements to provide guidance on choosing a system.

SQA has also published guidance on using e-portfolios. This resource will give you advice about good practice, and highlights some of the things you’ll need to take into account when you’re supporting learners as they create e-portfolios for SQA qualifications. The guidance also provides examples of e-portfolio forms and templates (51 KB) which could be useful for demonstrating good practice in using e-portfolios for assessment.

SQA Deskspace

SQA DeskSpace allows candidates to record their course evidence online, keeping everything together and accessible to teachers/tutors and assessors, even when they are at more than one centre. Work can be uploaded and submitted online and the user friendly interface helps candidates structure their learning through the use of online templates.

The DeskSpace e-portfolio is available to all SQA Skills for Work and Baccalaureate candidates and is available to centres free of charge until June 2012.

Other SQA-endorsed e-portfolios

SQA offers an endorsement service for organisations which provide resources to support the delivery of SQA qualifications and awards. This service also applies to e-portfolios.

The e-portfolio is evaluated against endorsement criteria and SQA's requirements for e-portfolios to ensure it is robust and fit for purpose.


 

You can find about the e-portfolios endorsed by SQA by using this link.

JISC e-portfolio resources

Two resources from JISC offering guidance on large-scale e-portfolio implementation are available by using this link.

The e-Portfolio Implementation Toolkit has been developed from the JISC- funded e-Portfolio Implementation (ePI) study of successful institution-wide practice in the UK, Australia and New Zealand. The toolkit contains over one hundred exemplars of e-portfolio use in a wide range of further and higher education contexts and eighteen case studies of approaches to wide-scale implementation. These form the basis of the Toolkit guidance that addresses the challenges, issues and stages in wide-scale e-portfolio implementation and presents an implementation model to assist your institution or programme in embedding and sustaining e-portfolio use.

Also available are five new video case studies, Stories of e-portfolio implementation, illustrating implementation strategies taken by five of the case study institutions. These rich-media resources provide additional personal insights into the challenges and benefits each institution has encountered. The videos are embedded into the Toolkit and can be downloaded or played by using this link. Transcripts are available for users of screen readers.

Both resources are made freely available, providing practitioners and managers in further and higher education with a unique insight into current UK and overseas e-portfolio practice. Feedback, thoughts and comments are welcome on these resources, and the facility to do so provided in the Toolkit itself. The resources can be used in conjunction with Effective Practice with e-Portfolios (JISC 2008) and the JISC e-Portfolios infoKit to provide a comprehensive package of support and guidance from JISC on e-portfolios in further and higher education.

Information on other e-portfolios

Open source e-portfolios

Open source e-portfolio products are free for anyone to use, however centres would have to host and manage their own versions of these products.

One example is Mahara, an open-source e-portfolio that can be integrated into Moodle, allowing learners to transfer content from their learning environment, such as forum posts and assignments, to their e-portfolio. Mahara authenticates with Moodle, so learners signed into Moodle can access their e-portfolio without logging in to Mahara. Mahara has a range of partners who can supply hosting, training and development. There’s also a Mahara User Guide.



In June 2011, Cumbernauld College won an award for the Link opens in new windowImmersive Use of Mahara for Creative Industry Courses.

VLEs, wikis and blogs as e-portfolios

If you’re thinking about using a VLE, wiki or blog as an e-portfolio, consider developing an e-portfolio ‘approach’ with these. This means replicating the structures, templates and security features, generally found in e-portfolio products, to organise and present assessment evidence.


Adopting e-portfolio approaches means:

  • establishing clearly structured secure storage locations within, or linked to, the VLE or social software product, to collate and present learner evidence
  • providing access to evidence of the assessment process, such as how assessment and marking is handled
  • providing access to evidence of the internal verification process, such as how this is carried out
  • showing how assessment results is recorded and stored securely.

 

All of this should be available and easily located within, or linked to, the VLE, wiki or blog.

Web 2.0 & Social software

Web 2.0

Web 2.0 technologies include social networking sites, blogs, wikis and video sharing sites and tend to focus on information sharing, user-centered design and collaboration.

Clearly there is a great deal of potential for using Web 2.0 technologies for assessment – for example to assess collaborative-type activities and group tasks, and for peer assessment.

You may find this Assessment 2.0 paper from SQA to be of interest.

SQA CLASS (Collaborative Learning Assessed by Social Software)

Assessing an individual’s contribution to a collaborative project can be difficult, however social software can help with this. In 2007/2008 SQA piloted the assessment of group work using a wiki and a blog. Learners were asked to work with two or three others to plan tasks, carry out research and then jointly develop a product. Each learner kept a record of the activities they completed as an individual, and shared this with the group through the blog. These records formed part of the learner’s assessment evidence and helped them plan their evaluation reports.

CLASS (Collaborative Learning Assessed by Social Software) is a tool for learners studying for the Project-based National Course (PBNC) in Health and Safety in Care Settings. It offers them a blog and a wiki to record group activities and generate a group project.

CLASS Students

The learners’ contributions to the ‘product’ were stored in the wiki, which, by capturing the entries and amendments made by all members of the group, created an authenticated record of every learner’s contribution to the project. Tutors were able to keep track of each individual’s progress; the wiki showed progress with the project, at all times, while the blog provided information on ongoing activities (research etc) and a means of communication for the tutor. A secure login contributed to authenticating the learners’ work. SQA Markers then marked the group project for each learner, using evidence from the learner’s entries to the blog and the wiki.


The Report into the pilot study:

Link opens in new windowCLASS Summary Report


The CLASS Project:

Link opens in new windowCLASS Project


The group wiki and blog software provided to students and teachers of the PBNC in Health and Safety in Care Settings is currently available for any other qualification. The eAssessment and Learning team is keen to extend the use of CLASS and will facilitate its adoption by any interested teams. If you are interested in learning more, please contact Veronica Harris or Tel:  0776 855 4004.

Games-based assessment

JISC

JISC has published a briefing paper aimed at policy makers, senior managers and practitioners who are interested in an overview of game-based learning and how it may be used to support effective learning and teaching practice.

Although this paper concentrates on game-based learning, it does contain information relevant to game-based assessment.

Game-based Learning

Mobiles

Mobile learning and assessment resources

You may find the following links about mobile learning and assessment useful.

  • JISC
    JISC is investigating the potential for mobile technologies and what impact these might have on learning and teaching practice.
    Link opens in new windowKey mobile learning activities
     
    A review undertaken for the JISC e-Learning Programme by JISC infoNet.
    Link opens in new windowJISC Mobile and Wireless Technologies Review
     
  • NMC
    The New Media Consortium (NMC) is an international not-for-profit consortium of learning-focused organizations dedicated to the exploration and use of new media and new technologies.

    The 2011 Horizon report includes case studies of use of mobiles for assessment.
    Link opens in new windowOne Year or Less: Mobiles
     
  • ESCalate
    ESCalate produces and disseminates resources for staff and students in Higher Education and Further Education.

    Making mobile learning work features some assessment elements in the case studies.
    Link opens in new windowMaking mobile learning work: case studies of practice
     
  • Learnosity
    Use of Mobile Phones for Language Learning and Assessment for Learning. Link opens in new windowA Pilot Project
     
  • Scotlands’ Colleges
    Link opens in new window
    Mobile assessment for ESOL

SQA DeskSpace

The Deskspace Mobile approach has now been trialled and tested and is available for download from the Add Ons section in the Deskspace Tools menu. The application is a simplified version of the e-portfolio with ability to send pictures and record evidence straight into your e-portfolio account. We hope this will help overcome difficulty in accessing suitable ICT equipment and time available to upload material to the e-portfolio. There are three formats available to suit a range of mobile phones.

Deskspace Mobile - Basic: A basic WAP enabled interface which is visible on all web enabled phones including an upload function.

Deskspace Mobile - PDA: This includes the functionality above but is designed to operate on Windows Mobile PDAs and Smartphones.

Pebblepad iPhone app: This is an approved app by Apple. However, this version is currently only available under the Pebblepad branding and costs £1.79 to download, but may be customised for Deskspace at a later stage.

Link opens in new windowDeskSpace Mobile final report

Link opens in new windowSkills for Work ePortfolios (DeskSpace)

How do I measure the success of my e-assessments?

Measuring Success

Evaluation

Assessment refers to measuring learner performance (either before or after a teaching intervention, or both). Thus, assessment can be part of an evaluation, but assessment and evaluation aren’t synonymous.


Educause - Measuring Success: Evaluation Strategies for Distance Education.


The article above suggests that, in order to evaluate the success of your (e-assessment) programme, you may first need to define what you mean by ‘success’. Can success be measured in terms of higher uptake, for example because you’re able to reach a wider group of learners using online assessment? Is it related to higher retention rates, better learner results or higher learner satisfaction? If e-assessment is to work effectively, meeting the needs of all users, including teaching professionals and support staff, as well as learners, each of these factors will probably be relevant. Therefore you may need a comprehensive and carefully planned evaluation plan.


Your evaluation plan could include regular surveys to capture information on learner results, staff views and technical and procedural issues. Gathering this information should allow you to monitor how well e-assessment does or doesn’t contribute to supporting your centre’s objectives, and so could also inform future plans.


You may be able to use your centre VLE to survey learners and centre staff, or one of the wide range of online survey tools available for creating questionnaires, delivering them and analyzing the responses.


Another potential evaluation tool is the ‘Interview’, which can be used instead of, or to supplement, a questionnaire. Whichever approach you use, you’ll need to decide on the areas you wish to evaluate. The Educause article suggests that most areas of interest fall under inputs (resources, personnel etc), outcomes (performance, attitudes etc) or implementation issues. Subcomponents of the implementation issues, for example, on which to focus evaluative questions, could include:

  • Technological concerns – stability; maintenance
  • Student support concerns
  • Faculty concerns – preparedness; involvement in curriculum & course development; professional development needs; incentives & rewards
  • Learner concerns – preparedness; access to delivery systems; communication / interaction with faculty & peers
  • Organisational concerns – quality assurance; accreditation criteria

 


This resource from Oklahoma State University (2002) offers advice on designing questions for both questionnaires and interviews -

Questionnaire and Interview as Data-Gathering Tools

Monitoring learner attainment

Reporting

As mentioned elsewhere, good e-testing systems will have Teacher Reporting facilities to allow you to monitor learner attainment in tests taken across different groups of learners and over time, as well as identifying which areas of the tests present problems for learners. These systems should be able to generate reports about the performance of individual learners, a class, or a number of classes.

Feedback

Feedback from quality assurance processes

Your centre will be involved at some level in coordinating, or participating in, internal and external quality assurance processes, designed to improving standards in the centre, and in education generally. These processes may require monitoring, tracking or benchmarking of learners’ progress, and could take place, for example, through:

  • The centre’s self-evaluation activities
  • SQA external verification
  • Other awarding bodies’ external verification
  • External evaluation from Education Scotland
  • Education authorities’ quality improvement processes.

 

If learners are using e-assessment, you may be able to gather data from the recommendations, findings or comments resulting from these quality processes which could contribute to your evaluation of the e-assessment provision.

How do I use my experiences to inform future practice?

Contribute to shared resources

This website

If you have good examples of e-test questions, or e-assessments which can be delivered via e-portfolios or social software which you think may be valuable for others to use, please contact Campbell White at SQA.

National Assessment Resource (NAR)

Use the following link to find out how you can contribute to Curriculum for Excellence resources in the NAR, for example by:

  • Uploading a resource
  • Creating a question
  • Assembling a task
  • Building an assessment
  • Linking related resources
  • Describing a resource
  • Publishing a resource.

 

Contribute to Curriculum for Excellence resources in the NAR.

Other

If you would like to contribute resources to JISC, the e-Assessment Association or Scotland’s Colleges, which could in turn be linked to this e-assessment resource, please contact these organisations directly by using the links below.