Column

Digitalization of Tests: The Development of Computer-Based Testing and Its Benefits

There are various "tests" around us in our daily lives that measure certain abilities. If you are a student, you have taken regular exams and entrance exams to measure your academic ability in a subject, and even after you enter the workforce, you may have the opportunity to take qualification exams that measure your knowledge of specific content. These "tests" have traditionally been conducted using paper and pen (also known as Paper Based Testing [PBT]).

However, in today's world where computers have become indispensable, many tests, including qualification exams, are shifting from paper-and-pen to computer-based testing. In recent years, many people have heard of CBT (Computer Based Testing). As the name suggests, CBT is a form of testing in which all or part of the processes involved in the administration of the exam (displaying questions, recording answers, grading, feedback of results, etc.) are conducted on a computer. Compared to PBT, CBT is known to have various advantages. For example, it can reduce the costs of printing and transporting answer sheets and labor costs such as data entry and grading, and it can also provide technology that selects questions of the optimal difficulty for each individual based on the correct/incorrect situation, just like a vision test (adaptive testing), and it can provide more realistic questions using a variety of information such as videos. CBT technology has already been introduced in large-scale public exams overseas (Alrababah & Molnár, 2021).

Until the beginning of the 21st century, when information and communication technology was not as developed as it is now, when high-stakes exams such as qualification exams and entrance exams [1] were conducted on a computer, Candidate gathered in a university computer room or a testing center, and the Test Result was administered in a proctored format. This is because the higher the stakes of the exam, the greater the risk of cheating (test fraud). In other words, CBT still had a major restriction that candidates could not freely choose the place to take the exam. Even now, when taking CBT, Candidate usually have to go directly to the designated Test Center. In recent years, test centers have been established all over the country, but it is still not uncommon in rural areas for it to take more than an hour to get to Test Center one way. In particular, the COVID-19 pandemic and natural disasters in recent years have made it possible to clearly see the risk of it being difficult to gather many Candidate at one Test Center. In fact, in 2020, the national Information-Technology Engineers Examination was forced to be canceled due to the impact of the COVID-19 pandemic (Information-Technology Promotion Agency, 2020). In addition, with CBT (or traditional paper-and-pencil tests), which forces test takers to take the test in an environment different from their usual one, it has been pointed out that Candidate may be unable to perform at their best due to strong test anxiety (e.g., Butler-Henderson & Crawford, 2020; Stowell & Bennett, 2010), and there are still issues that have yet to be resolved with CBT.

 

Innovation in Internet-Based Testing: IBT

In recent years, Internet Based Testing (IBT) has been attracting attention as a technology that can solve these problems of CBT [2]. In IBT, questions are distributed and answers are recorded via the Internet, so Candidate can take the test at their own convenience without having to go to Test Center. If IBT can be implemented properly, not only will it bring the benefits to Candidate mentioned above, but it is also expected to bring various benefits to the organizers, such as reduced costs for terminals and venues and an increase in the number of test applicants.

However, as mentioned above, the risk of cheating is very high in IBTs where Candidate are not gathered at Test Center, especially in unproctored IBTs. For this reason, IBTs were previously thought of as being usable only for low-stakes tests where Test Result were not used for important decision-making, or as practice tests (Parshall et al., 2002). However, in recent years, with the development of remote proctoring using cameras built into computers and technology that automatically detects suspected cheating using AI, it has become more realistic to conduct high-stakes tests using IBTs. In fact, for example, in Canada, some of the exams for graduating from medical school (The Medical Council of Canada Qualifying Examination) allow candidates to take the exam at home or elsewhere with a remote proctor. In addition, in 2022, the Fundamental Information Technology Engineers Examination and the Information Security Management Examination, which are national exams, conducted demonstration experiments with a view to switching to IBT (Information-Technology Promotion Agency, Japan, 2022).

The most important issue to consider when administering high-stakes exams using IBT is how to prevent cheating. Noorbehbahani et al. (2022) classify and list the cheating behaviors observed in recent studies on IBT. In addition to cheating seen in traditional exams, such as sneaking a peek at or searching textbooks, cheating behaviors specific to IBT, such as "taking the exam by proxy with a fake face or voice (Vegendla & Sindre, 2019)" and "using smart glasses and tiny wireless earphones (Srikanth & Asmatulu, 2014)," have also been reported. An effective way to suppress such cheating behaviors in IBT is to introduce an online proctoring system (OPS). In general, OPS not only monitors Candidate using webcams, but also enhances test security through a combination of functions such as identity verification, limiting the functions of the computer during the test, remote management (e.g., the ability to immediately disconnect suspicious Candidate), and generating reports (e.g., flagging suspicious test Candidate) (Hussein et al., 2020). In addition, the use of a lockdown browser is essential to restrict the functions of the computer during the test and securely implement IBT. A typical lockdown browser reduces the risk of cheating by forcing full-screen display during the test and disabling the use of software other than the test and certain keyboard operations. Of course, as cheating patterns are becoming more sophisticated, the test implementation guidelines presented by the International Test Commission in 2022 require the ability to control various functions such as virtual machines (VMs), remote desktops, multiple monitors, virtual cameras, and virtual microphones. Since it is not easy to design an OPS system that meets such advanced requirements, the cooperation of specialized vendors is almost essential when actually implementing IBT.

In addition to OPS, the use of AI-based proctoring systems (AIPS) has been increasing in recent years. In a typical OPS, in order to check for suspicious behavior, proctors would observe all Candidate Candidate real time and also conduct post-test checks using recordings (Noorbehbahani et al., 2022). However, when many Candidate take the test at once, manual checks quickly reach their limits. Therefore, an increasing number of vendors are introducing AIPS that can automatically flag suspicious Candidate using AI technology. AIPS often detects cheating based on machine learning, deep learning, and other techniques, for example, using recorded facial movements of Candidate (e.g., Indi et al., 2021; Yulita et al., 2023) and mouse cursor trajectories (e.g., Hassan Hosny et al., 2022; Li et al., 2021). As such, with the development of technology, the quality of cheating detection is improving day by day. Of course, the level of technology used to cheat is also increasing, so it may still be difficult for IBT to completely prevent cheating to the same level as CBT (held at Test Center). However, it is already realistic to improve the security of IBT to a practical level by introducing appropriate technology.

　

Towards practical application and widespread use of IBT

Of course, we do not think that all tests are suitable for IBT. For example, in public high-stakes tests such as university entrance exams, IBT is expected to pose not only technical but also psychological hurdles. This is because, particularly in Japan, there is a deep-rooted belief that the fairness of a test is ensured by gathering Candidate in the same Test Center and having them take the same questions on the same day and at the same time, and in some cases, the merit of IBT, that "you can take the test anywhere," may be perceived as "undermining fairness."

Even if security issues are ignored or resolved, research on IBT is still in its infancy, and no clear conclusions have been reached on many of the issues and concerns at present. For example, in the first half of this column, we mentioned "reducing test anxiety" as an advantage of IBT, but in reality, it has been pointed out that IBT has its own (technical) concerns, such as "Will the connection be lost during the test?" and "Will the answers be properly recorded?" (e.g., Conijn et al., 2022; Dikmen, 2022). In addition, online/AI-based test supervision requires various interventions, such as having the Candidate 's surrounding environment (room or desk) projected in advance, having Candidate install special software on the test taker's computer, and collecting and recording various information such as the test taker's behavior and operation logs. However, Candidate generally do not seem to be happy about having access to various information (Terpstra et al., 2023). In recent years, as the movement to protect personal information has increased worldwide, discussion of the legal and ethical aspects of privacy protection regarding such data use will also become important.

Thus, although there are still many issues remaining with IBT, as we have seen, IBT has already reached a practical stage, and in fact many exams overseas are being converted to IBT. If you are thinking of implementing a new IBT-format exam in the future, it may be a good idea to first fully understand the current situation and issues surrounding IBT through experts, and then carefully consider whether it is really possible to implement the exam using IBT (especially in terms of security) and what the benefits are of using IBT instead of CBT.

Alrababah, SA, & Molnár, G. (2021). The evolution of technology-based assessment: Past, present, and future. International Journal of Learning Technology, 16 (2), 134–157. https://doi.org /10.1504/IJLT.2021.117765

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (Eds.). (2011). Standards for educational and psychological testing. American Educational Research Association.

Butler-Henderson, K., & Crawford, J. (2020). A systematic review of online examinations: A pedagogical innovation for scalable authentication and integrity. Computers &Education,159, Article 104024. https://doi.org/10.1016 /j.compedu.2020.104024

Conijn, R., Kleingeld, A., Matzat, U., & Snijders, C. (2022). The fear of big brother: The potential negative side‐effects of proctored exams. Journal of Computer Assisted Learning, 38 (6) , 1521–1534. https://doi.org/10.1111/jcal.12651

Dikmen, M. (2022). Test anxiety in online exams: Scale development and validity. Current Psychology. https://doi.org/10.1007/s12144-022-04072-0

Hassan Hosny, HA, Ibrahim, AA, Elmesalawy, MM, & Abd El-Haleem, AM (2022). An intelligent approach for fair assessment of online laboratory examinations in laboratory learning systems based on student's mouse interaction behavior. Applied Sciences, 12 ( 22), Article 22. https://doi.org/10.3390/app122211416

Hussein, MJ, Yusuf, J., Deb, AS, Fong, L., & Naidu, S. (2020). An evaluation of online proctoring tools. Open Praxis, 12 (4), 509–525. https:// doi.org/10.5944/openpraxis.12.4.1113

Indi, CS, Pritham, V., Acharya, V., & Prakasha, K. (2021). Detection of malpractice in e-exams by head pose and gaze estimation. International Journal of Emerging Technologies in Learning, 16 (08), 47–60. https://doi.org/10.3991/ijet.v16i08.15995

Li, H., XU, M., WANG, Y., WEI, H., & QU, H. (2021). A visual analytics approach to facilitate the proctoring of online exams. CHI '21: Proceedings of the CHI Conference onHuman Factors in Computing Systems, Yokohama, May 8-13, 1–17. https://doi.org/10.1145/3411764.3445294

Noorbehbahani, F., Mohammadi, A., & Aminazadeh, M. (2022). A systematic review of research on cheating in online exams from 2010 to 2021. Education and Information Technologies, 27, 8413–8460. https://doi .org/10.1007/s10639-022-10927-7

Parshall, CG, Spray, JA, Kalohn, JC, & Davey, T. (2002). Practical considerations in computer-based testing. Springer New York. https://doi.org/10.1007/978-1-4613-0083 -0

Srikanth, M., & Asmatulu, R. (2014). Modern cheating techniques, their adverse effects on engineering education and preventions. International Journal of Mechanical Engineering Education, 42 (2), 129–140. https://doi.org /10.7227/IJMEE.0005

Stowell, JR, & Bennett, D. (2010). Effects of online testing on student exam performance and test anxiety. Journal of Educational Computing Research, 42 (2), 161–171. https://doi.org/10.2190/ EC.42.2.b

Terpstra, A., De Rooij, A., & Schouten, A. (2023). Online proctoring: Privacy invasion or study alleviation?: discovering acceptability using contextual integrity. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 1 –20. https://doi.org/10.1145/3544548.3581181

Vegendla, A., & Sindre, G. (2019). Mitigation of cheating in online exams: Strengths and limitations of biometric authentication. In AVS Kumar (Ed.), Biometric authentication in online learning environments (pp. 47–68). IGI Global. https://doi.org/10.4018/978-1-5225-7724-9.ch003

Yulita, IN, Hariz, FA, Suryana, I., & Prabuwono, AS (2023). Educational innovation faced with COVID-19: Deep learning for online exam cheating detection. Education Sciences, 13 (2), Article 2. https: //doi.org/10.3390/educsci13020194

Information-technology Promotion Agency (IPA). (2020). Press release: Cancellation of the Information-technology Engineer Examination and Information-technology Security Specialist Examination scheduled for April 19th (Sun). https://www.ipa.go.jp/archive/press/2019/press20200324.html

Information-technology Promotion Agency, Japan. (2022, August 4). Internet-based demonstration tests conducted for the Fundamental Information Technology Engineer Examination and Information Security Management Examination. https://www.ipa.go.jp/news/2022/shiken/topic_2022_ibt.html