INTRODUCTION
The development of science and technology in the era of the 4th Industrial Revolution has led to various changes in daily life, including virtual reality (VR) [1]. VR is a hu-man-computer interface technology that allows users to interact in a virtual space via a computer [2]. It is used in various fields (e.g., video, medical care, architecture, and education) and has been shown to be effective in increasing student immersion and inducing self-directed learning, especially in terms of education. Therefore, educational content development using VR technology has been actively conducted in various disciplines [2]. Nursing education emphasizes education that reflects practicality [3]. With the recent increase in non-face-to-face education due to the coronavirus disease (COVID-19) pandemic, online education, such as videos, is being used as an alternative. However, currently used video education has practical limitations in nursing practice education; therefore, it is necessary to develop educational media or technology that can replace it [4]. To solve these limitations, interest in the use of VR for practical training is increasing.
VR is a three-dimensional environment or technology that provides a realistic and interactive experience through virtual situations that imitate reality [5]. VR-based education programs can be used effectively when empirical learning is complex, owing to limitations or risks in implementing real situations. In addition, in the field of nursing education, interest in VR-based practical education as an alternative to ensure patient safety and compensate for insufficient clinical practice opportunities is increasing. Learners have the advantage of being able to practice and learn skills repeatedly in a safe clinical environment and virtual space [6].
In particular, disaster nursing practice is a field in which sufficient training and empirical learning are difficult because of physical and human resources limitations, such as implementing disaster situations and preparing for casualties. As the frequency and intensity of disasters increase worldwide, nurses should play a pivotal role as initial responders, severity classifiers, direct care providers, information providers, and educators throughout the entire disaster [7]. In general, patients requiring urgent first aid in emergencies account for 10~15% of all emergency patients, and prompt transport to a hospital where treatment is possible, depending on the severity, impacts the patient's prognosis [8]. Therefore, nurses need to pos-sess sufficient competence in disaster nursing triage programs. To perform this role, nurses must improve their disaster nursing capabilities, and as future medical professionals, nursing students need sufficient education to perform disaster nursing effectively. On the other hand, a small number of nursing colleges and institutions in Korea operate disaster education programs. Therefore, if experi-ential learning and training are achieved through VR-based simulation education for high-risk situations or practical education that is difficult to reproduce, it will be an effective educational method for strengthening the capacity to cope with disaster situations properly [4].
Currently, VR-based training programs developed for nursing practice are used when learners find it difficult to access, for example, core basic nursing practice education [9], patient triage in emergency rooms [5], and intravenous fluid injection education [10]. However, it is difficult to directly apply programs developed abroad because they dif-fer from domestic and reproduced sites and situations. Additionally, because development is often led by compa-nies specializing in VR technology, differences in nursing techniques and conditions necessary for the clinical field may exist. On the other hand, most programs developed in Korea are in the research stage, wherein researchers develop and measure their effects. Furthermore, programs for disaster nursing are insufficient. Thus, developing a VR-based disaster nursing practice education program through collaboration with a professional company is necessary and should be based on the needs of the academic nursing community. In this study, nursing students preparing for disaster nursing careers were trained to classify patients accurately and quickly according to a triage system. Triage is the process of quickly assessing patients, as-signing an appropriate severity level, and determining the facility where patients can receive treatment [11]. In foreign countries, 70~99% of the severity classifiers in the emergency room are nurses [12]; in Korea, the percentage is 76.5%[13]. Park [14] reported that nursing students need to learn triage before practicing disaster nursing simulation. Therefore, in this study, we tried to develop VR- based emergency patient classification content.
It is noteworthy that VR education occurs in an unfamiliar virtual environment reconstructed through a computer (not reality). Therefore, to achieve the learning goal, it is necessary for learners to evaluate the information presented in a given virtual environment and main-tain their motivation and concentration while interacting and gaining confidence and skill in their problem-solving abilities. On that account this purpose, Keller's [15] ARCS (attention, relevance, confidence, and satisfaction) Model of Motivational Design is widely used in teaching and learning design. Keller's problem-solving approach in-duces and focuses on learners’ interest in the classification process, relates what learners will learn by focusing on the necessity of the classification, and develops students’ confidence in their ability to solve problems on their own through learning. The model continues to induce learning motivation by creating satisfaction with the learning results [15]. Therefore, in this study, educational content was developed using the four concepts of ARCS proposed by Keller to achieve the learning goal of VR content.
The purpose of this study was to develop content for VR-based simulation education for nursing students regarding emergency patient classifications. The specific aims included the following: i) develop educational content for VR-based simulation of an emergency patient classification that can be used by nursing students in disaster nursing practice and; ii) conduct a usability evaluation among the study participants on the developed VR-based simulation educational content.
METHODS
1. Study Design
The aim of this methodological study was to develop a VR-based simulation for an emergency patient classification system and evaluate its usability by applying Keller's [15] ARCS model as a basic strategy.
2. Sample and Ethical Considerations
This study was conducted after obtaining approval (No: JBNU 2022-03-016-001) from the Bioethics Review Com-mittee of Jeonbuk National University institutional re-view board. To evaluate the usability of the content developed in this study, a research assistant who did not directly participate in the study recruited fourth-year students at the College of Nursing at Jeonbuk National University in Jeollabuk-do from May 22 to 31, 2022, through a departmental bulletin board announcement. The criteria for participation in the study were 1) fourth-year nursing students who understood the purpose and method of this study and 2) nursing students who chose and completed emergency disaster courses in their third year. Thirty people expressed interest in participating in the study in writing, and the research assistant provided specific information on the research procedure, purpose, and schedule. Interested students provided written consent. All 30 students participated in the study; there were no dropouts. Thus, 30 questionnaires were analyzed and a small gift was provided to thank students for participating.
3. Measurement
1) General characteristics
The general characteristics of the study participants were obtained from their responses to eight questions about gender, grade, age, current health status, college life satisfaction, nursing major satisfaction, and experiences with VR-based educational programs for non-learning purposes.
2) A tool for evaluating the usability of VR-based educational content
The tool for evaluating the usability of the developed VR-based severity classification educational content (eight questions) was selected with the approval of the tool developer and in consideration of the VR content usability evaluation tools developed by Jeong et al.[16]. The researcher developed additional questions after evaluating content and composition validity. Specific usability evaluation factors included the degree of focus on content, the reality of the VR, the confidence to achieve learning goals, and satisfaction with the overall content experience. The evaluation tool consisted of 45 questions: 11 related to attention, such as “ I was deeply immersed in VR content,” 17 regarding relevance, such as “ I felt like I was exploring something when I was doing VR experience,” six regarding confidence, such as “ I quickly reached the goal of VR content,” and 11 related to satisfaction such as “ I felt tired after the VR experience.” The higher the measurement score, the more useful is the developed content. Three nursing and two emergency medical professors verified the validity of the tool. Accordingly, the item-level content validity index (I-CVI) was .81~1.0. In this study, Cronbach's ⍺ reliability of the entire tool was .84, and in each subdomain, satisfaction was .80, confidence was .67, en-gagement was .67, and relatedness was .75.
4. Statistical Analysis
The collected data were analyzed using SPSS/WIN 26.0. First, content validity was verified using the I-CVI, and the reliability of the measurement tool was verified using Cronbach's ⍺. Second, user evaluations of the general characteristics and content were analyzed using means and standard deviations.
RESULTS
1. Development of VR Content
Educational content for classifying the severity of emergency patients’ injuries during disaster nursing practice was designed based on four elements - attention, relevance, confidence, and satisfaction - according to Keller's [15] ARCS model. For attention, an immersive VR simulation was used, and the relevance was increased by making the VR background, sound, and avatar similar to those in reality. It also increased confidence by providing re-learning and repetitive learning opportunities through reflection. It was designed to improve satisfaction by providing realistic experiences in safe VR for situations that are difficult to experience in person. The content development sequence was carried out according to the system development life cycle (SDLC) consisting of analysis, design, implementation, and evaluation stages. The SDLC is one of the main methods for developing systems for com-puters and mobile devices [17]. As mentioned previously, the study period was from November 2021 to June 2022.
1) Analysis of content development needs
Literature related to domestic and foreign disaster nursing and emergency nursing was reviewed to analyze the demand for disaster nursing practice. In addition, inter-views were conducted with three nursing professors who had been in charge of adult nursing and emergency medicine lectures and practice at the university for more than five years and two emergency medicine professors. As a result of the interview, it was confirmed that nurses' disaster nursing capabilities are essential in all areas due to the increase in various disasters at home and abroad, and for this purpose, it is necessary for students to acquire sufficient capabilities to properly cope with disasters. In addition, it was concluded that VR-based simulation practice is very appropriate as an efficient educational method to replace disaster-related practices in the current practical curriculum of nursing students.
2) Design content
Experts (three nursing professors and two emergency medicine professors) participating in the study confirmed that the most basic and high-priority content for practical training in disaster nursing was accurate and rapid patient classification according to the Emergency Patient Severity Classification System (Triage). Accordingly, in this study, the content was constructed based on simple triage and rapid treatment (START). This tool is the first taxonomy implemented at an accident site by first responders in a disaster situation to quickly classify multiple casualties according to severity. It is a tool that examines the casualties that have occurred and allocates them based on their conditions as immediate (red), delayed (yellow), minimal (green), and expectant (black)[18,19]. As this taxonomy is intuitive and easy to use, it is widely utilized as a severity taxonomy for patients in the field (Figure 1).
A team of three professors from the nursing department set the following learning goals for this educational program: i) to identify and report an initial mass patient out-break, ii) to learn how to classify severity given a large number of patients, and iii) to classify patients according to the patient severity classification method. According to the learning goal, a simulation practice scenario using START was developed by researchers at University A based on the simulation practice scenario for a hospital's response to mass casualties. In accordance with the specific scenario described below, a nurse dispatched to the scene as a disaster medical assistance team (DMAT) mem-ber attempted to classify 20 casualties at the accident site according to triage criteria in VR. This scenario was analyzed for content validity by three nursing professors who had been in charge of adult nursing and emergency medicine lectures and practice at the university for more than five years and two emergency medicine professors. As a result of the content validity analysis, there was no dis-agreement among experts, as this scenario was appropriate for achieving the learning goal.
< Situation Introduction >
At 9:30 p.m. on Wednesday, November 23, the Area A, Emergency Medical Center (disaster base hospital) was notified of the accident via the hotline of the Central Emergency Medical Center's Disaster Emergency Medical Situation Room. Onsite medical support from the DMAT team was requested because a multiple collision car accident occurred at the intersection 500 m from A University Hospital, injuring approximately 20 people. The DMAT of Area an Emergency Medical Center was immediately dispatched to the scene and the center began treating patients according to the response plan for mass patients at A University Hospital. The DMAT dispatched to the scene of the accident classifies the patients and plans to transport them to the hospital. A University Hospital declares that a mass patient disaster has occurred according to the response manual and is discussing whether to initiate activ-ities of the disaster management operation center.
3) Implementation
To implement the developed scenario, actual virtual room content was developed and pilot-operated through programming work involving VR developers. Figure 2 shows the user interface (UI)/user experience (UX) design of the scenario-based VR content. Using modeling, the simulation was conducted when the car had stopped and overturned or crashed on a mountain road in a virtual space. The casualties were scattered in several places. One could survey the virtual space using the head-mounted display (HMD) and a mouse (UI).
The preliminary VR content developed was reviewed by the 3 nursing professors, all of whom had more than 10 years of experience in emergency nursing lectures and clinical practice guidance at universities. Subsequently, the content was modified and supplemented. After modifying the size and type of font used, the final VR content was completed.
4) Evaluating content usability
To evaluate content usability, the research study participants, under the guidance of a research assistant, studied the developed VR-based educational content in the emergency and disaster practice room of C University in three stages. The first stage was the preparation stage: study participants were introduced to the device and its operation before learning and using the VR-based emergency patient severity classification content. The researcher ex-plained that cybersickness could occur when viewing im-ages using the HMD, and for safety management purposes, students were asked to sit in chairs while operating the equipment. Further, they were to use the tool only with a research assistant present. The time dedicated to the introduction and explanation was approximately 10 minutes.
In the second stage of simulation practice, study participants used the HMD to watch the video, find emergency patients in the VR practice space, and classify them according to the severity of their injuries. Study participants reported to the scene as members of the DMAT in a state where mass casualties had occurred in a multi-collision car accident. As a DMAT nurse, each study participant had to quickly triage 20 casualties scattered in various places and transport them to the hospital. The situation ended when the conditions of the casualties had been assessed and all 20 casualties were classified into one of four stages according to the triage system [11]. The study participants were conducted one at a time, after the situation ended, study participants checked the evaluation results to see if casualty classifications were performed correctly. This activity took 7~10 minutes.
The third stage was reflection: getting out of VR and reflecting on the questions, “ How did you feel after the simulation?” and “ Do you think simulations will help disaster nursing?” The researcher was a facilitator to allow study participants to reflect on and discuss their VR experiences with each other. The activity took five minutes to com-plete. After completing all practice stages, a usability evaluation questionnaire was disseminated, and the usability of the content was evaluated according to Keller's [15] ARCS model.
2. General Characteristics of Study Participants
Ninety percent (n=27) of the study participants were fe-male students, and the average age was 23.5±1.14 years. Furthermore, 96.7% (n=29) said they were in good health, 60.0% (n=18) said they were satisfied with college life, and 63.3% (n=19) said they were satisfied with the nursing major curriculum. In total, 80.0% (n=24) had no experience using VR-based education programs, though 56.7% (n=17) had played VR-based games, which was higher than the percentage of students who had not (Table 1).
Table 1.
General Characteristics of Partic
3. Usability Evaluation of Developed Educational Content
To evaluate the usability of the developed educational content for nursing students, attention, relevance, confidence, and satisfaction with the content were assessed. Attention to content averaged 3.76±0.41 points (out of 5 points), and relevance averaged 4.23±0.51 points (out of 5 points). In addition, after applying the content, confidence in learning was 3.81±0.60 points (out of 5 points), and overall satisfaction with the content was 2.52±0.72 (out of 5 points) (Table 2).
DISCUSSION
As natural disasters and new infectious diseases are rap-idly increasing worldwide, the role of nurses in disaster situations has been emphasized. Accordingly, in this study, VR-based simulation practice content was developed, and its usability was evaluated for nursing students’ disaster nursing training practice. This research was meaningful in terms of the development of content for a VR- based disaster nursing practice environment that is closer to reality than traditional simulation practice because of its use of existing high-fidelity simulators or Web-based online simulations. According to a prior study, simulation education using VR technology is an effective educational method for improving the ability to cope with high-risk clinical cases [6]. Therefore, the content of the program proposed in this study, which allows users to experience emergency situations at disaster sites in virtual spaces, is appropriate for nursing students as they practice disaster nursing.
In this study, at a preliminary expert group meeting agreed that the most necessary topic for nursing students in disaster nursing was accurate and rapid patient classification according to the triage. In Yoon's [20] study of 119 paramedics, 98.4% of the respondents said that the appropriateness of severity classifications conducted at emergency sites was less than average, and 52.4% said they lacked professional knowledge of severity classifications.
In the content development for this study, the concepts used as important strategies for achieving learning goals were attention, relevance, confidence, and satisfaction, as suggested by Keller [15]. The Korea Institute of Nursing Certification includes nursing competencies in disaster and emergency situations as learning outcomes for the current four-cycle nursing certification evaluation [21]. Nursing education institutions strive to strengthen students’ core competencies in disaster nursing by conducting problem-based learning, simulation education using high-performance simulators, disaster situation response simulation comprehensive training, Web-based simulation, and tabletop exercises for safety and disaster nursing [22,23]. Most of these programs have improved participants’ abilities to perform disaster nursing tasks, moti-vated learning, and instilled confidence for solving problems [4,22,23], but it has been suggested that various scenarios should be developed to convey disaster situations vividly [24]. Although each learning method currently in use has its advantages and disadvantages, the VR-based simulation program developed in this study can simulate field situations accurately. To increase the validity of the content, the scenario was reconstructed based on the scenario currently used by the emergency medical team for triage training at A University Hospital.
A usability test was conducted after applying the content developed in this study. In general, usability testing is an essential factor in the introduction of a new educational system that measures students’ ease of use and satisfaction with the system [25]. In this study, the basic strategy for developing content was based on the ARCS concept presented by Keller [15]; therefore, the usability evaluation was also measured by focusing on the four concepts of attention, relevance, confidence, and satisfaction. As a result of this study, the degree of attention to the developed contents was evaluated as 3.76 points out of 5 points. Concentration is the power to focus one's mind on a certain event or activity [26] and is similar to immersion, which is the feeling of digging deeply into or falling into a certain thing [27]. The degree of attention paid to the learning situation is a major factor in achieving the learning goal [11], and there are many reports that VR practice programs have improved immersion [4,27,28]. However, Abelsson [29] stated that if the form of the patient in a virtual simulation is not similar to the real thing, students feel anxious and do not concentrate well. Therefore, when de-signing a VR program, it is important to increase the degree of similarity with reality so that students can increase their sense of presence and concentration [4]. In this study, the researcher determined that it was possible to increase the study participants’ attention by using an immersive VR simulation by sealing each study participants’ audio-visuals using an HMD. This immersive simulation has been identified as an effective educational method in emergency or disaster simulations [20]; compared to high- fidelity simulation, it has advantages in terms of cost, space, and instructor time. Moreover, individual repetitive learning is possible [6]; therefore, it is necessary to continue to use it.
As a result of evaluating how much content is related to learning practical skills or goals to be acquired, the score was 4.23 out of 5 points, the highest among usability evaluation items. Dubovsky et al. [5] reported that applying VR triage simulation for emergency room nurses who were tasked with classifying patients was equivalent to the actual job workload in all aspects except physical exertion. Further, the study participants’ skills and experiences were the same as in the actual situation. Since the participants in this study were fourth-year nursing students who had experienced practice using a high-fidelity simulator or road training for disaster nursing in school, it can be interpreted that they felt the VR simulation situation was more realistic because they had a basic understanding of disaster nursing.
In addition, in this study, the degree of confidence after applying the content was 3.81 out of 5 points, the second highest among the usability evaluation items and in the same context as that in a study by Choi and Kim [27], whose meta-analysis demonstrated the effect of immersive VR simulation on college students preparing for medical careers. The effect size of performance was 0.59, which was moderate. Most studies applying VR simulation showed a positive effect on nursing students’ self- confidence [28], and learning retention improved over time [30]. Kim's study [31] revealed that education applying scenario-based inpatient management VR simulation was more positive for nursing students’ learning confidence than traditional practical education, reflecting the advantages of VR simulation, which minimizes the risk of accidents in virtual spaces and gives nursing students confidence in practical performance through repeated training at an appropriate level of difficulty. Overall satisfaction with the content developed in this study was 2.52 points out of 5 points, which was the lowest score among the usability evaluation items. General fatigue and dizziness were found to be factors that reduce satisfaction. This finding is in line with that of Choi and Kim [27], who performed a meta-analysis regarding the effect of immersive VR simulation on medical students. Interest in the content had a large effect size of 0.74, but satisfaction had a medium effect size of 0.38. Differences in satisfaction levels could be attributed to visual fatigue, dizziness, and cybersickness, which are associated with VR exposure, depending on the individual using immersive VR equipment [30]. This finding supports the result of applying VR content to nursing students in Kim's study [31] in which satisfaction levels were higher than the levels for traditional practical education, but 56.7% of the study participants reported that they experienced discomfort. Kim et al.[24] reported that study participants felt anxious and confused and had low satisfaction levels because of their lack of familiarity with the software and technical factors related to virtual simulation devices. In comparison, Jung's study [32], which evaluated intravenous injection simulator practice using VR and haptic technology for nursing students, found that the VR simulation group had the highest confidence levels. In contrast, the group that prac-ticed with both the existing intravenous arm model and VR simulation showed the highest satisfaction. VR simulations facilitate safe learning of nursing cases and skills that are difficult to experience in the clinical field. Good orientation and content usage time must be adjusted to overcome the inconveniences experienced by students, and measures are needed for students who are un-comfortable or unfamiliar with VR devices.
However, since the subjects of this study were students who had not directly experienced the disaster, it is still not sufficient to evaluate their relevance. Therefore, the usability evaluation of nurses working in disaster nursing should be repeated. In addition, an efficient practical training program should be established according to an analysis of the strengths and weaknesses of using man-nequins and models— which have been widely utilized in the nursing classroom— and the effects of VR simulation practice, as well as cost analyses of operations.
The scope of this study was to develop VR-based triage practice content and evaluate its usability, but its limi-tation is the inability to measure the effect of the developed content. Nevertheless, in this study, students' concentration and satisfaction are high, so it is suggested as one of the efficient learning methods to improve their disaster response skills.
CONCLUSION
This study was conducted to explore the direction of practical disaster nursing education by developing and applying VR-based emergency patient classification content for nursing students. After applying the VR content developed in this study, nursing students rated its relevance to reality the highest, and it was determined that VR content was effective in increasing students’ confidence in learning through repetitive training and implementation of situations similar to those in the clinical field. However, because there may be problems with the use of VR devices or one's adaptation to the virtual environment, the systematic development of VR-based content for various disaster environments will help improve the core competencies of nursing students.
PDF Links
PubReader
ePub Link
XML Download
Full text via DOI
Download Citation 
Print 
