Evaluation of an Augmented Reality for Historical Context Experiences of 3D Restored Court Paintings

Court painting is a traditional Korean type of painting that depicts royal events. It should be understood in the context of royal rituals and historical writing. Therefore, it is only possible for visitors of a museum with historical knowledge to interpret the content accurately. This problem is often identified in traditional paintings in other countries, and existing methods have attempted to deal with it using digital media technologies. Most methods aim to reconstruct paintings as 3D virtual objects for visitors to view them. However, these methods prevent visitors from interacting with the historical events depicted in the paintings and do not explain the essential historical facts and stories that the paintings leave out. In this study, we propose an augmented reality system that provides 3D virtual restorations of information necessary for understanding court paintings and supports historical context experiences of royal events. Specifically, we restore the information required for the interpretation of the painting to a virtual 3D object so that visitors can intuitively understand the historical stories missing from the painting. In addition, the system allows visitors to experience royal events through various interactions. The structural design combines experiential learning theory and digital storytelling so that visitors can be exposed to historical knowledge in a stepwise manner. According to the user study results, our system provides visitors with a more immersive experience and gives more accurate information about royal events compared to the existing method. This result suggests that the proposed system effectively communicates historical information and stories of court paintings to visitors who view paintings that require prior historical knowledge.


I. INTRODUCTION
Traditional paintings contain a significant cultural heritage, which enables us to learn about historical cultures [1]. Traditional paintings depicted features such as personal portraits and landscapes, but they also show historical events and subjects. Researchers in the field of traditional paintings [2], [50], [51], find that the existence of traditional paintings is tangible, whereas the intangibles are invisible and The associate editor coordinating the review of this manuscript and approving it for publication was Orazio Gambino .
include historical stories and the associated cultural heritage. They further argue that more important than the materials, such as the paper, silk, and paint that make up traditional paintings, are the production techniques, historical facts, and cultural values inherent in traditional paintings. For this reason, it is difficult for people with little knowledge of the history hidden in traditional paintings to interpret and understand them accurately.
With regard to court paintings during the Joseon Dynasty in Korea, this point is even more strongly emphasized. Court paintings are traditional paintings in which skilled artists recorded royal events during Joseon, approximating a ''document'' [55]. These paintings were produced for a documentary to honor royal events such as royal weddings, the queen's 60th birthday, and military training events. These paintings should thus be understood in the context of royal ceremonies and historical literature rather than being discussed in terms of their artistic merit alone. Hence, visitors who do not know enough about history find it challenging to understand court paintings and to grasp their meaning. Discussing ways to help people appreciate and understand court paintings is therefore necessary.
As improvements in digital media technologies have taken hold, most museums and art galleries have introduced a range of visualization techniques to give the audience a better understanding of these types of paintings [3], [4]. For instance, a number of traditional paintings have been transformed into a panoramic view of high-density video images or animations with stereo sound for the audience [20] [21], while in other cases, digital canvases for interactive engagement with the paintings have been provided [52], [53], [54]. However, despite the various techniques introduced to provide audiences with a clear and detailed view of the paintings, audiences have not been provided with accurate historical context and information about traditional paintings.
Augmented reality (AR) has also been studied to support the appreciation of traditional paintings. Most studies have adopted AR based on images. These systems are designed so that users can receive various forms of information about the images. For example, the early AR systems made for traditional paintings [5], [6] were designed so that an informative user interface (UI) panel of the image is included next to the actual painting. These systems provide users with the production time, production technology, and a brief explanation of the painting next to the original painting through the UI panel. Another system that, combines an art appreciation map and AR [7], includes voice commentary to assist museum visitors in understanding paintings. Furthermore, the AR system includes a touch screen with additional explanations of the paintings [8] or provides information such as the appearance of the other side of the image which the visitor cannot usually see [9]. Other researchers [10] have developed a system that creates an environment similar to that depicted in a traditional painting using a 360-degree-based 3D animation and presents it as virtual reality. However, in the AR environment created in the initial research described above, the historical space and story expressed in the painting cannot be experienced in detail. Also, finding information about the person or cultural heritage, omitted in the images, is impossible. They described above did not include an exhibition design aimed at helping visitors understand and learn about the paintings.
In this study, we design an AR system that, allows users to learn and experience history on their own using a military training image [55] among the Joseon Dynasty's court paintings (CP): ''Royal Visit to Hwaseong,'' produced in the 17th century. This study aims to support visitors' appreciation of paintings by providing specific historical stories and information about the CP.
The contributions of this study are as follows: • Designing to provide accurate information about the CP and related historical stories • Understanding the CP's historical space, creating 3D models for expanding the user experience space, and implementing marker-free AR to provide a natural, historical context • Virtual restoration of omitted content (i.e., objects, people, and spaces) in the CP aspects of court paintings • Encouraging users to learn systematic historical information using experiential learning delivered via theory-based digital storytelling • Identifying the value of the proposed AR-based CP experience through a user study The functional, general, and empirical requirements for using CPs with the AR system are determined for the purposes of this study. The AR system developed to meet the requirements was evaluated by 34 participants in an experiment. Comparing the systems in previous studies with the proposed system showed that our work provides a better learning method and more exciting experiences. Furthermore, the results suggest that the proposed system can be used as a means of learning and experiencing paintings such as traditional paintings with stories pertaining to them.

A. EXTENDED REALITY (XR) CONTENTS IN TRADITIONAL PAINTINGS
Existing methods have proposed an immersive exhibition based on an interactive system that allows visitors to view traditional paintings [2]. The exhibition method [20], which uses panoramic images to show the entire image of traditional paintings, allows visitors to explore high-definition images in a 360-degree direction. Ma et al. [21] also used highresolution images of panoramic and added stereo sound to traditional painting exhibitions, allowing visitors to become more immersed in the paintings. One study used 42 projectors to add 3D animation and music to traditional landscape paintings in an immersive exhibition [22]. The screen-based 360-degree experience method introduced in previous works is beneficial as a simple approach, allowing users to inspect the screen from various angles. However, because these studies only show high-definition images and simple interactions, they cannot explain the story of the painting. Therefore, the details must be provided in a separate text. To compensate for these problems, Alexander et al. [23] suggested an interactive art-based system that lets visitors search for traditional paintings by subject; the system piques the interest of the user in the paintings, increasing their curiosity about them. However, this method does not give visitors a sense of immersion and instead only provides information.
Other studies have used AR and VR (virtual reality) to augment the viewing of traditional paintings. A method that uses VR for traditional Chinese paintings [24] developed eleven scenarios to reproduce the Great Canal represented in a figure and restored the appearance of the ancient region using computer graphics and animation. The AR system in a museum exhibit [25] added virtual screens with cameras that could recognize traditional paintings. This method gave visitors a new way to see the paintings. The back of the painting could also be observed to see the artist's signature or explore the image-generation processes. Kolstee and Van Eck [9] that reproduced a painting by Van Gogh with AR drew interest from visitors as the object in the painting is provided in a 3D format, relying on camera recognition of Van Gogh's paintings. However, these studies did not show the story in the picture as an animation or virtually restore omitted objects in a 3D format. The system in this study aims to provide details about CPs in a 3D format in an AR environment and to present historical stories as animations.

B. 3D VIRTUAL RESTORATION OF CULTURAL HERITAGE (CH)
The virtual restoration of CH is becoming increasingly important in the field of archaeology, influenced by the development of various forms of visualization media [26], [27], [28]. Laser scanning and photogrammetry are mainly used to create digital records of CH (e.g., buildings, artifacts) or existing sites [32], [35]. Regarding Monreale Cathedral in Palermo, Italy, photogrammetry and terrestrial laser scanning were used as methods to construct a 3D model that can also be used to present heritage or historic building information modeling for the main portico [30]. St. George's Church in Croatia virtually restored the curved parts of the round ceiling in a 3D format using laser scanning equipment to record images and structures of late medieval murals painted on the archshaped church ceiling [31]. Rizvi? et al. [34] also reconstructed destroyed or damaged CH using a crowd-sourced photogrammetric method.
On the other hand, some CH does not exist and is only seen in pictures and documents. In such cases, reliable historical records and reinterpretations by archaeologists play a significant role [29]. In particular, restoring damaged cultural heritage and studying visualization methods are essential for sharing cultural heritage that has disappeared with humanity again. Digital and historical experts use various research methods to restore historic buildings, landscapes, and artifacts in 3D.
Guidi and Russo [36] developed a diagram showing how they can use historical records (e.g., historical documents, pictures) and 3D data to rebuild historic buildings. A study to reproduce virtually historical and cultural areas that have disappeared [34] reconstructed the topography of a historical site in 3D using LiDAR. The equestrian monument of Francesco III d'Este [11], made in 1774, was destroyed and disappeared. However, the giant statue was reconstructed using 2D and 3D historical information based on small replicas and remaining large images, prints, and historical literature. Furthermore, 3D restored virtual mounted artifacts were implemented as AR at the original locations specified in the historical record. Another study [33] that attempted virtually to restore a Greek waterwheel also used AR to show how the waterwheel was designed and made.
In this study, characters, objects, and spaces missing from CPs were restored in a 3D format. Virtual restoration of objects omitted from figures can augment our understanding of these paintings.

C. DIGITAL STORYTELLING OF CH
Digital storytelling is a combination of the story, images, audio, and text. This represents multimedia as the traditional way to tell a story [19], and the framework design determines how well digital storytelling works [37]. Reitmaier and Marsden [38] noted that the most crucial aspect of digital stories is the actual story, explaining that this means more than ''images with descriptions.'' In addition, it was confirmed through a user study that users are delighted when experiencing digital storytelling-based content on a mobile system. Oberhuber et al. [39] also confirmed that using digital storytelling in AR games for children can provide them with high satisfaction. A study that developed children's mobile games in public libraries [40] used digital storytelling for systems for interaction with children and confirmed its usefulness.
In CH, digital storytelling is widely used to provide historical information. Creators of the AR system [42] of the Calouste Gulbenkian Foundation gardens consider that the role of the story is essential to explain the various features of natural heritage. The system proposes a framework that combines the aesthetic features of natural heritage sites with storytelling. Haahr [43] devised a way to integrate CH content and literary works. They propose a system to apply a location's story to a location-based game. In addition, Shin et al. [41] used storytelling to provide point-of-interest (POI) information about cultural sites in AR systems. Their research proved that storytelling-based information can be more effective than information-retrieval-based information. Ketchell et al. [37] studied how to apply contextual storytelling after implementing an AR system that supports SLAM [19], [20], [21] on mobile devices. Furthermore, a user study confirmed the usefulness of digital storytelling applied to the system.
Previous studies have investigated how to use digital storytelling to provide specific information about historical and natural heritage sites. However, digital storytelling has not been used for exhibitions or for system experience structural design. This study uses digital storytelling to design the experience structure for CPs.

III. METHODOLOGY
This research aims to motivate users with little prior knowledge of CPs to become interested in CPs and to learn naturally through the proposed AR system. For the user to use this system successfully, the requirements were organized based on the characteristics of CP.  The subject of this paper is the military training painting of CP, ''Royal Visit to Hwaseong,'' of the Joseon Dynasty. This painting depicts the king's military training with soldiers in the spring of 1795, during both the day and night. The characteristic of this painting is not that it depicts a moment but that it summarizes all times at which the training took place. In other words, much information is implicitly expressed in a single painting. For example, the painting contains images of soldiers participating in military training, images of areas transformed by oriental painting techniques, and information about historical buildings. Because it contains various forms of contextual information, it is necessary to interpret the picture and determine what information should be added. Therefore, historical research is required based on historical data related to this painting.
As a result, the requirements for learning and experiencing CP through this system are as follows: • Understanding the contents of history without the need for prior knowledge of CP • Exhibition planning to help understand the historical space in CP • POI areas, building enlargements, and human arrangements in the figure This image depicts training both during the day and night. However, our system focus on the nightly military exercises. The nightly exercises used torches, lanterns, and sounds to deliver the king's orders to the soldiers. The historical record has ten training courses, but they are combined into five core courses. Courses that overlapped are left out.
The key locations of the training can be seen as the command post where the king is located (Seojangdae) and the wall where the soldiers stand. Therefore, the system is designed to reveal the characteristics of these places.

2) GENERAL SYSTEM REQUIREMENTS
In order to achieve the goal of this study, we have devised a way to implement an AR system that accurately depicts the unique characteristics of CP. We believe that creating an image-based AR augmentation method to inform users about the historical space of CP would be challenging. Instead, we have created a model of the place depicted in the CP, and built an AR system that recognizes this model through camera recognition. This augmented method allow users to become more immersed in the system. To create the model, we collaborated with traditional landscape and topography experts. Our system is designed as a marker-free AR type that recognizes the model's features. Examples of implementing marker-free AR systems can often be found in CH AR systems, where electrical signals such as an iBeacon [12], [13] are primarily used as visual SLAM methods [14], [15], [16]. To build our AR system, we use MAXST's SDK powered by Google ARcore, which is optimized for mobile equipment environments compared to the existing visual SLAMs. Additionally, our system is implemented via Unity 3D and works on Galaxy Tab S7+. Moreover, it meets the following requirements: • Places natural 3D virtual objects on top of the model when the AR is running • Provides unshakable searches of matched virtual objects in the model • Provides UI according to the training course order and playing related animations • Synchronizes training-related sounds and operating an internal camera • Creates metadata related to individual instances of information • Identifies individual instances of information related to virtual objects through the UI design • Optimizes the system to reliably view large numbers of virtual objects

3) REQUIREMENTS FOR SYSTEM EXPERIENCE DESIGN
The system's experience process is designed to provide an immersive CP experience through digital storytelling. Principles and historical story analysis are necessary for exhibition planning with digital storytelling. We design this study while considering two aspects of digital storytelling. The first is the user's natural and exciting approach to experiencing CH. Paintings that can be seen in a museum cannot easily attract the interest of all visitors, and the contents are difficult to interpret by non-experts. Therefore, experience as it pertains to this study should induce user curiosity and convey interest. The second aspect is easy access and understanding of complex information. The research object of this paper implicitly contains unique historical information that is not general, from simple object information to complex training contents, as a result of 'recording'. However, if historical information is delivered for clarification, it can reduce the psychological distance between the CP and the users. This intention is also related to the first requirement.
In this study, because CPs are produced for recording purposes, akin to ''documents,'' it is assumed that experiential language learning theory (experiential learning theory) [17], [18] could also be applied to apply methods by which to understand CPs. Thus, we design the experience structure of the new AR system by combining the content of digital storytelling and experiential learning theory.

B. IMPLEMENTATION
The system architecture is constructed based on the three requirements described above. Figure 1 shows how the different elements connect and work within our system.
• Based on proven historical data, virtual 3D restorations (e.g., clothes, objects, buildings) and a physical 3D restoration (e.g.,site) are conducted  • For use in AR applications, digital assets including 3D virtual models, animations, audio sources, and so on are produced • When the built-in camera of the tablet device recognizes the physical site model, the AR scenes are augmented and displayed • For the experience of the AR application, four scenes (i.e.,'Intro,' 'Indirect Experience,' 'Information Acquisition,' and 'Direct Experience') are implemented and offered to users The system is implemented using the Unity 3D platform and the SLAM-based AR SDK(MAXST's SDK). The user's AR experience begins with the system being operated when the AR camera recognizes the actual model produced based on the CP.

1) DATA MANAGEMENT
The data sources used to build the system for this study are managed and put in order shown in Figure 1. The managed data are auxiliary materials that can interpret the CP and data newly generated by research progress.
First, the data used in the study are classified into basic, research, and generated data. Basic information includes the CPs, old documents with explanations (called ''Uigwe''), and images on scrolls and folding screens. Research data include ancient documents, images, historical materials, topographic drawings, architectural drawings, videos, and voices with related records, and refer to data that can interpret basic information in CPs. Finally, the generated data are the results calculated during the research process, such as JSON files, C# scripts, images, videos, CAD drawings, SketchUp files, and 3D files.
The accumulated data are organized into a list, and each record generated metadata.
• Data for implementation of a system: The organized data are summarized mainly with important information and then written in a JSON file, as indicated in Figure 1. The information entered in JSON consists of the 'name', 'group', 'bag', 'image', 'imginfo', 'imgtext,' and 'objectset'. When a user selects a virtual object, the UI displays the information existing in the JSON file (e.g., basic information about the virtual object, the form drawn in the CP, and related components). The UI of Figure 2 providing the information is divided into two columns, and organized information can be checked on the right. The left side is also intended to rotate the selected 3D object 360 degrees.

2) 3D VIRTUAL RESTORATION
The 3D data provided by the system of Figure 1 are reliable data generated based on the research of historical experts in VOLUME 11, 2023 39201 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.    each field. In order to convey accurate information to users, we study everything related to CPs, such as the characters' clothes, the tools and flags they use, the weapons they carry, and the landscape and buildings in the background. We then undertake a virtual restoration of the contents. Therefore, generating data varies slightly depending on the research subject, but the process of producing it after being verified by a history expert throughout the 3D data generation process is similar (Figure 3). Table 1 shows the types of 3D virtual objects provided to the user in this system.
• Costumes, tools, flags, and weapons: there is much omitted information in CPs. Typically, during the Joseon Dynasty, the king was not painted in CPs because paintings of the king were considered identical to the king. Moreover, many characters and tools were omitted from these types of paintings. The locations of the figures are also not accurately drawn.
Therefore, in the system, we restore historical facts that are relatively less represented in the pictures. Of these large numbers of virtual objects, only a few tangible instances of CH exist. Costumes and tools of great historical value have been destroyed or lost due to significant events such as wars and colonization. However, related information is recorded and kept during the Joseon Dynasty in Korea for future generations. Based on this data, we can restore the relics of the past virtually.
Thus, our virtual restoration process is carried out according to verification by historical experts based on accurate past records and artifacts. First, the experts decided which people, clothes, and weapons should be brought back to life on our system. Then, suitable shapes are created based on old documents, traditional painting materials, and other cultural artifacts. In addition, errors in forms generated with the advice of another history expert are identified, corrected, and produced with 3D data.
As a result, we virtually restore 1,014 objects, including 50 types of figures such as kings, subjects, and soldiers; The virtual objects of this system are manufactured with the 3Ds Max program. Each is produced with mid-range polygon data (4,000-5,000 polygons) in consideration of each character's face and animation characteristics. The textures of the virtual objects are produced according to the colors, patterns, and materials used. In addition, skeleton animation is utilized to position bones when applying animation to the modeled persons.
• Buildings: In the background of the CP used in this study, there are four large gates and a castle consisting of significant buildings to defend the area. The fortress included in this castle, the king's command post (Seojangdae), and the king's palace (Hwaseong Haenggung), where the king was in charge of military training, are the background in our AR system. 3D virtual restorations are done based on data surveys of these buildings and the results of research by experts. This process is conducted with 3D modeling and included drawing arrangements of validated historical data, CAD 2D drawing fabrication, 3D SketchUp utilization visualization, and comparative verifications with actual buildings shown in Figure 4.
• Historical Landscape and Topography: The historical area in the CP used in the system is restored to an AR target model and a 3D virtual space that users can check inside the AR system ( Figure 5). The actual shape of the historical area is similar to that of a circle, and it is different from the background of the CP (long vertically and short horizontally). This is done because the court artist drew it in a way that distorted the view, and the look of the land, which cannot be seen directly, is based on the artist's imagination and best guess.
We restore the distorted terrain to help us understand the CP, although it differs from the actual appearance of the historical areas. Moreover, we worked with historical terrain researchers to create an actual model. This work is carried out in the following order: data analysis -topographic composition analysis -GIS utilization planarization -CAD drawing work -mock-up production. . This is a schematic diagram of the system experience structure. The final system experience structure was created by designing the experience stages inspired by the experiential learning theory and the essential content of the storytelling-based CP for each stage.
The model, 180cm wide and 90cm long, was converted to 3D data to be used as the background for the virtual environment. The method of augmenting only virtual objects without augmenting the 3D topography is used because it has been confirmed several times that virtual objects are not positioned at the exact coordinates of the actual model. In order to prevent the position of the augmented virtual object from being distorted during the operation of the AR system, the system is created so that the 3D topography could completely cover the model. A photogrammetry method is used to generate 3D topographic data. First, we created a point cloud and removed noise by adjusting the bounding box for the photos in the model. The 3D topography generated in this way could be stably confirmed in Unity. The final environment was completed by placing trees, fortresses, the king's palace, and the king's command post (Seojangdae). This virtual terrain became the overall background of the AR environment.

3) SYSTEM EXPERIENCE IN DESIGN
The system's overall structure is designed such that users can easily understand and learn complex information without any additional prior knowledge of the CP. For the design of the experiential structure, digital storytelling and experiential learning theory are combined and adapted according to this research system. The method used is described as follows.
(a) Choose three types of experiential learning theory and combine them with CP storytelling. (b) The type of experiential learning theory chosen determines the flow of the experience, while the main contents of CP storytelling determine the details and specifics.
The experience structure designed in this way can be found in Figure 6. And the specific details of the designed structure are shown in Table 2. Experiential learning theory [22], [23] defines learning as a continuous process based on experience with the concept that knowledge is created from the learner's experience. The four types of experiential learning theory are specific experiences (motivation for new experiences), reflective reflection (observation of new experiences), abstract conceptualization (new ideas), and positive behavior (competence used in the field). This theory can motivate users to access and learn new information more naturally. The types we used here are specific experiences, reflective observation, and active experimentation. A new interpretation of these types was designed as a three-stage experience (New awareness and engagement, New acceptances and observations, Learning cultural content based on understanding). By combining this theory with CP storytelling, the design sought to ensure that users have experiences based on historical facts in stages. First, the storytelling of the CP, which will be the core content of the experience structure, was constructed. The historical information and stories included in the picture are summarized, and the critical elements of the organized content were derived.
The main contents were summarized as follows: (1) The value of the CP that recorded historical events, (2) the historical terrain, buildings, customs, objects, and information on the figures; and (3) the specificity of night military training conducted in a specific area.
The experiential structure designed based on these details is as follows.
• New awareness and engagement: The 'Introduction' part provides essential information. At this stage, it is possible to check the historical value of the CP that recorded historical events and the intention to produce an AR system with simple text and pictures.
• New acceptances and observations: This step provides 'indirect experience' and 'information acquisition'. The CPs contain detailed records of the figures, costumes, objects, buildings, and topography. In the stage of 'indirect experience', the user can zoom in on the CP, select parts containing objects in the painting and observe the information about those parts. It was impossible in museums and art galleries to enlarge the displayed paintings or check only certain parts of information without a guide. We design this system to help users understand the image before the AR experience and to allow them to relate the picture to the 3D virtual object during the AR experience. The stage of ''information acquisition'' is designed so that visitors can check the information expressed or omitted in the painting by selecting a 3D object when the AR is operating.
• Learning cultural content based on understanding: Finally, the stage of 'direct experience' is designed so that users can experience the historical events occurring in the CP. This represents the essential experience as the last step in constructing an AR system. Users can experience historical events in order through 3D animation.
This experience structure is a format in which the content, story, interaction, and animation effects are provided as the user experiences each stage. Therefore, it is possible to gradually deepen the user's understanding, learning, and experience of the information and story of the CP.
This experience structure consists of four AR system scenes ( Figure 7). First, the two scenes were designed as 2D screens instead of AR environments. Therefore, scene 1 (Figure 7 (b)) is provided as a 2D image slide in the form of an 'intro' step. In this part, users can check the production intention of the proposed system, the general prologue of the CP, the contents of the historical evidence, and the research members who participated in the study.
Scene 2 (Figures 7 (c), (d)) is the indirect experience step and is also provided as a 2D image. It is designed to allow users to select and learn screens autonomously by adding interactions to each screen. For example, users can zoom in to an image or check information to help interpret the images. Information to help users interpret the image is provided through the UI at the bottom when the user selects the actual location of the painting. Scene 3 (Figure 7 (e)) is the step in which information in the AR environment is acquired. When the AR system is activated, a virtual environment of late night with the torchlights of night military training is created. Each frame generates a ray that detects collisions with virtual objects from the camera, allowing users to observe their desired information, such as the architecture, historical figures, and topography applied with a Collider (one of the Unity component functions) with a single touch. In addition, an enlarged area is established to check the as-established important location (e.g., command post) so that a part of the terrain could be expanded when a user selects it. Scene 4 (Figure 7 (f)) is also provided in the AR environment and is designed so that the user can experience the night military training depicted in CPs. Again, 3D animation, particle functions, and timeline functions are used to implement the night military training, and visualizations were designed to suit the training process.

4) ANIMATIONS OF HISTORICAL EVENTS
In scene 4 (Figure 7 (f)), the following items are considered to implement historical events as animations.
• Scenario: Five critical military exercises were selected from the night military training stories in the CP. The selected training courses are as follows: ① Training to light torches placed in the castle; ② training to extinguish the torches, ③ training to light up the entire background, including the village; ④ training to shoot guns and cannons throughout the castle; and ⑤ training to turn off lanterns placed in villages and fortresses.
Because this training is done at night, it is characterized by the use of torches and cannons for an exchange of signals even out of the line of sight so that the training can be carried out in the correct sequence over a wide area. Therefore, the lighting effect was actively used to express the training process.
• Lighting effect direction: For the dramatic production of the training process, the lighting of virtual objects was directed using the Unity program, and particle effects were used for each virtual torch and lamp. The virtual Sam-an gun, which was used as a signal for military training, used particles that resemble smoke at the time of firing and particles that light up when torches and lights are lit or being used. When expressing virtual guns and cannons in the castle, we used explosive particle effects and firework particles provide by Unity.
• Animation setting: The night training course to be shown in AR as set in the scenario had to express the training involving the lighting of torches, the shooting of guns and, cannons and the use of firecrackers in sequence, and the extinguishing of torches at the same time. The existing animation production method in Unity can be used to write a script, but the timeline function in Unity was used here to control each animation simultaneously. Therefore, to ensure a realistic animation production by the system, the sequence of animation, audio production, and particle production of individual virtual objects was adjusted and used using the Timeline component provided by Unity.
• Internal camera setting: In the AR system, the background topography is vast, and there are many virtual objects positioned within it. Hence, if users do not move close to the scene, all objects will look small. Accordingly, we set up the internal cameras in the lower left corner of the screen to illuminate the main animation progression area. Cameras were positioned in critical areas and made to operate in order according to the training process. Therefore, users can check the overall screen and details on the screen of the military training process.

5) OPTIMIZATION OF VIRTUAL OBJECTS
The performance of a 3D application depends heavily on the number of polygons that the GPU has to process per frame VOLUME 11, 2023 8. User study progress: a) comparative AR system [49] experience, and b) system experience that we propose.

FIGURE 9.
Comparison of AR systems [49]: AR scene and brief information description.
in a 3D scene. In the AR system in this study, frame delays occurred because Unity processes many 3D models, such as buildings and people with high numbers of polygons. To solve this problem, we show a complete model only when the AR camera is close to the virtual object and apply a level-of-detail (LOD) function that increases the performance by reducing the polygon level of the model when the camera is far from the virtual object.
The LOD strategy refers to an adjustment of the detail level of the model according to the distance from the camera. For example, for the high-detail Seojangdae (the king's command post), the Blender program produced a version that reduced the number of polygons by 50% and a version that reduced the number of polygons by 20% while maintaining the shape of the Seojangdae building as much as possible.

IV. EXPERIMENTS AND RESULTS
A user study is conducted for a quantitative and qualitative evaluation of the immersion, learning, and usability aspects of the CP AR system. The user study compared the AR system and an earlier research system proposed by the authors. This study involved (1) collecting quantitative data that can be used to compare the two systems through a questionnaire and (2) interviews about the proposed AR system. There were 34 participants in the experiment (male = 16, female = 18) who were between their 20s and 40s (26 people in their twenties, seven people in their thirties, and one person in their forties). These participants were familiar with traditional culture but had little prior knowledge of the CP used in this experiment. Therefore, we devised the following hypotheses: • Our system provides better immersion and experience with CPs than existing systems.
• With a different exhibition space configuration relative to those of the existing system, our system enhances users' understanding of the background of the CP.
• Compared to the existing system, our system delivers better information about the stories and characters of the CP.
• Compared to existing systems, the experience structure of the proposed AR system is more helpful with regard to understanding the CP. Participants in the experiment experienced the comparison target system and the proposed AR system in turn (Figure 8). This process required approximately 15 minutes for each step. After completing the experience sequentially for 30 minutes, a questionnaire about the system is filled out and an interview is conducted.

A. EXPERIMENTAL CONFIGURATION 1) SELECT A COMPARISON AR SYSTEM
For the user study of the proposed system, an AR system similar to those used in previous studies is selected for comparison. Regarding the system to be compared, Table 3 summarizes the augmented methods, user devices, and information provision methods of the previous studies. The summarized items are as follows:  The AR system selected for comparison is that of Kim et al. [49]. This system is designed to allow the AR experience of a CP depicting one part of a royal event in the mid-18th century of the Joseon Dynasty of Korea (Figure 9). The system works similarly to previous studies and provides simple historical information. In addition, users can understand information about the painting before the AR experience begins. However, the historical information provided by this system is of a smaller volume than what we provide and does not contain any hidden historical stories.

2) QUESTIONNAIRE CONFIGURATION
The questionnaire produced for user research consists of a total of 22 questions. It was distributed to participants who experienced both AR systems. Table 4 explains the questions given to the participants in four sections. Section I is reported via the simple selection of items, and the remaining sections utilized a seven-point Likert scale. Section I collects information from participants to determine their age, gender, related knowledge level, and AR experience. Section II asks about their interest in and satisfaction with the AR experience and whether it engaged them in the exhibition. In section III, participants are asked about their understanding of the historical space through the exhibition space, the overall story, and the individual characters. Finally, the Section IV evaluate whether the experience order and pre-provided information improved the understanding of the traditional conversation to confirm the effect of applying the experience learning theory and storytelling. Detailed questions per section were prepared by referring to the user evaluation items of AR usability evaluations [44], [45], [46], [47] and understanding of CH paintings [7], [10] related to CH AR, VR, and the web environment as specified in previous studies.

B. DATA ANALYSIS RESULTS
This study aimed to identify meaningful differences in the improvement of immersion, learning, and usability compared to previous studies when users use the proposed AR system. We analyzed the score distribution using an independent-sample test and a correlation analysis to confirm any differences. Figure 10 shows the average score distribution based on the experimental results.

1) INDEPENDENT SAMPLE TEST RESULTS a: IMMERSION AND SATISFACTION
First, we evaluated the immersion and satisfaction characteristics of the two systems. For four of the five Immersion & Satisfaction items (Table 5), our system scored significantly higher (M = 6.35, SD = 0.693) than the comparative system (M = 5.54, SD = 0.396) (t = -5.967, p = 0.000). Looking at the specific questions, we found that our system received significantly higher scores on Q1 (the AR system experience was interesting/enjoyable), Q3 (natural expression of 3D graphics), Q4 (the exhibition environment enhances immersion when using the AR system), and Q5 (I would re-experience a similar system if given the chance). In particular, for Q4, we found that the exhibition environment in which the model was placed increased the user's immersion level (M=6.68, SD=0.534).

b: UNDERSTANDING THE CP (USABILITY AND LEARNING)
One of the important objectives of the proposed system is for users to understand CPs better. This evaluation was conducted with six questions. Our system received better ratings (M=6.72, SD=0.274) than the comparison  [49] M = 4.29) were found in the detailed questions (Q6 (the entire story of the figure is grasped by the use of the AR system) and Q7 (the space understanding of the figure is good). It is therefore confirmed that the proposed system can deliver details and stories contained in CPs to users as intended. It also scored relatively high on the details check

c: EXPERIENCE LEARNING THROUGH EXPERIENTIAL LEARNING AND DIGITAL STORYTELLING
In order to experience the CP effectively, this study designed the AR system experience structure by combining digital storytelling and experiential learning theory. The subjects were presented with four questions from the 'Experimental Learning' (Table 5) item to assess this. As a result, the proposed system (M = 6.52, SD = 0.360) received better reviews than the compared systems (M = 5.41, SD = 0.737). Furthermore, the response results of the detail item section confirm that the AR system experience sequence (Q12: M = 6.56, SD = 0.560) can help users to understand CPs. In addition, it was found that the preliminary information provided by the system helped users to understand the 3D virtual objects (Q15: M = 6.65 and SD = 0.485).

2) CORRELATION ANALYSIS
We analyze the correlation between the responses of questions about the proposed system. Among all questions, responses to questions excluding basic personal information such as age and gender are analyzed. P1 refers to the question about how much users know about the specific CP used in this study. P2 asks whether the user has experienced an AR system before. The correlated results within a significant range (p<0.05) are displayed in the yellow box in Figure 11.

a: BASIC INFORMATION
As a result of the analysis, the degree of knowledge about CP and the question focusing on identifying the relationship between the 3D objects and the picture showed a positive correlation (r=0.4). It was confirmed that basic knowledge helped used to identify the association between this CP and the 3D objects. The response to the AR experience showed a strong negative correlation (r=-0.9) with item Q2 (easy to operate when using the system). It is therefore noted that subjects without AR experience felt challenged when operating the proposed system.

b: INTERESTING AR SYSTEM
The questions that showed a positive correlation with Q1 among the individual questions of immersion and satisfaction are Q2 (r=0.3), Q3 (r=0.4), and Q5 (r=0.5). It was confirmed that if the proposed system is easy to operate and the graphic expressions are natural, interest in the experience increases. In addition, this result means that users are willing to experience AR systems similar to this in the future. The items that show a high correlation with Q1 were Q11 (r=0.4) and Q12 (r=0.6); these factors positively affected system commitment due to the increased level of understanding and insight into the CP realized, through AR systems.

c: RE-EXPERIENCE OF THE AR SYSTEM
It was confirmed that better immersion in the exhibition environment (Q4) affects the intention to re-experience the AR system (Q5) (r=0.4). In addition, Q6 (Assessment of Understanding of CP's Overall Story Question) and Q7 (Assessment of Understanding of the Historical Places depicted in Figure) were positively correlated with the question of whether the user would experience a similar system again (r=0.4). Likewise, it was confirmed that the AR system experience order influenced the intention to re-experience. When users felt that the experience order helped them to understand the painting (Q12), their expressed desire to re-experience it showed a strong positive correlation (r=0.5). It was confirmed that the more users feel the system helps them to understand 3D virtual objects by providing preliminary information in advance (Q15), the more positively it affects their intention to re-experience the system (r = 0.4).

d: UNDERSTANDING OF THE CP BASED ON THE EXPERIENCE METHOD
We also checked whether the experience method influences the users' understanding of the CP. There was a positive correlation between the question (Q12) that evaluates the effect of the AR system's experience process sequence on users' understanding of the painting and the question that asks about the degree of understanding (Q13) according to prior information (r=0.4). In addition, Q12 was correlated with the evaluation of the comprehension of specific target information (Q14) about the CP (r=0.4). This shows that the AR system experience sequence can have a positive effect when users attempt to understand objects shown in CPs.
Questions (Q10, Q11) evaluating the understanding of the CP through 3D characters showed a positive correlation with the provision of preliminary information (Q13) of the figure  (r=0.4). Likewise, Q13 showed a clear correlation between question Q9, evaluating the relationship between the 3D virtual object and the picture, and question Q8, evaluating the satisfaction with the provision of information about the specific target (r=0.4).

3) DIFFERENCES IN RESULTS DEPENDING ON THE PRESENCE OR ABSENCE OF THE AR EXPERIENCE
Among the detailed items pertaining to immersion and satisfaction in the evaluation, Q2 ( Figure 10) asked whether navigation was easy when using the AR system. As shown in Table 6, regarding this item, the proposed system received a lower rating than the compared system [49] (M = 6.38, SD = 0.493) (M = 5.29, SD = 0.932). The subjects found it difficult to manipulate the proposed system when using it. Several items among the survey answers were reviewed to find the cause of the low evaluation. A correlation analysis confirmed that it was related to the detailed question about basic information on 'AR experience.' Of the 34 subjects, 19 (55.9%) had not experienced an AR system. As a result, it was confirmed that there was a significant difference between subjects who experienced an AR system (M = 6.33, SD = 0.487) and those who had not (M = 4.52, SD = 0.512) before participating in this study. Hence, users who had never experienced AR felt that our system was confusing. This problem was confirmed in greater detail through the interview results.

4) INTERVIEW
The subjects who participated in this experiment expressed satisfaction with being able to experience the CP with AR. They reported satisfaction with characters not revealed in the painting being shown and animated contents that could not VOLUME 11, 2023 be seen in the painting. In addition, the participants noted the characteristic of showing hidden information that is difficult to predict when looking at the picture and the importance of expressing historical events in 3D animations. In particular, one participant explained that he felt he'd had a ''real experience of a CP.
Of the 34 participants, 18 (52.9 %) reported the model made the experience more special because it was in an exhibition hall. They responded that the model helped them understand the historical space and events. In addition, there was the opinion that the AR enhancement without markers on the model made the experience more natural.
One participant replied, ''It feels more realistic to experience AR while watching this model, which is implemented similarly to the CP. I have a better understanding of this painting. '' There were many positive opinions of the order of the AR system experience. For example, Participants who experience AR after learning prior information can relate the prior information to virtual objects.
''The character I'm looking at right now is the character that was described earlier. So it's more fun.'' The animation of the training course provided to the participants provided an overall story. They replied that they were satisfied with this. In particular, the advantage of our AR system is that we can check the training process in an order that is difficult to recognize in the painting. They reported that the night military exercises shown by the proposed system were not seen in other ARs, also reporting that their experience of lighting torches and firing gunpowder was impressive within the system experience.
''I loved being able to see the military exercises in person. The voice heard during the process also helped to understand the training process. And the part where the gunpowder was fired was the coolest.'' Participants wanted to be provided with detailed information about a specific target during the AR experience. Therefore, they reported that it was an excellent experience to be provided with information about names and roles when selecting augmented virtual objects. In addition, they noted that they were satisfied that detailed information about the costumes and items of historical figures unknown in the painting was provided. However, too much information confused them.
''I didn't know what to choose first. It was confusing.'' ''I don't know which training I should see first.'' ''I want to know what was more important to see without anyone explaining.'' In particular, in an additional interview about Q2, participants who used AR for the first time reported that they experienced confusion most often, as the AR was not easy to manipulate and because much more information existed than in other systems. In addition, there were other opinions about the inconvenience of finding and pressing the UI buttons.

A. THE VALUE OF AR EXPERIENCE IN TRADITIONAL PAINTINGS WITH HISTORICAL EVENTS
Our research goal was to provide users with historical information and story experiences as they relate to a CP through AR experiences and to induce better immersion and interest through methods differentiated from those in existing studies.
According to this user study, experiencing the CP with our system increased the users' insight into paintings. This increased understanding and insight into the CP led to greater interest in the exhibition experience. An exciting exhibition experience meant that users became more immersed in the CP. This result indicates that if the proposed AR system is used for traditional painting exhibitions, satisfaction will be higher than otherwise.
The participants in the experiment were most interested in learning information that could not be found or that was not visually identifiable in a CP. Participants' basic knowledge allowed them to recall the historical events included in the painting, but there was a limit to describing them in detail. However, the historical space and events could be known in detail using the proposed system. This experiment confirmed that making a model of the space of the CP could help users become more immersed in the scenes in the painting. The CP AR experience augmented without a marker on the model enhanced the spatial understanding of the painting. This better spatial understanding increased user satisfaction. As confirmed in the interviews, users could better grasp the story in the history based on their understanding of the historical space of the painting.
The order of our system's experience process positively affected the users' grasp and understanding of the information in the painting. Users naturally experienced AR through the step-by-step learning experience. Furthermore, the general information provided during this type of learning and the details of the figures could supplement the users' insufficient prior knowledge. As a result, users responded that our experience method effectively matched the pictures with the augmented virtual objects in AR.
The results of the user evaluations demonstrated that the difficulties associated with CPs mentioned in previous studies [2], [3], [4] can be resolved by the use of the proposed system. Furthermore, the results of the comparison with other systems [49] produced by primary research methods also confirmed that the system proposed here allows users to experience better immersion and learning.

B. LIMITATIONS OF AR USABILITY AND CONVENIENCE
Our experimental results show that satisfaction increased when users felt that manipulation was easy and that the graphical representations were natural. However, users for whom this system was their first experience with AR had difficulty understanding the UI.
According to previous studies [48], the user interface must be easy and straightforward for the user to find on the screen.
The interviews indicated that the design of the UI provided by our system was not complicated, but they also showed that various colors on the screen could confuse the user's gaze. Moreover, users found it challenging to experience the system when having to hold a large tablet PC with both hands. In addition, most of the participants responded that the difficulty with the system stemmed from the large amount of information and inaccurate guidance. The size of the proposed system exceeded that of a tablet PC. Therefore, it was difficult for users to know precisely what to focus on without a separate explanation of the screen. One user mentioned that a method that established the main object was needed because more than 1,000 virtual objects appeared against the backdrop of the broad terrain during the AR experience. In addition, others wanted the relationships between the characters to be visually expressed. We confirmed through the experiment and interview results that visual guides are needed when there is a large amount of information provided by the AR system. Thus, in such cases in the future, methods to guide the experience visually should be proposed.

VI. CONCLUSION
This study proposed AR system aims to enhance users' historical understanding and provide a more immersive experience by visually implementing both depicted and omitted information in the CP. Depicted or omitted historical information was restored as 3D virtual and physical objects, allowing users to experience it with AR. Furthermore, the system is designed based on digital storytelling and experience learning theory to provide new experiences to users.
Our system provided a better interaction and learning experience than existing methods according to the experiment. In particular, most users expressed great satisfaction with the visually provided information that was not revealed in the CP. Therefore, the proposed system was confirmed as a powerful means for users to experience CPs in more detail. Applying these results to museums and art museum exhibitions can improve the overall user experience and allow them to appreciate CPs much more.
As a future work, we aim to improve the usability of cultural heritage experience systems that provide a large amount of information. Particularly, we plan to design a visual guide method that enables users to efficiently access information within the system, providing them with a convenient experience.