Easy and Deep Media

Published Online December 2010 in MECS

There has been a growing trend toward applying computer technology in the field of cultural heritage education. This study aims to develop an educational media system that the audience could intuitively interact with while appreciating the sophisticated cultural content with ease. Ting, the type of bronze cauldron with three hoofed feet, is a symbol of regime legitimacy in traditional Chinese culture. The Mao-Kung Ting is one of the most important collections of the National Palace Museum, not only because of its symbolic significance in history, but also because of the length of the inscription engraved on its interior, which is the longest ever discovered among unearthed bronze. Writing characters play a significant role in supporting the cultural continuity. The creation of Chinese characters, although later than that of the Egypt and Mesopotamia cuneiform writing, generally believed to have started at least three thousand years ago. As a living language that is still being used by over a billion people thousand years later today, it goes without saying that the Chinese characters

This work was partially supported by National Science Council NSC97-2218-E-002-002.

is one of the most precious cultural assets for the mankind. Earlier Chinese characters are mainly the “Oracle” engraved on tortoise shells and animal bones and the inscriptions on the cauldrons. On the study of the cauldron inscriptions, however, it could never be thorough enough without mentioning the Mao-Kung Ting. In view of the historical as well as the cultural significance of the Mao-Kung Ting, our team endeavored to combine the historical content of Mao-Kung Ting with the evolution of the Chinese characters.

By employing inter-disciplinary methodologies, we have developed a set of intuitive interactive installation that emphasizes on both the historical content of the Mao-Kung Ting and the evolution of Chinese characters through an immersive audio and video environment. This educational media system aims to provide the audiences an engaging and in-depth interactive experience. The design concept is therefore audience-centered so that we want to make the installation: on the one hand, easy for the audiences to get involved without being superficial; on the other hand, rich in deep cultural content without being abstruse to the general public. The system is composed of two applications: the “Turning Rust into Gold” application and the Interactive Chinese Characters application. The “Turning Rust into Gold” application is a prior work of our research team [1]. It aims to translate the macro concept of space and time that the Mao-Kung Ting has traveled through in the past 2800 years into a virtual experience of de-/weathering. By visualizing the de-/weathering process, we also try to convey the importance of the cultural heritage preservation. The Interactive Chinese Characters application demonstrates the evolution of the Chinese characters from ancient pictographic characters to modern fonts. Exemplified by sets of individual evolution of ancient Chinese characters, the goal of the installation is to make the user feel like being in the time tunnel witnessing the evolution of Chinese characters from the oracle bones, cauldron inscriptions to seal script, official script, and regular script to cursive fonts. It aims to guide the audiences’ experience from understanding of changes in shape of the Chinese characters to appreciating the beauty of Chinese calligraphy. The ultimate mission of the system is to interpret the development of Chinese characters across time and space into a joyful visual presentation that further provokes users’ curiosity about the ancient Chinese characters. The appearance of the installation is shown in Figure 1 and the interaction between the user and the installation is demonstrated in Figure 2. This article is a revised and expanded version of Hsieh et al [2]. It is organized as the follows: In Section II, related works are described here. Then, the developments of the two applications are detailed in Section III, IV. Next, Section V presents the case study. Finally, the conclusion is summarized in Section VI.

Figure 1: The appearance of the Interactive Chinese Characters installation

Figure 2: The interaction between the user and system.

• II.  Related Works

A.    De-/weathering Simulation

The daylight, in conjunction with the effect of temperature, moisture and atmospheric oxygen, acts as the main parameters of outdoor weathering. The visual simulations of time-variant material appearance are rendering by computer graphics approach. In this work, the “Appearance Manifolds” (Figure 3) simulation technique has been chosen. A weathered material sample contains spatial variations that depicts different degrees of weathering can further be analyzed to extract spatial and temporal appearance properties for synthesizing the weathering process. The basic idea is that the spatial variations in the appearance of a weathered material sample at a given time will correspond to different degrees of weathering [3].

Figure 3: Weathered appearance manifold: “Modeling the time-variant surface appearance of a material from data captured at a single instant in time [3].”

B.    Breath-based Biofeedback

Several techniques for sensing breath status were surveyed. Optoelectronic Plethysmography (OEP), which is a method to evaluate ventilation of the chest wall surface motion through an optical movement detecting system. It requires several small reflective markers to be placed on the thoracic-abdominal surface by adhesive tape. Heart Rate Variability (HRV) is a physiological phenomenon where the time interval between heart beats varies. It is measured by the variation in the beat-to-beat interval. Respiratory Inductive Plethysmography (RIP) is a tool for ventilatory monitoring, measuring changes in rib cage and abdominal cross-sectional areas, it is a traditional breath detection method to detect respiration status of users [4, 5]. Ultra Wide Band (UWB) radar with IR laser diode can remotely measure the heart activities and respiration of the users [6]. As it is non-invasive, it does not require attaching a device to the body, which is comparatively more feasible in the museum setting. As a result, we have chosen the UWB to detect the breathing status of the users. Table 1 shows the comparison of these respiration detection methods [7].

Table 1. Comparison of Respiration Detection Methods

	OEP	HRV	RIP	UWB
Attached to body	Attached markers	Attached to skin	Attached to body	Non-invasive
Accuracy	High	Median	Median	Median
Limits	Least mobility	Detectors attached to fingers	Detectors attached to the body	Less mobility

C.    Research on Chinese Characters

Back in about 100 A.D., the first dictionary in China titled Shuowen Jiezi is composed by Xu Shen, an Eastern Han Dynasty Scholar. It summarized the six categories of the creation of ancient Chinese characters including Pictographic Characters, Indicative Characters, Associative Compounds Characters, Pictophonetic Characters, Mutually Explanatory characters, and Phonetic Loan Characters. We will explain the six categories with examples because of they are highly related to our research.

• •    Pictographic characters: Stylized characters that represent objects in the real world, such as the sun, the moon, the tree, and the horses. Their meanings could usually be apprehended immediately. A salience feature of Pictographic characters is that they usually cannot be divided

into two or more parts. As a result, they are also called “independent characters”.

• •    Indicative characters: Characters that use symbolic strokes to indicate some abstract things. For example, “ 木 ( mu )” is the image of a tree. When a horizontal stroke is added on its vertical stroke, “ 木 ( mu )” becomes a new character, 本 ( ben ), meaning “root of a tree”. Ancient Chinese people employed this principle to further indicate the abstract concept or idea. Like the pictographic characters, Indicative Characters are also independent characters. The added stroke is only functioned as a sign to indicate another idea.

• •    Associative compounds characters: The characters that are composed of two or more parts to express a new meaning, regardless of the original relation between the individual parts. For example, “ 看 ( kan )” is composed of “ 手 ( sho , hand)” on the upper part and “ 目 ( mu , eye)” on the lower part to illustrate the image of a hand over the forehead, representing the act of seeing something. Chinese characters like this are called Associative Compounds Characters, meaning that when people read these characters, they could understand the concept by assembling two or more elements.

• •    Pictophonetic characters : Characters that are composed of at least one pictorial part and one phonetic element. While the former indicates the property or the category of a word, the later is the symbol of its pronunciation. Ancient people used a phonetic element with different pictorial parts to resemble thousands of new characters so that most Chinese characters were created with this principle.

• •    Mutually explanatory characters : Characters that could mutually explain each other’s concept by the meaning, the pronunciation, or by the pictograph. As a result, different characters may have the same meanings and are mutually explanatory.

• •    Phonetic loan characters: Characters that are loaned from another character with the same pronunciation to represent a different meaning. Since some oral terms had no concrete sign to be written down, ancient people applied another characters with the same pronunciation to indicate the intended meaning. Hence, a character may have multiple meanings because of this applied principle.

Since Song Dynasty, the literati have been fascinated with the beauty of ancient bronze such as the cauldrons and the bells. The appreciation of the bronze has further drawn the literati’s attention to the ancient Chinese characters engraved on the bronze caldrons and bells— the cauldron inscriptions. Hundreds years later, scholars and literati in Ching Dynasty began concentrating on the oracle bones and the bronze inscriptions. After a great amount of research has been done by the scholars, various dictionaries have been composed as valuable supplements for research across disciplines.

With the advancement of modern technology and information system, researchers have put a lot of effort to integrate multiple resources of the Chinese characters. To date, several digital archives for ancient Chinese characters including the oracle bones and the cauldron inscriptions have been built, digitalizing thousands of inscriptions of ancient Chinese characters to be accessible to researchers [8].

• III.    The Turning Rust into Gold Application

The “Turning Rust into Gold” application is our prior work. It aims to integrate modern technologies and aesthetic design to reveal the unique features of the Mao-Kung Ting. The users will be guided across the spatiotemporal civilization in China to experience the Chinese philosophy. The application integrates the digital simulation of the de-weathering processes with the human interactive interface that uses a breathing-based biofeedback to signal the simulation process. The metaphor of “Qi”, which refers to the energy flow of living beings in the world, is thus converted into the nature breathing process.

A.    System Architecture

The interactive installation consists of a detection module and a display module. The main component of the detection module is the Mao-Kung Ting-shaped chair embedded with a UWB breath sensor and pressure sensors inside. When a user sits on the chair, the pressure sensors will sense the weight of the body. Meanwhile, the UWB breath sensor will start to detect the breathing frequency of the user. The display module, on the other hand, is composed of two display components: the primary display and the secondary display as shown in (Figure 4). The 52-inch touch screen TV as the primary display in the front is arranged to articulate the process and to provide related information. The secondary display comprises a large projection area with multilayer veils and a Mao-Kung Ting-shaped chair.

Figure 4: The installation has a primary display and a secondary display.

B.    Interactive Scenario

The designed scenario could be separated into two stages. In the first stage, the users will be asked to adjust their breathing frequency. As soon as they have accumulated energy from their breathing practice, they will be guided into the second stage, in which they could have the power to turn the rusty Mao-Kung Ting into its original golden appearance through the haptic interaction, which we called a de-weathering process. The immersing environment we designed with the application of “Turning Rust into Gold ” is shown in Figure 5 [1].

Figure 6: The main components of the installation: the Primary Display Module, and the Peripheral Display Module.

• IV.    The Interactive Chinese Characters Application

Another important feature of Mao-Kong Ting is the inscriptions engraved on its interior. These ancient characters could hardly be understood by modern audiences without sufficient guidance. The Interactive Chinese Characters application is created in order to arouse viewers’ interest in appreciating the ancient inscriptions. With the support of innovative multi-media presentation as well as the intuitive multi-touch interface, these ancient characters are animated to interact with the users. They demonstrate their individual evolution through visual presentation that the users can interact with. Our goal is to inspire the modern users by offering them a new experience with the Chinese characters.

Figure 5: The immersing environment designed with artistic atmosphere and Chinese cultural elements.

A.    System Architecture

The main structure of the installation is divided into the display module and the detection module. A 42-inch touch screen is designated as the primary display device and the detection device. Three projectors installed on the ceiling are the peripheral display device, which generates the surrounding image. The structure is shown in Figure 6.

B.    Interactive scenario

• 1)    Magic Lens: The magic Lens is a designated selection area that functions like a magnifying glass. The participants could use their finger tips to manipulate the Magic Lens on the screen intuitively. Instead of magnifying the image underneath it, the Magic Lens will simultaneously convert the ancient inscriptions into modern characters when the users direct the magic lens to the position desired, as shown in Figure 7.

  

  Figure 7: As the users direct the magic lens to the position desired, the Magic Lens will simultaneously convert the ancient inscriptions into modern characters.

  
  

• 2)    Seamless Connection: Seamless Connection suggests two aspects of the interpretation on each Chinese character, the formal aspect and the affective aspect, that are made possible through the cooperation of the Primary display and the Peripheral display. The affective aspect refers to the continuity of the plot that seamlessly connects with the audiences’ daily experience. Take the character “自(Zi)” for example, to demonstrate its creation and evolution, we created an animation of Sun Wukong (i.e. the Monkey King in the Journey to the West) who is pointing to himself to show the inspiration of the creation of the character “自(Zi)”. When the Gold-Banded Cudgel in the Primary Display is thrown by Sun Wukong, it will be picked up right away by another Sun Wukong in the Peripheral Display, as shown in Figure 8. It is designed not only to catch the users’ attention but also to suggest that the development of Chinese character is intimately linked to our daily lives.

Figure 8: The Gold-Banded Cudgel in the Primary Display thrown by Sun Wukong, as shown in the upper image, will be picked up right away by the Sun Wukong in the Peripheral Display, as shown in lower image.

The formal aspect of the seamless connection, on the other hand, refers to the seamless connection within the evolution of each character. To further demonstrate the formal evolution of the character “ § (Zi)”, while the performance followed goes around the Peripheral Display, the Primary Display will simultaneously present the historical changes in the form of “ § (Zi)” from ancient pictographic characters to bronze inscriptions, seal script, official script, regular script, cursive fonts, and other stages. This consecutive animation therefore manifests the seamless connection between each form of “ § (Zi)” historically, as shown in Figure 9.

Figure 9: The evolution of the Chinese character “ § (Zi)”, which means “self” in Chinese, from an to an Phonetic Loan Characters. It was originally a Pictographic Characters as a representation of the nose. Then, since people usually point to their nose to express themselves, this character is later borrowed to signify a person self-identification. As time goes by, the form of “ § (Zi)” gradually evolved from bronze inscription to the modern regular script that indicates the concept of “self” as shown in the figure.

The following is another example to show the design concept of the animation. To demonstrate the Chinese character “ Ж (Ma) ”, which came from a pictographic character that captures the shape of a jumping horse, as shown in Figure 10.

Figure 10: The Chinese character “ Ж ( ma, horse)” originally came from the description of the image of a horse. It was a pictorial character in the era of the oracle bones. Although it gradually gets rid of the pictorial image and becomes a written character, it still preserves the key features of a horse, such as the head, neck, hair in its upper part; and its legs in the lower part that are represented by four dots.

• V.    The Case Study of Education and Promotions

Education is one of the most important missions of the museum. [9] To reach out diverse groups of audiences and to share cultural assets to the public, the National Palace Museum has been conducting educational activities outside the museum. The Mao-Kung Ting Interactive Media has provided an educational media design model as the following. At the very beginning, favorable insights are brought up through the analysis of the artifact. Next, while the insight-driven approach is applied in the content development, several media technologies are evaluated to support the system design concept. After the matching technologies are found, they are integrated into the interactive scenario to create a blending learning experience. The process is illustrated in Figure 11. To get a closer examination on how the media system enhances visitors’ experience with the cultural object, two participant observation case studies were conducted. The “Turning Rust into Gold” application has been exhibited at the LOHAS exhibit in the Future Museum at Taoyuan International Airport, as shown in Figure 12. The Interactive Chinese Characters application has been showcased at the traveling exhibit, “National Treasures of NPM in Chiayi” for two months. They further brought the new challenges on developing educational media in diverse museum setting to our attention.

Figure 11: The education model of the case

• A.    The Future Museum: Turning Rust into Gold

The design of the Future Museum emphasizes on transforming the essence of Chinese culture with the latest digital technologies. [10] The “Turning Rust into Gold” application of the Mao-Kung Ting Interactive Media was designed to support the mission of the Future Museum. The Future Museum is special in terms of its location. As in the departure area within the international airport, the target audiences are overseas visitors who are either leaving the country or stopping by for transferring.

Figure 12: The entrance of the Future Museum

• The key challenges

The Future Museum embraces people from extremely diverse background. It is a great opportunity for the Museum to introduce itself to people with distinct race, nationality, and culture background. In the meantime, it is also a great challenge since the Museum could only seize the tiny times slot to impress people who certainly do not come for the purpose of visiting the Museum. Except the fact that they are all waiting for boarding, these people have nearly nothing in common. The collections in NPM, however, possess a profound cultural context that is not easy to be absorbed within a short time. Take the Mao-Kung Ting for example. It may not be hard to explain who Mao-Kung is and why this Ting is named after him. But it would be difficult to explain to a foreigner why is this historical background important. Not to mention to explain why would it matter, especially for people who do not inherently feel connected to the cultural object? On appreciating the cultural objects, it could be easier to understand a cauldron in the form of a dragon, but it would be much more difficult to help people understand the beauty of the threefooted Mao-Kung Ting as a serious ritual vessel.

• • How would the multimedia help on enhancing the visitors’ experience?

It would be more intuitive to understand the historical significance of Mao-Kung Ting if one can personally feel connected to the historical context. As a result, to create a virtual time travel becomes the primary mission on developing the Mao-Kung Ting educational media. The use of UWB sensor to detect the breathing and the employment of the de-/weathering visual effects, as well as the use of multi-touch screen as an intuitive interface are all applications to facilitate this virtual timetravel journey. By employing these key technologies, visitors can intuitively experience a virtual time-travel with the ancient Mao-Kung Ting through the simple breathing activity. We have observed that most of the visitors who come to the exhibit would participate in the entire interactive scenario and further be curious about the ancillary information of the object itself. Even for those who do not have enough time to complete the entire interactive scenario are still excited about the system, as well as the ancient artifact that has become friendly to them.

• B.    National Treasures of NPM in Chiayi: Interactive Chinese Characters

The NPM has been conducting a series of traveling exhibition to further the Museum's stated mission on advancing the understanding of Chinese cultural heritage. The Chiayi Museum in central Taiwan is one of the National Palace Museum traveling exhibition locations, as shown in Figure 13. As in a relatively remote city, people who come to the exhibit attribute to local families and schools. To arouse people’s interest in learning the historical evolution of the Chinese characters in associate with the Mao-Kung Ting, the Interactive Chinese Character of the Mao-Kung Ting Interactive Media was presented as the main exhibit. To bring more local communities to the exhibit, the curatorial team also organized a series of educational activities, including volunteer training, media campaigns, and family activities. Local volunteers are mostly seniors. It has also been noticed that the local community has less sensitivity to the digital interactive media. As a result, besides introducing the cultural content, the educational training also covers the instruction on using the media application. After a thorough training, all the volunteers were not only having greater interest in the exhibition themselves, but also able to give proper guide tour for the visitors. The volunteer training is shown in Figure 14. In addition, the exhibit also arranged a family activity area to present sets of interesting project that all family members could participate jointly, such as the puzzle and quiz games. With the help of the well-trained volunteers, all visitors from little children to senior adults could have a great time interacting with the digital facilities intuitively (as shown in Figure 15). The media campaign, on the other hand, aimed at attracting local people to visit the exhibit. Through the media platforms including the television, newspapers, as well as a variety of new media online, the interview of the exhibit was widely spread to reach out potential audiences [11,12] (as shown in Figure 16). Among all, the report of the Exhibit’s Opening has successfully caught up local community’s attention.

More than 30,000 people visit the exhibit within 2 months.

Figure 13: The entrance of the Chiayi Museum

Figure 14: Structured volunteer training helps to provide high-quality educational activities.

Figure 15: The interactive family activities have successfully enhance the learning experience.

Figure 16: The interview of the exhibit was widely spread to reach out potential audiences

• •    The key challenges

Different from the Future Museum at the Airport, this traveling exhibition was designed to serve the general public, especially the family groups. The challenge, therefore, became how to design the activities that not only the children, but also the senior family members could feel highly involved. As a middle city in Taiwan, the urban-rural discrepancy still remains a prominent issue in Chiayi. In addition to the fact that only a small portion of local people visit museums and galleries on a regular basis, some of the audiences are not familiar with computers and the digital installation. As a result, the curatorial team has to put more effort on bringing people into the exhibit and making the museum experience relevant to them.

• •    How would the multimedia help on enhancing the visitors’ experience?

The Interactive Chinese Characters application of the Mao-Kung Ting Multimedia was designed for the traveling exhibitions. The above challenges were taking into account at the very beginning. In addition to the traditional way of showcasing such as the detailed explanation on the text panel, the multi-touch interface was employed to improve the convenience for younger children and senior visitors. The visual presentation through animated computer graphic helps visitors from different ages to appreciate the evolution of Chinese characters with ease and further prompt reciprocal learning among family members. It is also worth mention that the Magic Lens has caught a great amount of attention as it can intuitively translate the ancient cauldron inscription into modern Chinese characters under one’s fingertips. Overall, the original design concept has successfully produced the expected results.

• VI. Conclusions

The Mao-Kung Ting Educational Media has utilized two new media applications: the “Turning Rust into Gold” application and the Interactive Chinese Characters application to interpret the precious state vessel Mao-Kung Ting into an engaging and in-depth interactive installation. While the “Turning Rust into Gold” application has translated the space and time that the Mao-Kung Ting has traveled through into a virtual experience of de-/weathering, the Interactive Chinese Characters application has visualized the evolution of the Chinese characters from ancient characters to modern fonts. In addition, the immersing and contextual environment of the Chinese aesthetic atmosphere has creatively provided a sensational experience of crossing time and space through Chinese civilization. The installation has been exhibited in the traveling exhibition, National Treasures of NPM in Chiayi, at the Museum of Chiayi City for 2 months. It has been observed that it has effectively engaged people of different ages. It has provided visitors new approaches and perspectives to appreciate ancient artifacts and has further provoked their curiosity about the evolution of Chinese characters. In sum, this installation has successfully translated deep cultural knowledge into an easy-to-understand interactive educational media. It has also contributed to the feasibility of integrating new media art in Cultural Heritage Field. Finally, through the participant observation, the case studies have revealed the opportunities, as well as the challenges on designing the educational media in modern museum setting. The new technologies have helped to enhance visitors’ experience and also suggest new opportunities in modern museum education.

ACKNOWLEDGMENT

We would like to thank EECOM Multimedia Co., Techart Co Ltd, Microsoft Research Asia, Academia Sinica, Industrial Technology Research Institute (ITRI) and curators of National Palace Museum.