How digital humans are reinvigorating Chinese cultural heritage

Leading museums worldwide are experimenting with augmented reality and virtual reality technologies to create new experiences for audiences to engage with art and culture. Earlier this month, the Metropolitan Museum of Art in New York launched the Replica app powered by the gaming platform Roblox.

It allows users to scan a selected number of artworks in the museum, and collect and wear them in the digital world. For instance, one can choose a character and dress it in Vincent van Gogh’s hat and shirt adapted from his Self-Portrait. Users can clearly distinguish themselves in the real world from the characters in the game.

To reinvent an immersive experience, museums in China have started using digital humans to transform visitors’ encounters with history and culture. Digital humans are computer-generated 3D images that look like real people.

The industry has seen rapid growth in China in the past few years, revolutionising e-commerce, healthcare, tourism and entertainment. According to a strategic plan by Beijing’s Bureau for Economy and Information Technology, the city alone is expected to see the market for digital humans grow to 50 billion yuan (US$6.9 billion) by 2025.

In collaboration with top artificial intelligence (AI) companies such as Baidu and Tencent, Chinese museums have introduced digital humans created by mixed reality technology to become tour guides and ambassadors of Chinese tradition. These virtual humans powered by chatbots can tell stories of massive artworks produced throughout the country’s history. They are knowledgeable and tireless companions for visitors.

A visitor looks at an exhibit of Dazu rock carvings at the National Museum of China in Beijing on November 13, 2021. Photo: Xinhua

This new development exemplifies how AI can help China scale the preservation, education, circulation and promotion of cultural heritage and can inspire the world to do the same.

Ai Wenwen, a digital employee at the National Museum of China, can instantly change outfits, from modern clothing to Chinese traditional dress, and give guided tours of the museum. Ai Wenwen’s name references “AI” and “culture”, suggesting the tremendous potential of AI applications in the cultural industry.

The creation of virtual avatars uses image generation, 3D modelling, movement and facial expression capture, as well as rendering techniques that call to mind the world’s foremost mixed reality technology by industry leaders such as Microsoft and Nvidia.

To give its digital employee the appeal of a human celebrity, the National Museum collaborated with entertainment company Migu Music to create a song that is presented as if sung by Ai Wenwen herself. The melody combines traditional Chinese musical instruments with electronic music. The lyrics praise the beauty of Chinese art such as painting, porcelain and embroidery. The title of the song, “Wei Ai”, means the future of AI, or the infinite development of AI.

Will you be replaced by your AI clone? Maybe one day, but not quite yet

In addition to museums, digital humans can help bring historical figures to life. In 2022, the Zhonghua Book Company launched a virtual avatar of Su Shi, an acclaimed Chinese poet and statesman. The creation of the avatar was enabled by the machine learning of 7 billion Chinese characters of historical texts and more than 100 paintings and images depicting Su Shi in the past 900 years or so.

With the texts about Su Shi’s life and literary work being fed into the virtual avatar, it can respond to real people in a manner and language resembling this great man in Chinese history. It made its television debut in a CCTV youth talent show about Chinese poetry, surprising audiences with its remarkable human likeness.

The virtual avatar had conversations with live audiences about poems Su Shi wrote about 1,000 years ago. The TV show even presented an episode of the digital version of Su Shi acting with a human who played his close friend, the great calligrapher Huang Tingjian.

The virtual avatar of Su Shi has even appeared as an influencer on Weibo. Its account has posted quizzes and online education material, teaching readers about Chinese literature, geography and traditional customs. Apart from online programmes, Zhonghua Book Company has also designed in-person events for students to learn Chinese literature and history through engaging with digital humans.

A sculpture of Su Shi, seen in Meishan, Sichuan province. The acclaimed Chinese poet and statesman has now been brought to life in the form of a virtual avatar. Photo: Shutterstock

Museums and cultural institutions across China have incorporated the virtual avatar of Su Shi to create interactive exhibitions with themes including afternoon tea, poetry and calligraphy. Audiences at these exhibitions have opportunities to experience the daily life of a renowned poet from the distant past.

Digital avatars of historical figures can help popularise traditional Chinese culture and aesthetics among schoolchildren, university students and the public. If a similar technology was used in the United States and European countries to create interactive, digital versions of the likes of Plato, Aristotle and Shakespeare, it could be helpful in attracting more young people to study the classics.

More importantly, these initiatives can boost cooperation between science and technology educators and those in the humanities. Culture-embedded digital humans can create new opportunities for humanities students to contribute to the digital economy.

The development of digital humans demands collaboration among public institutes, private companies, universities and non-profit organisations. Museums creating new applications of AI can improve cohesion across social sectors, ultimately increasing education equity and access in an era of rapid change.

According to the 2022 Innovation Index Report, published by the World Intellectual Property Organization, China and the US are host to the same number of science and technology clusters at 21 each. As the two global powers compete to be the leader in AI and advanced technology, it is important to remember that countries can learn from each other to harness emerging technologies to preserve the rich cultures of global civilisations.

Gerui Wang is a visiting scholar at Stanford University

South China Morning Post