Digital mask

The digital transformation of traditional art

Digital Mask utilizes AR filters to transform traditional theatrical mask art into digital art. Transform these on-stage masks into digital masks that everyone can wear. I'm thrilled to use multiple techniques to convey traditional mask art more broadly.

Time

2022

Project Type

AR filter
generated art

Skills&Tools

expression recognition
Lens Studio, Adobe PS, AI

Traditional opera art is fading from the public's view.

Beijing Opera is a synthesis of stylized action, singing, dialogue, mime, acrobatic fighting and dancing to represent a story or depict different characters and their feelings of gladness, anger, sorrow, happiness, surprise, fear and sadness. The characters may be loyal or treacherous, beautiful or ugly, good or bad. Their images are always vividly manifested in bright costumes that show the styles of ancient China.

The masks of Beijing opera incorporates the personality, role and background of the characters. Different mask styles and colors represent different historical figures.

How can design bring traditional art into digital era?

Customized digital AR masks

In the Peking Opera makeup, artists use exaggerated and abstract lines to express the characters’ face, use different colors to express the personality of the characters.

Digital masks digitize styles and colors of Peking Opera mask and use expression recognition to customize each mask.

Scan in snapchat

Design process

Research

Understand the essential of opera mask

Ideate

Generate ideas
weigh main elements

Technical demonstration

Ensure technical path
choose the right tools

Design

Iterate

Prototype design

Test the prototype
improve the performance

The relationship between color and personality in Beijing opera masks.

In Peking Opera masks, people use exaggerated and abstract lines to express the characters’face, use different colors to express the personality of the characters. Different color schemes help viewers understand thepersonality of the roles. For example, red often represents loyalty, blue stands for straight and unruly.

Styles + Color shifting

Link emotions with mask colors.

expression recognition supported by deep learning network

Color can affect mood, and emotion can be expressed in different colors.
In order to preserve the relationship of color and personality in the digital mask. I use emotions to replace personality. Using facial expression recognition technology, emotions can be quickly identified. This can serve as the basis for the real time changing of digitized masks.

A simple neural network can distinguish A from B.

A more complex neural network can further distinguish same color shape into circle and square.

Facial expression recognition.

The principle of deep learning networks is similar to neurons in the human brain. It can greatly improve the intelligence of the computer and make it possible to deal with problems that was previously only humans could do, for example, art creation.

The neural network I use, called convolutional neural networks, can classify and identify different emotions.

Successfully recognized the expression in the picture

Design digital opera masks as a snapchat fliter

I want people to experience traditional mask art in a fun and easy way. Snapchat has powerful filters in it. I use Lens Studio to create a filter and publish it in snapcht.
Through this snap fliter, people can get their own Peking Opera masks and interact with AR masks. They are also able to post these in social networks.

Technical validation of customized AR masks

Two parts of technical issues need to be tested and validated.

-The first was a custom mask. I decided to use self-generated art instead of designing a lot of mock-ups and letting users choose their own. Each user's mask should be unique. Just like the masks for characters in Peking Opera are exclusive.
-The next step is to make filters in Lens Studio. I want it to be able to accomplish the effect of changing faces. By turning back once or using gesture to change the AR mask, just like the face changing magic in the traditional theater stage.

Develop customized masks

Step1 Build database

Collect Peking Opera masks and extract the elements to create a database for deep learning network learning and training.

Step2 Recognition + classification

Using the recognition program to detect and classify the face, and then the tester's facial expression is recognized. Then pick the appropriate parts in the image library and recombine them.

Step3 Refactoring

According to the appearance of a person, the program selects the appropriate materials in the content library to combine, and then selects different color combinations according to the facial expressions.