"Similar discussions have been aroused in numerous science fictions about aliens or time travel, but they are no longer fictional anymore when we humans face AI."
​
Haoheng Tang
A Mystery for You
Collaborative Academic Project
Apr 2023 to July 2023
Teammate: Mrinalini Singha
Instructor: Marcelo Coelho
A Mystery for You
Keyi Zhang: Congratulations on winning the Student Game Competition in CHI 2024 with the project “A Mystery for You”! Would you like to share something about why and how you did this project?
Haoheng Tang: Thank you! The project began as a course assignment, where we were asked to design an interesting application for Large Language Models and make an interactive tangible product. I experimented with various game-like interactions with ChatGPT and found that it could play situational puzzles with me, often providing unexpected yet logical answers. It could also generate situational puzzles, answer my yes/no questions, and guide me step by step to find the solution. This demonstrated ChatGPT’s logical thinking and storytelling ability. This inspired me to see if ChatGPT could guide players in detective games like CLUE. The project, "A Mystery for You," was named after the adventure fiction series "A Mystery for You and the Tiger-Team."
My teammates and I decided to take it further and create an educational game to teach children how to identify misinformation on social media and cultivate media literacy. Our game mechanism soon took shape: ChatGPT presents a news report and asks the players to investigate it with tools in their hands. ChatGPT responds to players' actions, providing more information, and unfolding the story until the players determine the report's authenticity or exhaust their investigations. ChatGPT then reveals the truth. The news reports are closely related to political propaganda and social conflicts in today's society. Our goal is to help kids and teenagers evaluate online news critically, and use the tools in their hands to discern truth from falsehood instead of being easily misled.
Cartridge insertion and receipts printing are the main forms of interaction in this project. Users insert a pair of cards for each survey step: one representing the method of investigation and the other the character to investigate. The machine processes the player's input, sends it to ChatGPT, and then prints ChatGPT's response on a receipt for the player to read. Such tangible interactions outweigh digital screen interaction in many ways. Cards, like Pokémon and Yu-Gi-Oh, carry memories and emotions, and kids may want to collect them. The printed receipts can also be collected as souvenirs or posted on the wall as storybooks. Kids can draw and compare plot developments from different actions each time. Additionally, most parents prefer their children to avoid screens, making this interaction method more appealing.
This project should have been completed a year ago, but I have made many improvements to it over the past year. For example, I designed and made printed circuit boards (PCBs) myself to improve the circuit stability. I designed an old computer-style shell for this product inspired by telegraph machines and attendance punch card machines. The end product looks cool and is commented as having a sci-fi yet nostalgic style, resembling a high-tech gadget from the ancient east.
Narcissus and the Machine
Collaborative Academic Project
Oct 2023 to Dec 2023
Teammate: Carly Lave, Sky Araki-Russel
Instructor: Panagiotis Michalatos
Narcissus and the Machine
Keyi Zhang: The video of Narcissus and the Machine looks really interesting. The AI captures people’s movements and gives real-time feedback. Would you like to tell us more about this project?
Haoheng Tang: This project stems from a provocative inquiry: if AI has its own consciousness, how would it perceive humans? Even if a very simple AI model has its own understanding and response to human behaviors. Can we envision AI as a "mirror"? Can we gain a deeper insight into our own selves by observing our "reflections'' through the AI's perspective? In response, we have breathed life into an AI with a VGG-16 architecture. Though still in its early stages, this AI can observe human movements and translate them into various water motion expressions, encompassing the majesty of ocean waves, the descent of waterfalls, the gentle pattern of rainfall, and the serene spread of ripples. It also presents the classification results in the form of a three-dimensional water surface animation. The model reached an accuracy of 100% on the test set, but we deliberately misused this water motion classification model on human behavior classification. By passing real-time images captured by a webcam to the model, we get unpredictable classification results and a constantly changing water surface animation projection.
It turns the computer vision problem into an intriguing topic - what do human behaviors look like in the eyes of a simple AI? How can humans learn the language to communicate with a machine or intelligent individual that can only understand simple things like water motions? Similar discussions have been aroused in numerous science fictions about aliens or time travel, but they are no longer fictional anymore when we humans face AI. We saw audiences constantly changing their movements with the expectations to see what they wanted from the machine's response. This reminds us of the image in Caravaggio's painting Narcissus, in which a man stares at his beautiful reflection on the surface of water. As a result, my teammates and I named the project "Narcissus and the Machine".
Narcissus and the Machine
Keyi Zhang: Is there any interesting finding when audiences interact with the installation?
Haoheng Tang: Yes, there is! We were happy to see some predictable classification results. For example, when the audience waved their arms in front of the webcam, the machine projected an ocean wave animation accordingly, indicating that what the audience did was consistent with what the machine understood. However, the unpredictable ones were even more impressive. When the audience pretended to punch the camera, the machine would classify the movement as ripples. That’s because when the fist approached the lens, it was enlarged and its expanding contour was captured, making it looks like a circle of ripples spreading out.
Tree's Breath
Personal Project
Aug 2021 to Oct 2021
Instructor: Xun Liu, Zihao Zhang
Keyi Zhang: The art installation Tree's Breath was inspired by a tree-cutting issue in Guangzhou, where you lived at the time, right? How did you come up with the idea of ​​recording people's activities under banyan trees and posting them on Weibo?
Haoheng Tang: In May of 2021, Guangzhou government proposed an urban renovation plan to replace some large banyan trees with other flowering plants. This aroused the dissatisfaction of citizens and heated discussions on social media. Netizens spontaneously started the #HugGuangzhouBanyanTree campaign on Weibo to show that banyan trees are an indispensable part of their city, as they carry the memories of Guangzhou citizens. So, I wanted to make a device to record the companionship and interaction between citizens and banyan trees to reveal the importance of banyan trees in Guangzhou citizens’ lives.
Tree's Breath
Tree's Breath
Tree's Breath
Haoheng Tang: This breathing light device installed on the aerial roots of banyan trees can sense stimuli such as shaking, sound, people's stay and walking. With the help of an AI model, it classifies and records the number of different activities under the tree, including singing, dancing, chatting, sitting in meditation, etc. Each tree has its own Weibo account. When the number of records reaches the threshold, the device will access the network through the Wi-Fi module to post on Weibo. The Weibo data from banyan trees in various places will be integrated into a website and presented as a responsive heat map. The breathing light also gives corresponding feedback to different activities by changing breathing frequency, brightness, and color. Though it won’t be noticed during daytime, you can see the color difference between trees or different parts of a tree at night as a visualization of daytime activity data. Both online and offline visualizations provide a strong basis for proving the important role of banyan trees in Guangzhou citizens’ lives. Citizens can download and print 3D models, buy electronic components, import the code I shared on GitHub, and make their own devices to hang on more banyan trees, which is a way of public participation in response to #HugGuangzhouBanyanTrees. Fortunately, the government finally heard the voices of Guangzhou citizens. Six months later, Guangzhou City issued a new tree protection policy, stipulating that, in principle, no more trees will be cut down, and banyan trees are listed as backbone tree species.
Odyssey: An AI Artist
Individual Academic Project
Feb 2024 to May 2024
Instructor: Allen Sayegh, Katarina Richter-Lunn
Odyssey: An AI Artist
Keyi Zhang: Congratulations again on graduating from Harvard in the past month. Your capstone project, Odyssey: An AI Artist, impressed me a lot. Would you like to share the way you explored quantitative aesthetics, games, and AI art creation in this project?
Haoheng Tang: Sure. By chance, I found a screenshot of the game Snake similar to the MIT Press Logo designed by Muriel Cooper, and is also similar to TSP Art. This is not a coincidence. The game mechanism of Snake causes it to take many parallel routes to fill the entire map and prevent self-intersection, which is consistent with the ideas of Muriel Cooper and TSP Art. An idea came to my mind that if the reward mechanism of the game is not to increase playability but to guide players to explore gameplay policies more in line with specific aesthetic preferences or value orientations, then we can regard the game as a new form of art creation.
In this context, the game's appeal is not in the fun it provides but in defining a framework that limits the range of game states and actions while allowing players to explore gameplay to achieve unexpected results. There are precedents for this type of art creation, in which the artist defines the framework and allows creators to create their own works. Sol LeWitt (1928-2007), an American Jewish artist, once compared artists to architects, who draw blueprints of buildings but don’t need to construct by themselves. The conceptual art he advocated also separated artists from the production of artworks. Artists conceive a work and then entrust the actual production to others. For example, many artworks in galleries were painted by Sol LeWitt’s apprentices under his instructions.
Odyssey: An AI Artist
Haoheng Tang: I onced compared my computational drawing with the one by Sol LeWitt's apprentices. Despite both following Sol LeWitt's instructions for Wall Drawing 273 and involving randomness, they looked different: my lines were chaotic while the apprentice's were more evenly distributed. The difference stems from the apprentice's unconscious incorporation of aesthetic preferences during the drawing process, resulting in randomness that is not truly random. Such aesthetic preferences do not exist in computers, so computer-generated paintings lack any tendency.
I wanted the computer to recognize and learn this subtle difference in aesthetic preference. By analyzing the intersections of the lines in both paintings, it became evident that the apprentice's line intersections were evenly distributed, while mine were clustered. The computer can "sense" this difference by calculating the global Moran’s I, quantifying the aesthetic preference. Through reinforcement learning, computers learn to create artwork with “higher aesthetic value” by making countless attempts. It still keeps the potential of creating diverse artwork rather than being constrained by algorithms. Using the same methods, a computer can achieve higher values in other senses. For example, different spiders weave different webs to maximize their hunting abilities based on their environments and prey, focusing on survival rather than beauty. Flowers, plants and trees grow randomly while following certain rules to maximize their chances of survival. Those things are difficult to simulate with traditional algorithms. The AI artist, as I called, can learn the natural rules through reinforcement learning under game-like reward systems
INTERVIEWER: KE ZHANG
CURATOR: KE ZHANG, WANTONG YAO
EDITOR: KE ZHANG, HAOHENG TANG, CHENYU LIN
GRAPHIC DESIGNER: YUXUAN WEI
Haoheng Tang, graduated from Harvard University, as Master of Design Studies (MDes Mediums). He also holds bachelor of science in urban and rural planning from South China University of Technology's and Master of Urban Spatial Analysis (MUSA) from the University of Pennsylvania. He is currently conducting research on artificial intelligence in the computational design group of HDR Architecture. Haoheng Tang's recent work was shortlisted for CHI 2024 and became the winner of the student game design competition. He has worked in architectural design and urban planning companies on generative AI training, urban data analysis, and urban design. While studying at Penn, he participated in a collaborative project between the Philadelphia Fire Department and Penn, and built an API and website for "Real-time Information Query and Fire Risk Prediction of Buildings in the City''. In addition to studying, he likes painting, singing, photography, and playing badminton.