Independent MA Major Project · 2026 · Physical–digital interaction prototype
Making visitor responses visible alongside artwork interpretation.
Living Labels connects a physical emotion display, a shared gesture interface and a mobile companion. Visitors can express an immediate reaction, explore other perspectives alongside interpretive context, and choose to save a voice privately or contribute their own response.
See how the three touchpoints work together.
Duration · 2:36
Figma · MediaPipe · ESP32-S3 / RLCD · Supabase · Web Development
Conventional labels provide curatorial context, while visitors’ feelings and personal interpretations are less easily seen or exchanged in the gallery.
Designed for visitors interested in an artwork who may not have specialist art-history knowledge.
Early concept exploration
Early ArtLens exploration and concept feedback from three non-specialist participants pointed to clearer entry points and interaction purposes, and the value of keeping one’s own response rather than only bookmarking an artwork.
Design response → a clear first action and a private visit record.
Personal contextual observation
During a personal visit to Unique Dream of Red Mansion in Langfang, movement, light and sound prompted attention and curiosity before extended explanation. This informed a simple physical invitation followed by optional interpretive depth.
Design response → a quick emotion response, followed by optional exploration.
The three early concept-feedback participants are separate from the later prototype trial.
Physical emotion response
Shared perspectives
Personal continuationSaving and contributing are optional, separate choices. Visitors do not need to complete every step.
Emotion Wall
AMUSED / UNEASY
A press creates an expression, sound and counter feedback. Responses fall, collide and settle into a collective trace. The connected emotion event also appears as a quiet background layer on the public page.
AMUSED and UNEASY offer two quick responses to the example artwork.
Button guidance, a waiting queue and local feedback support consecutive responses.
Abstract lines became difficult to distinguish after rotation and overlap. I returned to simplified expressions so that different responses remained distinct as they accumulated.
Voice Constellation
A selected visitor perspective (Voice) connects through a shared theme to Museum Context and Critical Voice. Other perspectives sit alongside interpretation, retaining curatorial knowledge without prescribing a single correct answer.
Design example: “The drone makes me feel like the artwork is watching me back.” → Surveillance & Public Gaze → Museum Context / Critical Voice.
The final public display uses a computer; the Redmi tablet was a development and performance-test device.
Visitor perspectives and interpretive copy are prototype examples; their sources need verification before exhibition use.
Example artwork: The End by Heather Phillipson (2020), originally commissioned for the outdoor Fourth Plinth in Trafalgar Square. The indoor prototype uses artwork imagery and reference content; the example is replaceable.
Scan the corresponding QR code to continue with the selected visitor perspective and artwork context on your phone. Save it in the phone’s browser and revisit it in My Visit Log.
Visitors can type or use editable speech-to-text, choose a theme and identity, and review their response before submitting. New contributions remain pending until approved for the public constellation.
Figma design
Key mobile screens · select to enlarge
Focus: text or editable speech-to-text input.
Focus: response preview and public-review rules.
Focus: contribution status and privately saved voices.
Earlier Figma walkthrough; a separate review step was added later.
Development and trials informed three key adjustments: the physical display’s role, the operating device and the gesture controls.
The original E-Ink proposal showed the most-saved responses, repeating content already available on the website. I changed the physical role to quick emotion response and collective accumulation; integrated RLCD / ESP32-S3 hardware also made implementation more practical.
“Most Saved” belongs to the early proposal. It is not the final private-save mechanism.
Device test readings
Laptop test: about 22 FPS
Redmi tablet: about 4–6 FPS
The target tablet’s low frame rate made the early hold rule difficult to complete reliably. I adjusted detection, tolerance and camera input load. Although hold detection could complete, overall response remained constrained by the device, so the final public gesture interaction runs on a computer.
The early rule required four stable samples in 320 ms. At approximately 22 FPS, that window could contain around seven frames; at 4–6 FPS it contained only one or two. I changed to time-based stability, a short grace period and more tolerant detection, reduced camera input to 640 × 360 and added cooldown. Hold detection could complete, while the tablet’s overall response remained constrained. These adjustments concern the development-stage hold detector; the final visitor actions are wave and fist.
Earlier lowering and holding gestures could be confused with natural movement. I simplified the final interaction and required the hand to reopen before another confirmation could be accepted.
Reopening the hand resets confirmation, readying the system for the next selection.
3 participants · informal trial · qualitative reflection
Trial notes: three participants tried physical responses, gestures and mobile continuation. Issues were reconstructed from the session and subsequent changes; no verbatim transcript was recorded.
| Observed issue | Adjustment |
|---|---|
| Unclear QR destination | Carry the selected voice ID and show a content preview |
| Typing interrupts the visit | Offer editable speech-to-text |
The connected prototype links physical input to the public emotion layer and carries selected voices to a phone. Private saving remains separate from reviewed public contribution.
Artwork adaptation
Select an adaptation image to inspect it at full size.
Layered participation, public exploration, cross-device continuation, private saving and reviewed contribution.
Artwork imagery and information, emotion prompts, themes, and curatorial and critical materials need fresh curation for each artwork and display context.