Concept Development: Interactive 3D Graffiti Wall

Category: Concept Development, Interactive Design

1. Introduction

After reconsidering my past work. I though about experimenting with 3D Line Tracing , And i could scale up the idea and apply it to a larger, interactive experience. This led me to the concept of a digital graffiti wall, where users can paint virtually using body tracking and real-time 3D rendering.

This post defines the concept, the technical approach, and expected challenges. The actual creation and implementation will be covered in a later post.

2. The Idea: A Digital Graffiti Wall

The goal is to create a projected 3D graffiti wall where visitors can use their hands to spray digital paint onto a brick wall in real time.

How It Works

  • A 3D brick wall is projected onto a real white wall.
  • A webcam tracks a visitor’s hand movement using AI hand tracking (Mediapipe). Previously explored here with mask reveals
  • A virtual spray can appears at the hand position.
  • The spray can triggers a paint effect when a visitor makes a fist.
  • The spray follows the hand’s movement, leaving a painted trail in 3D space.

This allows visitors to create their own graffiti art, whether it’s a name, a simple drawing, or abstract designs.

3. Why This Concept?

I developed this idea as a follow up to a previous post where I realised that creating the original interactive 3D wall I was planning to be too much work. to connect multiple experiments into one cohesive digital installation. Since I aim to present a working interactive installation on campus, this project allows me to:

  • Apply my previous work on 3D line tracing.
  • Experiment with AI-driven body tracking.
  • Merge real-time graphics with user input.
  • Create a meaningful, participatory installation.

4. Technical Breakdown: Step-by-Step Plan

Below is the current workflow. Some steps are clear, while others will require more problem-solving along the way.

Step 1: Create the 3D Assets

To achieve realism, I need custom 3D models in FBX format that can be imported into TouchDesigner.

  • Wall Model – A simple brick texture to act as the canvas.
  • Graffiti Spray Can Model – Includes:
    • A pressing animation (cap moves down when triggered).
    • A spray effect (smoke or mist coming from the cap).
  • Pointer Model – A 3D circle to indicate where the paint will land.

Tools: Blender (leveraging knowledge from my Expert Design class).

Step 2: Import & Position the 3D Assets in TouchDesigner

Once the assets are built, they need to be placed correctly in the 3D scene in TouchDesigner.

  • The wall should be static and serve as a background.
  • The spray can should be positioned dynamically, linked to user movement.

Step 3: Prototype with Mouse Input

Before integrating body tracking, I will first use the mouse as a testing method.

  • The spray can follows the mouse cursor.
  • Clicking the mouse triggers the spray effect.

This allows me to debug core interactions before moving to body tracking.

Step 4: Transition to Body Tracking

Once the mouse-based prototype is stable, I will:

  • Replace mouse tracking with AI-powered body tracking.
  • Make the spray can follow the user’s wrist in real time.
  • Detect fist gestures to activate the spray effect.

Tools: Mediapipe AI tracking for hand detection.

Step 5: Handling Paint Trails & Line Generation

One of the biggest challenges is generating persistent paint trails in 3D space.

  • Lines must merge together seamlessly rather than reset every frame.
  • The spray paint should behave naturally, following the hand’s path.

Problem to solve: I am unsure how to merge multiple lines efficiently in TouchDesigner.

Optional Features (If Time Allows)

If I manage to complete the core functionality, I would like to expand the experience with:

  • Add call to actions and user feedback
  • Sound effects – Adding a realistic spray sound when painting.
  • Color & width options – Letting users change spray color and thickness.

5. Challenges & Open Questions

Although I have a clear plan, some technical aspects will require further experimentation.

1. How to Merge Spray Lines?

  • Right now, I don’t know how to connect spray strokes in a realistic way.

2. Gesture Recognition Accuracy

  • The fist detection needs to be precise to avoid false triggers.

3. Performance & Stability

  • Real-time rendering of 3D objects and motion tracking can be performance-intensive.

6. Conclusion & Next Steps

This concept lays out the foundation for a fully interactive 3D graffiti experience. Before I can bring it to life, my next step is to start building the components one by one.

Next Blog Post:
I will begin the creation phase, starting with Building and importing the 3D assets into TouchDesigner scene.

This project is an opportunity to combine motion tracking, 3D interaction, and user participation, making it an exciting step toward a final interactive installation.