So I finally got my hands on 2 kinects through someone from my network. I got a 360 and an xbox one kinect. As I did research to how the kinect would operate I overlooked the fact that both used different adapters. And I only had one for the xbox one kinect. Normally I was planning to compare both, since I found out that the xbox one kinect has a better FOV, lagged less because of a better sensor which would be resulting in way better bodytracking overall. Since I only got my hands on the kinect as of now and it was too late to order a 360 kinect adapter I had to skip the overall comparison. And firstly focus on getting my current grafitti wall setup to work with the kinect. I did a couple of experiments with the kinect which should be included,can be dummy ones for now with a link to their individual blog post and a place for little summary. As a final test I also converted the existing grafitti installation to work with the kinect data. Increasing accuracy and adapting to different lighting conditions. As I only got my hands on the kinect the weekend before the deadline my time for experimenting was short but proved to be a very powerful tool if you’re looking to get into advanced bodytracking.

Now for the kinect grafitti post; Because I was having some accuracy problems as of now with the mediapipe model losing the hands when moving too fast, turning hands and general movement in front of the projection. I knew already that the kinect provides a similar output of body joint data as the mediapipe model so it was only a question of finding the right kinect mode, selecting the right joints and feeding these to the existing setup. After doing a live test in the same conditions as the mediapipe setup I quickly found out this was a lot more accurate and so really improved the process. I will include a picture of the nodes, video of the nodes in action and both a video of the output(projected installation image) next to a setup recording so you can see the setup.

For the next post I want to get into one of the bodytracking methods using the nvidia bodytrack chop on the rented windows computer. As for the bodytrack chop I actually followed a youtube tutorial (https://www.youtube.com/watch?v=ETQLq1EvXJI&list=PLNcHiqcsI4naQjwCY1H0-kWCAWuKh1Tw8&index=49) on how to connect the kinect joint data and turn this into a skeleton. Which I would then be able to use as a mask.

For testing purposes, and so i don’t have to stand in front of my webcam all the time, I recorded a video of myself outside walking in front of a wall, waving my hands in a way a visitor might do when they want to reveal certain parts of the hidden room. You can include the video in the blog post. So i first would connect the bodytrack chop to the video. This would generate joint data once I walked into the video frame. Afterwards I used sever other CHOP’s (which you can analyse and turn into different steps based on the node image i included) to then eventually turn into a mask. Which I could then again feed to the masking setup. I will include a video of the final nodes with the video playing. And another video where you can see a side by side view of the walking video and the result on the mask. Which should then be projected. Since i did not have a projector in this time I did not do any further testing on this technique. As I found that the bodytrack chop does not provide a mask that is accurate enough to work in an actual installation. I will also include a screenshot of the nodes and the final result video for you so you can analyse and create the according steps.

Now that I’ve built the functionality with the mouse trigger I want to replace the mouse triggers for detecting a fist. Since the media pipe AI doesn’t have any fist detection; I will have to figure out a different way. After taking a closer look intro the different body points the hand tracking feature detects I have found that it detects many different points in each individual finger. When making a hand there is more distance between these than when you are holding your hand open. If I could find a way to calculate the distance between these points, analyse the changes in distance I can set some bounds for a fist and hand and create a 0/1 output depending on which bounds it is in. I found out during my initial TD tutorials that there are script chops which can take data, transform it and output it again. These scripts are written In python which is a language I have not really used a lot before so I will be using ChatGPT to assist me in this process.