This project was created as part of an assignment for Programming 8.
Come up with a concept for working with facial expression recognition, body pose recognition, hand pose recognition, object detection, or the image feature recognition. Read data with javascript and provide feedback to the user via the UI. Build a simple UI for this concept using HTML and CSS. So the user does not have to look into the console.
Live demo 👉 here. View Project 🎬 : https://www.youtube.com/watch?v=nk0pNSxGqP0
-
What is the purpose of the experiment? GestureBeat is a tool to make music with your hands.
-
What is the data (input) and what is predicted (output)? The data used as input is the landmarks of each finger. I have chosen 2 points of the landmarks per finger and made a small calculation on the x and y-axis. Then if it is detected via your camera, it will play a sound.
In this way I have programmed certain "hand poses".
-
Which algorithm or library is used? I used Media Pipe library. Media Pipe is from Google and uses Machine Learning technology.
-
Has the goal been achieved? Can you use the acquired knowledge for the final project? The goal has been achieved. However I could optimize it even more because it mixes up certain "poses" sometimes.
-
What did you find difficult or easy about programming? I found it quite difficult to achieve my goal (playing music through hand poses detection) because I had researched different libraries and each one got stuck at some point. I've used p5, ml5, media pipe and handsfree.js and finally chose Media Pipe because I found it faster and more accurate.
- Hand pose recognition
Project is: completed
- Optimalization of the hand pose recognition.
- Additional sounds
Created by @Creative.Dev - feel free to contact me!