Creating a 3D Hand Controller Using a Webcam with MediaPipe and Three.js

submited by

Style Pass

2024-10-26 12:30:03

From our sponsor: Meco is a distraction-free space for reading and discovering newsletters, separate from the inbox.

I love interactive things, but I’m not a fan of expensive devices, especially those we have to buy year after year to have the latest hardware in our hands!

With that in mind, today I’m going to show you how to control elements in a 3D scene using just your hands, a webcam, and a web browser. The key focus here is converting a 2D screen into a 3D space, with full depth control. I’ll keep the focus on that!

I have created a class called MediaPipeHands that contains this implementation, so we can easily add it to the code and reuse it anywhere.

The onMediaPipeHandsResults is a callback from the library that returns the mentioned landmarks, it comes from hands.onResults. Now, let’s create our WebGL scene with Three.js, only to have some elements we can interact with. Nothing special, a simple Three.js scene 😉

Here’s another class for rapid prototyping, a plug-and-play static class that can be accessed anywhere in your code. Keep in mind that it doesn’t follow any standard, it’s just to make my life easier.