from Hacker News

Show HN: A tool for motion-capturing 3D characters using a VR headset

by diegomacario on 1/17/23, 2:03 PM with 31 comments

Hi everyone! I'm one of the authors of this project. The demo you see here is powered by a tool that I recently helped develop and open-source at Shopify called handy. You can find the repo here: https://github.com/Shopify/handy

Most people don't realize that VR headsets have become really capable motion capture platforms, so we decided to release this tool to bring motion capture into the hands of everyone who owns a headset.

With a cheap Quest 2 you can capture your hands using the headset's hand-tracking feature and your head. With an expensive Quest Pro you could capture your facial expressions using the headset's eye and face-tracking features.

Thanks for checking this project out! I'm here to answer questions if you have any.

  • by endergen on 1/17/23, 9:54 PM

    This is great, well done, and thanks for open sourcing it.

    I started to build something like this early last year, but got too busy with another pet project and having a baby. Here's some demos and notes if you're interested. https://twitter.com/convey_it/status/1433163282597171200

  • by Keyframe on 1/17/23, 10:13 PM

    With an expensive Quest Pro you could capture your facial expressions using the headset's eye and face-tracking features.

    Any more on this?

    As for hands - is it able to track hands and fingers only or also an object within a hand? I can see this being useful to 'puppeteer' a 3D object by hand used as a proxy of it... Or a simple object within a hand.

    Outside of maybe a cliff-climbing game I struggled with finding a reason for buying a headset at all. This seems like a plausible use for animation.

  • by pronlover723 on 1/18/23, 3:21 AM

    This is extremely cool!

    I haven't much of a clue on motion capture. But, the capture I want to do requires feet, hands, face, shoulders, arms, knees, fingers, AND the ability to do it in any position. Curl up in ball. Roll over on the floor. I want to capture more than just a head and hands upright but AFAIK there's no solution under $20k.

    See user name. if I want to make pr0n, I need to be able to capture those positions. Is there anything that can do it on the cheap?

  • by daniel_iversen on 1/17/23, 9:44 PM

    That’s cool! What would the typical use case for this be (and why did Shopify build it)?

  • by ngokevin on 1/17/23, 9:13 PM

    Plugging in this one we did for reference a long time ago! Capture + share dances https://blog.mozvr.com/a-saturday-night/

  • by madlag on 1/17/23, 10:05 PM

    Thanks for this! I have been working on something similar for an upcoming education app, record a course and play in the same app with a compact file format (1MB per minute, could be much less with some tricks). You can see a demo here : https://youtu.be/zcHAzQXm3Hg and more at https://explayn.me I will definitely check your lib, and will be happy to switch if it’s better !

  • by Scalene2 on 1/17/23, 11:47 PM

    This could be great for people doing work with various sign languages.

  • by cmehdy on 1/18/23, 5:09 AM

    That's great!

    I assume the hand capture (+ head position) doesn't give you enough to apply inverse kinematics reliably (i.e. better than VRchat), is that correct? It's a naive assumption on my part given that wrists aren't captured, but perhaps there are really smart ways to solve this. It feels like it could be a "next step" to have upper-body motion capture.

  • by klaussilveira on 1/17/23, 9:09 PM

    How well does it do with occlusion? Holding, or manipulating an object, and capturing those interactions.

  • by Mindwipe on 1/18/23, 12:05 PM

    This is a really great project, thanks, will definitely have a play with this and it might enable some project prototyping for me that are too small for a proper mocap setup.

  • by Scalene2 on 1/17/23, 11:54 PM

    Oh, while you're here, I recently started exploring the possibility of starting a small shop. When I looked into Shopify I had absolutely no idea how to get started or how much it cost and your website failed to even come close to answering those things in the few minutes I had to investigate. Whereas with SquareSpace provided those answers in seconds.