from Hacker News

If you're interested in eye-tracking, I'm interested in funding you

by pg on 8/27/23, 12:51 AM with 209 comments

  • by Componica on 8/27/23, 4:38 AM

    My three partners and I have be developing and selling multi-camera arrays specifically for eye tracking as well as measuring other physiological features for several years now. Our main customers are a couple university research groups, a human factors group in Lockheed, and just recently the US Air Force. In fact we just returned from a trip to Wright-Patterson installing an array in a hypobaric chamber to perform gaze tracker and pupil response for pilots under hypoxic conditions. Phase two will be a custom gaze tracker for their centrifuge. Our main features are accurate eye and face tracking up to a meter from the array, minimal calibration per subject (about 10 seconds staring at a dot), pupil response for measuring fatigue and other things, plus we can adapt the array for the client ranging from a cockpit to a large flat screen TV. We've looked into medical usage such as ALS, but we're bootstrapped based in Iowa and found the military niche as a more direct way to generate cash flow. It's ashame we can't apply this work towards people with medical needs, but we don't have the funds nor the clients to make such a pivot.

  • by AndrewKemendo on 8/27/23, 1:44 AM

    I responded to the thread but Senseye has been working on this for a while now. Originally they were working with the US Air Force to help with improving pilot training - fatigue etc.. inference with retinal reading

    https://senseye.co/

    They have generally struggled to find funding for their eye tracking focused work, and have recently had to pivot away from the really exciting but hard to fund stuff into PTSD screening (which is important too).

    I can connect you with the founder if desired via the email in my bio

  • by justinlloyd on 8/27/23, 4:21 AM

    I do hardware. I do software. I do computer vision. I built some software that ran on a cellphone used by LEO (law enforcement officers) to determine if the person they are quizzing is inebriated or impaired through controlled substances by examining the person's eyes and having them focus on images displayed on the phone screen. I've done eye tracking using fully custom solutions and also through a few of the off-the-shelf SDKs such as GazeSense from eyeware and a few other SDKs.

    The problem is not the eye-tracking, it is reasonably easy to build robust systems that can do that easily enough, even with custom hardware under all sorts of lighting conditions. The hard part is the UX if you are trying to build something that isn't hampered by current UI paradigms.

    Rapid typing and menus of custom actions with just eye movement, though fatiguing, shouldn't be hard to solve, and then render the output however you want; text, text to speech, commands issued to an machine, etc. Making a usable user interface to do anything else, that's where the rubber hits the road.

    @pg, which software is your friend using? If it is anything like I've looked in to in the past, it's over-priced accessibility crap with a UI straight out of the 1990s.

  • by dewarrn1 on 8/27/23, 1:43 AM

    EEG recording is an alternative that would outlast the potential disease-related degradation of eye movements. Manny Donchin gave a brown bag at UIUC about the possibilities of using this approach to support communication by ALS patients many years ago. It's clever: they use the P300 marker to index attention/intention. I do not recall whether he and his colleagues ever commercialized the tech. I believe that this publication is representative: https://doi.org/10.1016/j.clinph.2005.06.027

  • by blackguardx on 8/27/23, 1:46 AM

    I worked on eye tracking hardware for Microsoft HoloLens. Several AR headsets offer decent eye tracking, including Hololens 2 and Magic Leap's ML2. I think Tobii's eye tracking glasses are probably better as a stand-alone solution though: https://www.tobii.com/products/eye-trackers/wearables/tobii-...

  • by sam_goody on 8/27/23, 11:48 AM

    So, guy who has deployed eye-scanning machines all over Africa and has found that many of them have been hacked and are giving incorrect responses suddenly has a friend with ALS and is willing to fund better quality eye tracking?

    Either:

        - Part of the whole world-coin thing was privately trying to get the data to help his friend
    - He doesn't want to say "looking to develop eye tracking tech for my world-coin scam", since most devs won't touch that thing. Conveniently found a "friend" with ALS.
    Saying, on behalf of a friend, that he doesn't believe PG.

  • by musesum on 8/27/23, 2:58 AM

    I've been working on the menuing side [1] based on crossing Fitt's Law with Huffman trees. But, don't know the constraints for ALS.

    Hopefully, whomever takes this on doesn't take the standard Accessibility approach, which is adding an extra layer of complexity on an existing UI.

    A good friend, Gordon Fuller, found out he was going blind. So, he co-founded one of the first VR startups in the 90's. Why? For wayfinding.

    What we came up with is a concept of Universal design. Start over from first principles. Seeing Gordon use an Accessible UI is painful to watch, it takes three times as many steps to navigate and confirm. So, what is the factor? 0.3 X?

    Imagine if we could refactor all apps with a LLM, and then couple it with an auto compete menu. Within that menu is personal history of all your past transversals.

    What would be the result? A 10X? Would my sister in a wheelchair be able to use it? Would love to find out!

    [1] https://github.com/musesum/DeepMenu

  • by kubi07 on 8/27/23, 2:20 PM

    There is a Turkish ALS patient, he has a youtube channel, he is creating youtube videos, podcasts, streams on twitch thanks to eye tracker.

    He is using tobii eye tracker. There is a video he made about the eye tracker. It's in Turkish but you can see how he uses it.

    https://www.youtube.com/watch?v=pzSXyiWN_uw

    Here is a article about him in English: https://www.dexerto.com/entertainment/twitch-streamer-with-a...

  • by fartjetpack on 8/27/23, 1:42 AM

    Another route might also be sub-vocalization[1], like TTS for your thoughts. I recently picked up some cheap toys to get started trying to emulate the results[2].

    1. https://www.nasa.gov/centers/ames/news/releases/2004/subvoca...

    2. https://github.com/kitschpatrol/Brain

  • by fastball on 8/27/23, 1:25 AM

    Is the lack of mentioning Apple deliberate? It seems like they've already poured a lot of R&D into this for the Vision Pro, which might be exactly the kind of thing the friend needs.

  • by readyplayernull on 8/27/23, 1:49 AM

    > A friend of mine has ALS and can only move his eyes. He has an eye-controlled keyboard, but it's not very good. Can you make him a better one?

    When I worked for one of the big game engines I got contacted by the makers of the tech that Stephen Hawking used to communicate, which includes an eye tracker:

    https://www.businessinsider.com/an-eye-tracking-interface-he...

  • by lostdog on 8/27/23, 10:38 AM

    I would love to hear pg's analysis of the business case for this company.

    By my math, 5k people in the US are diagnosed per year, and if your keyboard costs $1k, then your ARR is $5m, and maybe the company valuation is $50m. Numerically, this is pretty far from the goal of a typical YC company.

    I hate to be so cold-hearted about the calculations, but I've had a few friends get really passionate about assistive tech, and then get crushed by the financial realities. Just from the comments, you can see how many startups went either the military route or got acquired into VR programs.

    The worst I've seen, btw, is trying to build a better powered wheelchair. All the tech is out there to make powered wheelchairs less bulky and more functional, but the costs of getting it approved for health insurance to pay the price, combined with any possible risk of them falling over, combined with the tiny market you are addressing makes it nearly impossible to develop and ship an improvement. I do hope that we reach a tipping point in the near future where a new wheelchair makes sense to build, because something more nimble would be a big improvement to people's lives.

  • by caspar on 8/28/23, 9:22 AM

    As someone who suffered some severe mobility impairment a few years ago and relied extensively on eye tracking for just over a year, https://precisiongazemouse.org/ (Windows) and https://talonvoice.com/ (multiplatform) are great. In my experience the hardware is already surprisingly good, in that you get accuracy to within an inch or half an inch depending on your training. Rather, it's all about the UX wrapped around it, as a few other comments have raised.

    IMO Talon wins* for that by supporting voice recognition and mouth noises (think lip popping), which are less fatiguing than one-eye blinks for common actions like clicking. The creator is active here sometimes.

    (* An alternative is to roll your own sort of thing with https://github.com/dictation-toolbox/dragonfly and other tools as I did, but it's a lot more effort)

  • by splatcollision on 8/27/23, 1:42 AM

  • by arketyp on 8/27/23, 5:17 AM

    Tobii have been doing eye-tracking since 2001 and have a product for that. https://www.tobiidynavox.com/

  • by bacon_waffle on 8/27/23, 8:56 AM

    Edit: Check out Dasher for a much better interface to enter text with a cursor, compared to a virtual keyboard.

    https://dasher.acecentre.net/ , source at https://github.com/dasher-project/dasher

    ---

    I remember seeing a program years ago, which used the mouse cursor in a really neat way to enter text. Seems like it would be far better than clicking on keys of a virtual keyboard, but I can't remember the name of this program nor seem to find it...

    Will probably get some of this wrong, but just in case it rings a bell (or someone wants to reinvent it - wouldn't be hard):

    The interface felt like a side-scrolling through through a map of characters. Moving left and right controlled speed through the characters; for instance moving to the left extent would backspace, and moving further to the right would enter more characters per time.

    Up and down would select the next character - in my memory these are presented as a stack of map-coloured boxes where each box held a letter (or, group of letters?), say 'a' to 'z' top-to-bottom, plus a few punctuation marks. The height of each box was proportional to the likelihood that letter would be the next you'd want, so the most likely targets would be easier+quicker to navigate to. Navigating in to a box for a character would "type" it. IIRC, at any instant, you could see a couple levels of letters, so if you had entered c-o, maybe 'o' and 'u' would be particularly large, and inside the 'o' box you might see that 'l' and 'k' are bigger so it's easy to write "cool" or "cook".

    (I do hardware+firmware in Rust and regularly reference Richard Hamming, Fred Brooks, Donald Norman, Tufte. Could be up for a change)

  • by maccard on 8/27/23, 10:26 AM

    Huh. I wrote a paper for my undergraduate dissertation on eye tracking using a laptop camera, and it ended up published and I won a scholarship award (for €150, imagine that). I wonder if it's time to dust off that project

  • by gwurldz on 8/27/23, 1:32 AM

    This seems like it would fit

    https://thinksmartbox.com/products/eye-gaze/

    I once interviewed at this company. Unfortunately didn't get the job but very impressed nonetheless.

  • by modeless on 8/27/23, 4:27 AM

    I agree, eye tracking is going to have really broad applications. I've been interested in eye tracking for over a decade, and in fact built my own eye tracker, joined a startup, and got acquired by Google[1]. But there's way more to do. We've barely scratched the surface of what's possible with eye tracking and I'd love to take a second crack at it.

    [1] https://techcrunch.com/2016/10/24/google-buys-eyefluence-eye...

  • by sprocket on 8/27/23, 3:49 AM

    I used this software when my mom was battling ALS:

       https://www.optikey.org/
    which ran on a < $1k computer

    At the time, the other options were much more expensive (> $10-15k) which were sadly out of out budget.

  • by sailplease on 8/27/23, 2:58 AM

    Adhawk, adhawk.io, has the only all day ultralight eye tracking wearable I'm aware of, all MEMS based with ultra high scan rates, 500Hz+ and research grade accuracy. For ALS u likely need something light and frictionless, wearing a hot and heavy headset all day probably doesn't work.

  • by yyyk on 8/27/23, 10:35 AM

    I suspect the real moneymakers for such startups have very little to do with ALS. ALS demand is fortunately small, and can't lead to VC desired growth curve. Imagine instead using it in a classroom to ensure the kids pay attention. Or making sure you see the advertisement.

  • by acyou on 8/27/23, 5:53 AM

    Yes, the ALS/disability angle is noble. Viewed another way, the entire human race is afflicted by the disability of not having access to eye-tracking (and other) technologies. Paul Graham and co. are also invested in companies that are going to be highly enabled and boosted by the growth of eye-tracking and related technologies. I don't view his statement of motivation related to ALS as insincere, I just also notice that it's accessible, easily understandable, and also in line with other aspects of Paul's motivation (and that's a good thing).

    I would also recommend Jean-Dominique Bauby's Le Scaphandre et le Papillon to anyone interested in this topic. Typing using eye movements was used in that book in a slow, inefficient manner. In the book's case, the question one should ask is, was his UI paced at the exact correct speed? I was and still am deeply emotionally moved by what the author was able to accomplish and convey. I am unsure if a faster keyboard would have made a meaningful and positive difference in that particular case, to the author's quality of life. I'll need to give that book another read with that question in mind.

    Happily, I expect eye tracking to find fascinating, novel and unexpected applications. As others have stated, UI/UX design is an interesting part of this puzzle. For example, if you ask an LLM to output short branches of text and have a writer look at the words that he wants to convey. It's definitely blurring the line between reading and writing. Myself, finding writing to be a tactile exercise, I think that emotional state comes into play. That's what I'm interested in. Yes, can you literally read someone's eyes and tell what they are thinking?

  • by ZeroCool2u on 8/27/23, 3:38 AM

    I literally just bought this last night. Works with just a webcam and is shockingly accurate. https://beam.eyeware.tech/

  • by zefzefzef on 8/27/23, 1:44 PM

    I'm very interested in eye-tracking and see a lot of potential in this tech.

    For inspiration, check out the Vocal Eyes Becker Communication System: https://jasonbecker.com/archive/eye_communication.html

    A system invented for ALS patient Jason Becker by his dad: https://www.youtube.com/watch?v=wGFDWTC8B8g

    Also already mentioned in here, EyeWriter ( https://en.wikipedia.org/wiki/EyeWriter ) and Dasher ( https://en.wikipedia.org/wiki/Dasher_(software) ) are two interesting projects to look into.

  • by anupamchugh on 9/5/23, 9:39 PM

    Does something like a blink, wink tracker for swiping UI pique your interest? I built a PoC a while back: https://github.com/anupamchugh/BlinkPoseAndSwipeiOSMLKit

  • by Schwolop on 8/28/23, 2:50 AM

    I'm consulting with an Australian group called Control Bionics. They have a US company & office, with CTO and sales team in Ohio, but software engineering is done in AU. Their primary product is an electromyography and accelerometer hardware device to detect muscle activations or movements, and then most commonly used as a mouse-click substitute in conjunction with third-party eye-gaze hardware proving the cursor. (I've also designed them an autonomous wheelchair module, but that's another story...)

    @pg - If your friend has not tried adding a mouse-click via something they can activate other than eye-gaze, this would be worth a shot. We have a lot of MND patients who use our combination to great success. If they can twitch an eyebrow, wiggle a toe or a finger, or even flex their abdomen, we can put electrodes there and give them a way forward.

    Also, my contact details are in my profile. I'd be happy to put you in touch with our CEO and I'm confident that offers of funding would be of interest. The company is listed on the Australian stock exchange, but could likely go much further with a direct injection of capital to bolster the engineering team.

    Cheers, Tom

  • by mhb on 8/27/23, 1:40 PM

    In responses, there seem to be dozens of experts and companies already doing this. Where does he think they fall short of meeting his friend's needs?

  • by mercurialsolo on 8/27/23, 5:49 AM

    Eye tracking is essentially a model of visual attention. Visual attention is part of the overall attention space and big companies and use-cases are built around visual attention. Today we track attention by explicit interactions, if we can model around implicitly observable interactions - then we have a much larger observable data space around the user.,

  • by claytongulick on 8/27/23, 12:08 PM

    I did a fun project a few years ago with eye tracking.

    We built a prototype for roadside sobriety checks. The idea was to take race/subjectivity out of the equation in these traffic stops.

    We modified an oculus quest and added IR LEDs and cameras with small PI zero's. I wrote software for the quest that gave instructions and had a series of examinations where you'd follow a 3D ball, the screen would brighten and darken, and several others while I looked for eye jerks (saccades) and pupil dilation. The officer was able to see your pupil (enlarged) on a laptop in real time and we'd mark suspicious times on the video timeline for review.

    It was an interesting combination of video decoding, OpenCV and real-time streams with a pretty slick UI. The Pi Zero was easily capable of handling real-time video stream decoding, OpenCV and Node. Where I ran into performance problems I wrote node -> c++ bindings.

    We did it all on something silly like a 50k budget. Neat project.

  • by archo on 8/27/23, 9:51 AM

  • by dimask on 8/27/23, 11:53 AM

    What do you think are some challenges that an eyetracker in this specific context has to face? What is your friend mostly struggling with the current solutions? Are there tracking specific challenges related to ALS? Is it mostly a UI/"better prediction" interface issue?

    With my group we are developing an eyetracker for studying developmental and clinical populations, which typically present challenges to conventional eyetrackers. It is a spin off from our academic work with infants, and we already have a study almost done that uses it. We are still into the very beginning phase in terms of where this may lead us, but we are interested in looking into contexts where eyetracking for different reasons may be more challenging.

  • by MasterYoda on 8/27/23, 8:59 AM

    PG mention that the solution his friend used wasn't any good. How does the best system there is out today work? And what different solutions are there?

  • by imranq on 8/27/23, 1:15 PM

    I found this tool to be interesting to play with and seems to work pretty well assuming you stay in the same position: https://gazerecorder.com/

    I'm guessing a combination of projection mapping, built in lighting, and some crowdsourced data will get accuracy to very usable levels

  • by user3939382 on 8/27/23, 2:21 AM

    How about a library that starts loading a link when you look at it with intent. Or maybe with BCI integration that detects the moment you decide you want to access it.

    Or how about a UI that automatically adapts to your eye movement and access patterns to minimize the amount of eye movement required to complete your most common tasks by rearranging the UI elements.

  • by ricardobayes on 8/27/23, 10:29 AM

    I thought this was solved long time ago, I wrote a program many years ago using kinect that tracks the center of the eye pretty precisely, using color gradients. The pupil is pretty uniform in every human being (it's black) surrounded by some color and then white. Even just a few pixels are enough to do it.

  • by PBnFlash on 8/27/23, 1:32 AM

    I suspect a foveated system is going to be a big thing in machine vision as well.

  • by quietthrow on 8/27/23, 4:26 AM

  • by TheGuyWhoCodes on 8/27/23, 5:22 PM

    https://www.eyecontrol.co.il/ was founded exactly to solve this problem

  • by peter_retief on 8/27/23, 8:06 AM

    I would like to look at the problem more deeply, the eyes can be tracked but what about facial movement, the more data the better training for machine learning

  • by amelius on 8/27/23, 9:42 AM

    I hear the Apple Vision Pro has a good implementation. If this were Microsoft, you'd be able to find the details in the Microsoft Research website.

  • by tmalsburg2 on 8/27/23, 6:37 AM

    Solving eye-tracking keyboards is not so much a task for a company with eye-tracking expertise but for one with expertise in large language models.

  • by DoingIsLearning on 8/27/23, 5:56 PM

    Is there any solution out there that does not use IR + dark pupil segmentation?

    Seems like all the solutions out there are some flavour or variation of this.

  • by 6stringmerc on 8/27/23, 6:07 AM

    Jason Becker is a great subject because if you can help him compose with his eyes the world can use his music he’s a genius.

  • by joshm93 on 8/27/23, 6:52 AM

    I can make an eye tracking keyboard with tensor flow, if anyone is interested in this problem.

    It would be great to hear from paul about how his friend uses the keyboard and what kind of tasks he’d love to do but can’t with current solutions.

    It seems like a throughput problem to me. How can you type quickly using only your eyes?

    Have people explored using small phonetic alphabets or Morse code style encoding?

    Once I got tensorflow working, I’d start mapping different kinds of ux. Throughput is king.

  • by jjbcb on 8/27/23, 3:32 AM

    I’m sorry this happened to your friend. I hope we can do something to help them.

  • by frakkingcylons on 8/27/23, 1:51 AM

    I hope this effort bears fruit. My uncle passed from ALS eight years ago.

  • by kken on 8/27/23, 2:19 AM

    In case anyone is interested: There are plenty of companies around.

    Both apple and Facebook acquired eye tracking companies to kickstart their own development.

    Here are some Top-lists

    https://imotions.com/blog/insights/trend/top-eye-tracking-ha... https://valentinazezelj.medium.com/top-10-eye-tracking-compa...

    Its also an active research field, this is one of the bigger conferences: https://etra.acm.org/2023/

  • by quickthrower2 on 8/27/23, 3:09 AM

    What about brain waves to control the keyboard?

  • by jacquesm on 8/27/23, 4:10 AM

    Whatever happened to GazeHawk and their crew?

  • by hardwaregeek on 8/27/23, 1:33 AM

    Aw, that's nice of pg to want something better for his friend. As cynical as we are about technology, new developments can be so fantastic for accessibility and better quality of life.

  • by rsync on 8/27/23, 6:48 AM

    So you're saying there's a final frontier in the mapping of intention and the discernment of preferences ... and you'd like some bright young things to explore that possibility space under the guise of (unimpeachable good cause) ?

    Go to hell.

    Unless, of course, you'd like to commit the funded work to the free commons, unencumbered by patents and copyrights, and free to use by any entity for any purpose.

    That's what we'd do for ALS, right ?

  • by aaron695 on 8/27/23, 1:27 AM

    https://nitter.net/paulg/status/1695596853864321055 to see more of the thread if not logged in.

    It'd be good to know what rate we need to beat and some other metrics.

  • by atleastoptimal on 8/27/23, 4:04 AM

    I bet Apple Vision pro eye tracking is already SOTA and will remain that way for years due to their crazy R&D spending

  • by dennis_jeeves1 on 8/27/23, 12:48 PM

    And now if only someone funded an ALS cure...

    As far I know, I don't don't think mainstream medicine is close to solving _any_ chronic condition, except managing it.

  • by turnsout on 8/27/23, 1:38 AM

    Someone should do this, but for the love of god, DO NOT take any venture capital to do it. No matter how well-intentioned the VCs are at the start, eventually your eye tracking startup will 100% be used for advertising as investors in your Series D need to make an exit or take you public.

  • by soligern on 8/27/23, 1:24 AM

    It feels like Apple has solved this problem with the Vision Pro? So just wait for a couple more months?

  • by morkalork on 8/27/23, 1:46 AM

    Please look directly at the screen to continue

    Whilst it plays an unskippable and unblockable ad (thanks weiapi!)

  • by FlamingMoe on 8/27/23, 1:39 AM

    Not really a huge PG fan, but this is what billionaires should be doing: see where a need is exists and put some of your insane wealth towards making an improvement. This why I respect Elon even though I don’t really like him; he puts his money to use, in a very public manner.

  • by adamnemecek on 8/27/23, 2:28 AM

    I have a new approach of doing ML, where autodiff is replaced with something better. Magically a lot of things fall into place. This approach should make problems like this relatively straight forward.

    Interested in hearing more?