by mattnewport on 3/8/18, 7:56 AM
Our VR surgical training startup has been working for the last few months towards a big medical conference this week where we're showing multiple training procedures for multiple customers on Oculus Rift, as well as having our own booth. The headsets all stopped working the morning of the conference.
Fortunately one of our engineers figured out we could get our demo rigs working by setting the clock back a few days. This could have been a huge disaster for our company if we hadn't found that workaround though. Pretty annoyed with Oculus about this
by r1ch on 3/8/18, 12:45 AM
This is not how Windows code signing is supposed to work. Normally you'd get a countersignature from a timestamp server so that the verification process can prove that the certificate was valid at the time of signing. It would appear that Oculus signed their binaries without using a timestamp server, so without a way to verify when signing happened they become invalid as soon as the cert expires.
by maxander on 3/8/18, 1:30 AM
One wonders if we've made technology unnecessarily complicated. In order to build something like the Oculus Rift, they obviously needed expertise in hardware design, optics, display technology, manufacturing, user interface design, etc etc. Also, they apparently needed expertise in managing the ins-and-outs of the Windows driver security system. Adding one more subject to their already crowded curriculum wasn't very nice of Microsoft.
A lot of applications and environments seem to be built with the assumption that they can add arbitrary complexity to their interface, since they're only going to be used by "experts" who can be expected to know everything of relevance and work through a thick documentation to understand the system. In truth, the "experts" who use your programs are going to also be using a dozen other applications, each with their own piles of documentation (or equal amounts of lack-of-documentation,) and have little brain-space left for the intricacies of your framework. So, they're going to use your system while knowing the minimum possible amount about it; if that system contains traps that cause problems for this kind of user, that's bad design.
by squarefoot on 3/8/18, 2:53 PM
This one will hopefully be solved quick by the company, but think of what would have happened if this was a piece of technology sold in hundreds thousands pieces by a company now out of business: instant tons of electronic junk that would be instead perfectly useable if there was a law mandating all software/hardware details to be released if either of these conditions are met: IP owner going out of business, company declaring the product obsolete and stopping any technical support or upgrade, product sales plummeting due to competing or new models. The first two are obvious while the third one would allow some of the devices to be repurposed instead of thrown away.
I've saved a good number of old access points / routers from the landfill by installing OpenWRT/Lede where possible o their latest available firmware,pairing them together, adding homemade external antennas (small Wifi antenna enclosed in white PVC pipe plus self bonding tape, silicone sealant and heatshrink, RF240 cable and RP/SMA or N connector: => years exposed to sun, rain and snow with zero problems). I install them at really low prices to customers who need a cheap wifi bridge from point A to B. I would love to do a similar "afterlife" service to old cellphones, but none of them could host a true native Linux install because of how tightly closed the underlying hardware is, and all of them sooner than later are doomed to be thrown away.
The problem lays in the IP. It's considered to be a vital asset so that when a company goes belly up it will survive kept years or decades in a safe by law firms in the hope someone will buy it, or just to make profits through litigation against infringers. Unfortunately this has a deleterious effect on products derived from that IP, the people who bought them and the people living where the unusable products will be trashed.
by legitster on 3/7/18, 7:27 PM
They let their certificate expire, essentially bricking all of their devices. And now the app running it won't start, so they can't push an update.
Just recently picked up a Rift. I love the hardware and their exclusives are top notch, but this confirms my suspicions that their backend is super goofy.
They sell Rifts at Best Buy and want to pretend that it's a consumer-ready product, but here's why I am recommending people stay away for now:
- Non-existant repair or service out of Warranty.
- Basic things in the platform like changing your name or photo don't exist.
- Lots of non-response over other basic features requested by the community.
- Questionable future investment in the platform or hardware. It sounds like they are moving their efforts towards "lighter" experiences.
In short, it feels like being a legacy customer for a new product.
by StavrosK on 3/8/18, 12:38 AM
Why is it possible for a device that is basically a display to stop working because a certificate expired? This future worries me.
by Kikawala on 3/7/18, 9:21 PM
by jimrandomh on 3/8/18, 12:55 AM
Saw this, opened Oculus Home, there's a message in the Updates tab saying "An update may not have installed correctly", and indeed, VR apps didn't work.
Nate Mitchell of Oculus posted on Reddit saying "We're working on resolving this issue right now. We'll keep everyone posted on progress here." https://www.reddit.com/r/oculus/comments/82nuzi/cant_reach_o... . Top-level of that thread has a workaround involving setting the clock back or using a utility called RunAsDate to fake the clock for a single application.
by lunch on 3/8/18, 4:01 AM
by scrollaway on 3/8/18, 2:26 AM
This, and many incidents like it, makes me think that running tests 1/10/100 years in the future should be a standard feature of test runners and CI systems. (on by default)
by m_fayer on 3/7/18, 6:16 PM
I borrow the office Rift every couple of months to play around for a weekend and see how the field is progressing. Unfortunately what I've mostly seen is a bunch of regressions, technical and ux, as they update their platform.
by FrantaH on 3/8/18, 4:47 AM
Wow, thanks to reading HN headlines in the morning I realized my own cert expired at midnight and I was able to fix it without any damage.
by Rebelgecko on 3/8/18, 12:31 AM
It sounds like the same expired certificate is also used to sign their autoupdater's exe, so they can't just roll out an update using a new certificate.
by 8bitsrule on 3/8/18, 3:59 AM
I'm -constantly- seeing 'certificate expired' in my browser. This certificate stuff is so hard that they can't pay some Chief Certificate Officer $15/hr. to -do nothing else- but assure that stuff is renewed in a timely fashion?
We furry 'self-reproducing' (YMMV) mammals are simply not ready for all of this.
by mikeash on 3/8/18, 1:01 AM
This seems to be a somewhat common type of problem. I wonder if companies should routinely test on machines with the clock set one year into the future to catch them before they hit customers.
by retromario on 3/8/18, 9:39 AM
by Angostura on 3/8/18, 12:29 PM
Note for company communication guys.
Please, please don't say: "Our teams apologize for any inconvenience this may be causing you"
instead opt for "Our teams apologize for any inconvenience this caused you"
by rb808 on 3/8/18, 12:39 AM
Has anyone got a good way of managing certificates in the wild? With no real management and staff turnover I've seen a bunch of expired certificate problems.
EDIT: presumably you need your client apps/libraries in the field write back when they use a cert that is <X months away from expiry.
by makecheck on 3/8/18, 8:35 PM
Rotation due to expired keys should be
frequent, enough to pretty much require automated methods to handle the changes. (One of the many great things in LetsEncrypt.)
If it’s a much longer time scale, people start to forget that it’s even possible for stuff to expire.
If my fridge filter can display a little reminder light on a timer every few months, cryptography-dependent devices might need something similar. That way, your customers could know in advance and be asking you for an update.
by khazhoux on 3/8/18, 1:17 AM
In 2091, an overworked developer will accidentally let the certificate expire for the Planetary Shield Defense Matrix, and the Zylorts will finally conquer Earth.
by agar on 3/8/18, 4:28 PM
OK. The issue arose because the expired certificate wasn't countersigned by a timestamp server.
So many comments agree that (a) security is hard, (b) countersigning with a timestamp server is easy to miss, (c) countersigning makes build processes difficult, and (d) they've done or seen similar things in other apps/companies.
This sounds like a classic UI/UX issue for developers around a literally mandated and mission-critical requirement of the OS.
At the least, MS should provide a validation tool to surface errors or risks before production. Better, signtool.exe should make omissions (like a timeserver) very difficult and make them an override, not a default. Best, they would do both.
I don't agree that the OS should reject non-timestamped signatures as faulty per se (and throw an error), as that puts the burden on the user to understand a developer's mistake. Sometimes running without a timestamp may be desirable - ultimately that's the dev's choice.
It should just be a choice made explicitly.
by navium on 3/8/18, 2:18 AM
This is what happens when a CTO goes for a programming retreat
by AHTERIX5000 on 3/8/18, 12:39 AM
So it's impossible to use Oculus headset locally without runtime calling home and checking something?
by juanmirocks on 3/8/18, 8:29 PM
I feel more understandable with Oculus in this one... Recently I also left an certificate, expire...
However, this affected only one single customer of ours and we had a fix within a couple of hours. -- I certainly learn from this mistake.
by logicuce on 3/8/18, 2:25 PM
I am yet to find any official statement from Oculus on this. Am I missing something?
by toomasr on 3/9/18, 8:06 AM
by nottorp on 3/8/18, 12:42 PM
So you're saying Rifts and Windows 10 drivers do not work offline? That basically Windows 10 will be functional only while Microsoft keeps the update servers on?
Edit: I don't follow Windows, I'm really curious what the consequences for stuff like this can be generally.
by sneak on 3/8/18, 12:31 PM
Minor nitpick: “soft-bricked” is like “soft-pregnant”. That’s not what “bricked” means, Techcrunch.
by rixrax on 3/8/18, 9:00 PM
Oculus says you will receive $15 store credit if you used Oculus between Feb 1st and when it went kaput.
I don't see credit on my Oculus account? Am I supposed to have received it already? Or is this maybe because I don't have payment method added to my account?
by acd on 3/8/18, 4:33 PM
Why does a device you own have to have an expiring certificate?
by Robotbeat on 3/8/18, 4:20 AM
Ha! My dad was just telling me how some of our old SANs had this happen. Just had to change the date to 2017 then do a firmware update.
by kakarot on 3/8/18, 12:58 AM
Well, that explains why my Oculus wasn't working tonight. I hope this ends up being as easy to fix as downloading a new binary.
by melvinmt on 3/8/18, 1:46 AM
Move Fast and Break Things.
by theonewhocanfly on 3/8/18, 12:51 PM
The site has aggressive ads
by intoro on 3/8/18, 1:12 AM
Something similar just happened to me. I have a windows computer I only Use for gaming. After the last update My Samsung display is no longer usable. It has a polarized effect now only when using the windows Computer. However the Computer Works fine Connected to another brand monitor. So much money, yet windows still sucks when it comes to most basic things
by peterwwillis on 3/8/18, 2:29 AM
I've pointed out this consequence of "put TLS on everything" before, but people shrugged it off.
by tritium on 3/8/18, 12:42 AM
Beside the fact that you should be concerned about whether the controlling company goes out of business, or sells your data, here stands yet another reason to never trust devices that require an internet connection to activate in the first place, or phone home periodically to remain active.
This includes phones, cars, self-driving cars, watches, farm equipment, computing devices and anything marketed as an IoT appliance.
One glitch, as minor as an improper system time, and you’re dead in the water.