from Hacker News

Windows feature that resets system clocks based on random data is wreaking havoc

by dddddaviddddd on 8/16/23, 6:11 PM with 162 comments

  • by londons_explore on 8/16/23, 8:40 PM

    I wonder if you could use this 'feature' to exploit a system?

    Set up a bunch of servers all over the internet with innocuous web pages. Get all of them to include in their SSL headers the exact identical timestamp of July 5th 1998.

    Then get the user to connect to all those domains (eg. with a page with a bunch of iframes).

    The Secure Time service will see that lots of remote servers all agree with high confidence that the date is July 5th 1998. So the system clock gets set to then.

    Then you use a leaked yet expired cert (or maybe one signed with a broken algorithm) to impersonate an update server or steal valuable cookies/tokens.

    Expired certs typically fall out the back of revocation systems too - so it really is just the expiry date/time that protects against their use.

  • by jmuguy on 8/16/23, 7:17 PM

    Windows Time bullshit was one of the most annoying things I dealt with during my years as an IT guy. Registering and unregistering w32time, trying different NTP servers. Trying to figure out why domain systems werent getting their time from the DC. It always felt so... stupid. Surely having the correct time on a device isnt that complicated. Turns out, its not, unless you're on Windows. Somewhat ironic that these days the only Windows system I have to deal with is my gaming PC. It refuses to sync with time.windows.com.

  • by JohnFen on 8/16/23, 7:49 PM

    That feature is simply insane. How did MS think this was a good idea?

    To address the issue they were trying to address (what happens when a mission critical server's RTC malfunctions or the battery dies?), Windows should treat it no differently than any other hardware malfunction in a mission-critical server: raise an alarm so a system operator can take a look and address the issue.

  • by yetanotherloss on 8/16/23, 7:07 PM

    I can't imagine the sequence of horrible decisions that led to doing this. Like, why would the time service ever want to depend on all this insanity when if it has a network and everything else is bizarro world, just like scrape the time and date text from weather.gov.

    Or just accept that absent NTP, w32time maybe just shouldn't try to set the clock to whatever a circus clown tells it?

    This sort of reminds me how there is a buried registry setting to tell w32time to not place the firmware clock at very high stratum in the time sources list which also is just a giant WTF in how those two behaviors were decided on.

  • by nneonneo on 8/16/23, 8:14 PM

    I wonder if this feature might be “infectious”: if a handful of servers wind up with the wrong time, and start propagating that wrong time in their server handshakes, it might cause a cascade of other servers to reset their times too.

    In any case, it seems like a poor solution especially given that random timestamps can occasionally look valid (they’re random!).

    The problem they’re trying to solve seems to be to obtain a trustworthy timestamp in the presence of a potentially hostile network - one that could be presenting expired, cracked certificates for every connection to arbitrarily tamper with all SSL connections. It’s not a trivial problem, for sure, but resetting perfectly valid clocks using this algorithm seems like a strict downgrade….

  • by przemeq on 8/16/23, 7:09 PM

    Wow, we encountered a similar problem on one of our servers. We couldn't determine the cause. We set up an alarm to notify us when such a change occurs, and we've developed a procedure to handle the consequences. Great discovery!

  • by gorkish on 8/16/23, 8:27 PM

    God knows how windows machines keep time these days, except to say that they basically don't. WSL2 has not ever been able to keep time consistent between the host and VM. You can find the issue threads on gitlab going back years at this point.

  • by spartanatreyu on 8/16/23, 11:47 PM

    A bit of a tangent, but this time-checking heuristic reminds me of the game "Halo 3: ODST".

    The story takes place in a future African megacity before, during, and after an alien invasion. Part of the game's story is shown from an AI's perspective who controls the city's operations trying to evacuate the Human survivors.

    Whenever the story is shown through the AI's perspective, you can see an overlay of the AI's working memory including which suburb's cameras are being monitored, the emotional state of the AI, and the current running timestamp.

    Some of the aliens end up withdrawing from within the city by making an FTL jump which causes extreme localised time dilation.

    Immediately afterwards, the timestamp in the AI's working memory is frozen because it's not sure what the exact timestamp is anymore, each system it connects to gives it a slightly different time depending on how close it was to the FTL event.

    It guesses a likely time and uses that until sometime later it is able to reconnect with other unaffected human networks and resumes the correct timestamps.

    This time heuristic was shown in the game, but also makes an appearance of sorts in the announcement trailer (seen in the bottom right): https://www.youtube.com/watch?v=WroxHMo6B_k

  • by mlichvar on 8/16/23, 7:27 PM

    A better solution to secure bootstrapping of time would be NTP+NTS (RFC 8915) using self-signed certificates with unlimited time validity, which can be preloaded with the OS and updated via normal OS updates if the server key is compromised. They would probably need to run their own servers. There are some public NTS servers (e.g. Cloudflare and Netnod), but I have not seen any using long-term certificates specifically for this use case.

  • by nicolaslem on 8/16/23, 6:58 PM

    I sometimes experience a similar issue with my linux laptop where time jumps to the year 2077 when waking up from sleep. My guess is that it is a hardware glitch as it doesn't happen often, but when it does it is quite impactful.

    One of the annoying consequences is that some parts of the system decide to clean up "old" data. Surely data that has been stale for 50 years can be deleted, right? I cannot imagine the impact of something similar happening on a production server.

  • by zokier on 8/16/23, 7:30 PM

    While this Windows feature does sound quite bonkers as described, it is also baffling to me that the timekeeping on computers is such a mess; would it really be that difficult to have my multithousand dollar computer keep time at least as well as a dollar-store quartz watch? Have time already set in factory, and be correct to within few hours at least; enough to do networking and more accurate time syncing safely.

  • by toast0 on 8/16/23, 6:51 PM

    TL:DR, the w32time service will sometimes try to bootstrap the clock with data from TLS handshakes. TLS 1.0-1.2 use a random value in Client Hello/Server Hello which was originaly specified as a uint32 gmt_unix_time plus 28 random bytes. It's better [1] to fill the whole structure with 32 bytes of random data (and TLS 1.3 specifies it without reference to time). If you interpret the random data as a unix time, sometimes you're going to get weird results. It's not clear to me what servers w32time probes to get these values, either.

    I've seen other services that use https to bootstrap time in case other clocks are unavailable or suspect (or use a limited expiration certificate to authenticate!), it's a bit difficult because you have to ignore or postpone checking certificate expiration when validating the x.509 certificates, parse the http date header, and then presumably check that the date provided is within the time the certificates involved are valid.

    [1] https://datatracker.ietf.org/doc/html/draft-mathewson-no-gmt...

  • by HankB99 on 8/16/23, 9:21 PM

    I always thought that it was a bad decision that Windows tracked time as local time. IMO they got it wrong right from the get-go. It made more sense to me to track time in an unambiguous manner(*) like seconds since midnight January 1. 1970 GMT and then convert to whatever representation is convenient locally.

    (*) I suppose this does not work as well on Mars. I suspect that the time on the various extra-planetary vehicles is tied to earth time but would not work so well if Mars were ever colonized.

  • by Terr_ on 8/16/23, 9:40 PM

    Hmm, so essentially it's sampling times from different servers it thinks are unlikely to be conspiring, and those times are not as reliable as expected.

    Since it's fun to imagine (and then find flaws in proposals) here's an idea for y'all:

    1. If the device reaches out to a dozen servers at different usually-trustable domains (e.g. microsoft.com, nist.gov) and they _all_ have "expired" certificates, that becomes a sign that My Local Time Is Unreliable. (Or that the machine is being booted up by an attacker who has connected it to a fake micro-internet.)

    2. The Windows device contains a cert or public key which does not auto-expire based on clock-time, but is only replaced as a side-effect of OS patches. This cert/key exists for one specific purpose, allowing it to securely get a time from an MS time fallback server. Such a server would not normally be queried, and can have very relaxed performance/accuracy/uptime requirements. It could literally just be a GET call returning a epoch integer.

    3. This way the client system can bootstrap up to something close-enough-to-current that the regular NTP process can be completed for a more-precise time.

    Obviously this can fail if the attacker injects a fake cert/key into the machine... but at that point isn't it Game Over anyway? The same attacker could just put in their own certificate authority for any service.

    It may not work for computers that are left unplugged and unpatched for a few years, but it sounds like the real issue are live servers, or ones that just suffered some power-supply hiccup.

  • by newman314 on 8/16/23, 9:04 PM

    PowerShell invocation to disable this:

    Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\W32Time\Config" -Name "UtilizeSslTimeData" -Type DWord -Value 0

    Reload using:

    W32tm.exe /config /update

  • by smallstepforman on 8/16/23, 8:17 PM

    I’ve used Linux on an embedded system which gets time/date via proprietary monitoring protocol. Seting Linux time/date isn’t instant, Linux will run an accelerated clock until it reaches the target time. This bizzare behaviour is a hack to fix apps/services which have too high delta time which cause other problems. Its frustrating for us since when we receive a clock update, our clocks can take hours to sync. Meanwhile our packet clocks are inaccurate. This only happens during testing (by gov regulators) and to pass the tests, we had to outsmart the way the system clock works when receiving a large date/time diff.

    All of these work-arounds for unneeded hacks (like in the article) drives my temper way up. Just do the simple thing in kernel space or the OS level. No suprises.

  • by bbarnett on 8/16/23, 10:12 PM

    “You may ask—why doesn’t the device ask the nearest time server for the current time over the network?” Microsoft engineers wrote. “Since the device is not in a state to communicate securely over the network, it cannot obtain time securely over the network as well, unless you choose to ignore network security or at least punch some holes into it by making exceptions.”

    Let me guess, time isn't set, so dnssec is broken, so ntp servers won't reolve, so you can't set the time?

    So lame, ntp servers shouldn't be using dnssec.

  • by xg15 on 8/16/23, 7:49 PM

    Why not use all that STS cleverness to get a "rough" estimate of the current time, then use that estimate to securely connect to an NTP server and get the actual time?

  • by Spooky23 on 8/16/23, 11:25 PM

    It makes me feel better, at least I’m not alone in getting garbage support from them.

    Our TAM is like a funeral director. Always expressing his deep sorrow.

  • by justinsaccount on 8/17/23, 12:31 AM

    > Microsoft recommending STS be turned off when the server receives reliable timekeeping through the Network Time Protocol.

    Why then are the first lines of the STS code not something like

        // server receives reliable timekeeping through the Network Time Protocol.
    if (ntp_enabled() && ntp_working())
        return 0;

  • by dullcrisp on 8/16/23, 11:22 PM

    Does anyone discuss the hypothesis that it’s a statistical bug in the implementation and that there’s some small but non-negligible probability that the servers see a pattern of random timestamps that convinces them of a false time? Because the would be my naive guess as to the cause.

  • by nightowl_games on 8/17/23, 12:07 AM

    How is a basic web application supposed to treat client time? In some of my systems, the server responds with its timestamp, the client records it's delta between itself and the server, and we use that delta as an offset into stuff like auth token refresh. My main PC has the incorrect timezone in its bios, so it's always 6hr in the future when I first log in..it's interesting to see what breaks. Rocket League for instance doesnt work at all. Starcraft 2 works fine. Windows was unable to sync itself in this situation for months, but has fixed itself lately so it can sync to the right time after a few minutes

  • by drcongo on 8/16/23, 7:05 PM

    Blows my mind that anyone would use Windows on a server.

  • by oskarw85 on 8/17/23, 3:27 AM

    In my opinion writing that Microsoft fucked up where clearly other software puts bogus data in timestamp fields is laughable. Shit goes in - shit goes out.

    I think MS should improve implementation in two ways: - ask known trusted sources (like their own servers) for SSL timestamp - use that timestamp only internally (as opposed to system-wide) to acquire valid NTP timestamp

  • by hospitalJail on 8/16/23, 8:53 PM

    The point that Windows became less reliable than Linux Desktop happened last month.

    Windows 7 is still more reliable, but with the updates being over, people are forced to upgrade.

    But Windows 11 is less reliable than popular Linux Desktops. There might be an exception if you have NVIDIA, but I'd say less than a few percent of people have a video card.

  • by fnordpiglet on 8/17/23, 1:56 AM

    I honestly struggle to recall a time when there was a windows feature that wasn’t :

    * broken on arrival and never really fixed

    * negligently implemented that causes serious harm

    * a dark pattern that no one actually wants but improves the windows margins through covert means (data harvesting, embedded adware, etc)

    I’m sure they exist, but the prevalence of the badness is so striking.

  • by stefan_ on 8/16/23, 7:15 PM

    Funny, they made a bad copy of tlsdate and sold it as a super new innovation.

  • by cryptonector on 8/16/23, 11:07 PM

    What on Earth!

    Look, you can use (with some care) time from a Kerberos KDC for the domain/realm to which the host is joined if you like, but you can't use the time from random application servers!

    What were they thinking?!

  • by binkHN on 8/16/23, 6:48 PM

    The article doesn’t talk about a fix—scary. I know OpenBSD uses a similar SSL-based method for time keeping on startup, but I’ve never experienced an issue like this.