by r1ch on 11/22/23, 3:25 AM
We shipped a shader cache in the latest release of OBS and quickly had reports come in that the cached data was invalid. After investigating, the cache files were the correct size on disk but the contents were all zero. On a journaled file system this seems like it should be impossible, so the current guess is that some users have SSDs that are ignoring flushes and experience data corruption on crash / power loss.
by ricardobeat on 11/22/23, 2:39 AM
Misleading headline since after testing eight more drives, none more failed.
2/12 is not nearly as dramatic as “half”, and the ones that lost data are the cheap brands as one would expect.
by IngvarLynn on 11/22/23, 2:21 AM
There is a flood of fake SSDs currently, mostly big brands. I've recently purchased counterfeit 1TB. It passes all the tests, performance is ok, it works... except it gets episodes where ioping would be anything between 0.7 ms and 15 seconds, that is under zero load. And these are quality fakes from a physical appearance perspective. The only way I could tell mine was fake is that the official Kingston firmware update tool would not recognize this drive.
by kristopolous on 11/22/23, 9:34 AM
Under long term heavy duty, I've routinely seen cheap modern platter outperform cheap brand name NVME.
There's some cost cutting somewhere. The NVMEs can't seem to sustain throughput.
It's been pretty disappointing to move I/O bound workloads over and not see notable improvements. The magnitude of data I'm talking about is 500-~3000GB
I've only got two NVME machines for what I'm doing so I'll gladly accept that it's coincidentally flaky bus hardware on two machines, but I haven't been impressed except for the first few seconds.
I know Everyone says otherwise which is why I brought it up. Someone tell me why I'm crazy
Edit: no, I'm not crazy. https://htwingnut.com/2022/03/06/review-leven-2tb-2-5-sata-s... this is similar to what I'm seeing with Crucial and Adata hardware, almost binary performance
by kmxdm on 11/22/23, 12:06 PM
Writes are completed to the host when they land on the SSD controller, not when written to Flash. The SSD controller has to accumulate enough data to fill its write unit to Flash (the absolute minimum would be a Flash page, typically 16kB). If it waited for the write to Flash to send a completion, the latency would be unbearable. If it wrote every write to Flash as quickly as possible, it could waste much of the drive's capacity padding Flash pages. If a host tried to flush after every write to force the latter behavior, it would end up with the same problem. Non-consumer drives solve the problem with back-up capacitance. Consumer drives do not have this. Also, if the author repeated this test 10 or 100 times on each drive, I suspect that he would uncover a failure rate for each consumer drive. It's a game of chance.
by loloquwowndueo on 11/22/23, 1:03 AM
Twitter yuk, can somebody just post the names of the four tested drives and which passed/failed please?
by CoastalCoder on 11/22/23, 1:46 AM
Does advertising a product as adhering to some standard, but secretly knowing that it doesn't 100%, count as e.g. fraud? I.e., is there any established case law on the matter?
I'm thinking of this example, but also more generally USB devices, Bluetooth devices, etc.
by handedness on 11/22/23, 1:25 AM
by sashk on 11/22/23, 1:05 AM
This is (2022).
Wondering if anything changed since the original tests...
by arglebargle123 on 11/22/23, 11:09 AM
Meanwhile I'm over here jamming Micron 7450 pros into my work laptop for better sync write performance.
I have very little trust in consumer flash these days after seeing the firmware shortcuts and stealth hardware replacements manufacturers resort to to cut costs.
by jauntywundrkind on 11/22/23, 1:22 AM
Losing flushes is obviously bad.
I wonder how much perf is on the table in various scenarios when we can give up needing to flush. If you know the drive has some resilience, say, 0.5s of time it can safely writeback during, maybe you can give up flushes (in some cases). How much faster is the app then?
It's be neat to see some low-cost improvements here. Obviously in most cases, just get an enterprise drive with supercapa or batteries onboard. But an ATX power rail that has extra resilience from the supply, or an add-in/pass-through 6-pin sata power supercap... that could be useful too.
by lxgr on 11/21/23, 11:30 PM
I guess it's time for `fsync_but_really_actually_sync_it_please(2)` (and the lower level equivalents in SATA, NVMe etc.)?
by tripdout on 11/22/23, 4:00 AM
Flushing in this case is from the SSDs internal DRAM cache to the actual NAND flash?
by nijave on 11/22/23, 4:01 PM
It'd be nice if there were a database of known bad/known good hardware to reference. I know there's been some spreadsheets and special purpose like the USB-C cables Benson Leung tested.
Especially for consumer hardware on Linux--there's a lot of stuff that "works" but is not necessarily stable long term or that required a lot of hacking on the kernel side to work around issues
by AlbertoGP on 11/21/23, 10:32 PM
Well, yes, but which were those 2 out of 4 vendors?
by caycep on 11/22/23, 5:52 AM
The model I’d be interested in would be the SK Hynix/Solidigm P44 Pro, as that model competes w the Samsung 9xx evo and pro models
by dtx1 on 11/22/23, 1:23 PM
I am a bit annoyed that everyone here takes this at face value. There's 0 evidence given, not even the vendors and models are named to confirm this.
On a related note I tested 4 DDR5 Ram kits from major vendors - half of them corrupt data when exposed to UV light.
by 39 on 11/22/23, 12:17 PM
This has always been the case? At least it was a course learning when we wrote our own device drivers for minux, even the controllers on spinning metal fib about flush.
by naasking on 11/22/23, 2:37 PM
At this point, any storage vendor should be required to pass the Sqlite test suite before they can sell their product.
by caycep on 11/22/23, 5:52 AM
Also…would modern journaling file systems protect against this sort of data loss?
by pleoxy on 11/22/23, 3:46 AM
If you need PLP use an enterprise drive. That's what they're for.
by Joel_Mckay on 11/22/23, 8:32 AM
Cheap drives don't include large dram caches, lack fast SLC areas, and leave off super-capacitors that allow chips to drain buffers during a power-failure.
"Buy cheap, buy twice" as they say... =)
by pajko on 11/22/23, 3:22 PM
Without any more information this post is just bullshit. For example, it's not documented how the flushing has been done. On Linux, even issuing 'sync' is not enough:
https://unix.stackexchange.com/questions/98568/difference-be...The bottom answer especially states that "blockdev --flushbufs may still be required if there is a large write cache and you're disconnecting the device immediately after"
The hpdarm utility has a parameter for syncing and flushing device buffers themselves. Seems like all three should be done for a complete flush at all levels.
by nik736 on 11/22/23, 10:59 AM
That's what PLP is for.
by tw1984 on 11/22/23, 8:50 AM
Don't use home-grade SSDs for storing anything that is considered critical.
The rule is not that hard to remember.
by martincmartin on 11/22/23, 12:43 PM
2022
by jbverschoor on 11/22/23, 6:46 AM
Brands please. It’s time they have some pressure to fix these data corruption issues
by xbmcuser on 11/22/23, 11:32 AM
this is from Feb 2022
by java-man on 11/21/23, 10:32 PM
Name the offenders please.
I am sure it might be easy to see visually - a lack of substantial capacitor on the board would indicate a high likelihood of data loss.
by babberman on 11/22/23, 1:04 AM
That is unfortunate, but I guess those SSDs performed really well and outclassed all others in performance benchmarks? lol
by hackerfactor1 on 11/22/23, 2:13 AM
The posting is from Feb 2022, nearly 2 years ago.
How is this suddenly trending on Hacker News?