from Hacker News

Google Drive does a surprise rollout of file limits, locking out some users

by stereoradonc on 4/1/23, 4:16 AM with 306 comments

  • by chrisbolt on 4/1/23, 4:27 AM

  • by mcherm on 4/1/23, 12:40 PM

    Having a limit of 5 million files is perfectly reasonable. Failing to document that such a limit exists and refusing to publicly confirm it (which apparently is STILL the case) is extraordinarily poor customer service/communication.

    Google KEEPS setting new records for poor customer communication, to the point where I (and much of the HN crowd) now expect it. Android developer banned from the app store? There is no meaningful way to appeal but you'll probably never be able to find out why. Your best hope is to post on HN and hope someone with power at Google notices.

    Leadership at Google ought to recognize this; they ought to make an effort to improve the channels by which "customers" can communicate with Google. But I see no signs that they are even aware of the issue; I see no effort to change anything.

    I would try to tell them but... there's no communication channel. Maybe I should post about it on HN.

  • by bigiain on 4/1/23, 4:55 AM

    Ha!

    “ a safeguard to prevent misuse of our system in a way that might impact the stability and safety of the system."

    Google: We have identified modern web development as a threat to our systems, and have taken measures to ensure npm users cannot store their npm_modules directories on GoogleDrive. Please consider rewriting your nodejs projects in Go.

  • by mort96 on 4/1/23, 10:40 AM

    Hmm, there was a HN thread about this a few days ago [1] where everyone seemed to attack people for even considering the idea of storing 5M files in a cloud storage solution, going so far as to argue that even disclosing such a limit would be unreasonable to expect.

    In this thread, the prevailing thought seems to be that having a 5M file limit is unreasonable and adding it without disclosing it is egregious.

    Just a curious thing I noticed.

    [1]: https://news.ycombinator.com/item?id=35329135

  • by throwaway_ab on 4/1/23, 5:52 AM

    I pay for 5 TB and planned to use the drive to store a copy of my data.

    Things I store that have lots of files:

    - The frames for my Timelapse videos = 400,000 files

    - The files in my Eagle app photo database = 400,000 files

    - Other image files, my programming repositories, documents, music, stable diffusion Deforum frames = 400,000 files

    80% of these files I've accumulated in the last 12 months and can see myself easily hitting this 5,000,000 file limit well before I run out of TB's

    So now I know I will never be able to use all the space I'm paying for, I'm going to stop uploading my files and instead search for a proper backup service, something I should of researched in the first place.

    Anyone here have any recommendations for a backup service?

  • by fencepost on 4/1/23, 6:20 AM

    If the number of users affected is as 'vanishingly small' as a Google spokesman indicated then you'd think they'd be able to contact them - at least the paying customers?

  • by hedora on 4/1/23, 5:48 AM

    “In practice, the number of impacted users here is vanishingly small."

    Well, yeah, I imagine they’re moving elsewhere.

    Seriously though, do people actually trust them not to randomly intentionally break stuff at this point?

  • by rsync on 4/1/23, 5:42 PM

    I see it speculated, downthread, that this is a response to modern web-dev and node (?) creating millions of files, etc.

    I can’t comment on that but I do know that modern, encrypted, archive tools such as duplicity and borg and restic “chunk” your source files into thousands (potentially millions) of little files.

    We see tens of billions of files on our zpools and have “normal” customers breaking 1B … and the cause is typically duplicity or borg.

  • by greatgib on 4/1/23, 11:46 AM

    Good remainder again that "the cloud is just someone's else computer"!

    In my experience, GDrive is a piece of crap with a lot of weird behaviors and easy ways to lose your data if you sync your computer with it.

    The worse here, as said by multiple persons, is not to have a limit. A limit on their service is fair. It is that this limit is undocumented, and that their key selling point is to shout everywhere that if you pay you will have "unlimited" storage. And that it will scale more easily than using your own "not cloud" backups.

  • by exabrial on 4/1/23, 5:38 PM

    Once again: Don’t use Google for anything crucial or critical. Not Google Cloud, Google Docs, Google Drive, even Gmail is becoming a liability.

    Real Engineering involves developing forward looking designs and maintaining backwards compatibility. It involves a release schedule. It involves communication channels and releases notes. It’s hard. It’s unsexy.

    Google treats their product lineup with the seriousness of a social media platform. They don’t care about your puny business; even if it means the world to you, it means nothing to them.

  • by Beldin on 4/1/23, 9:10 AM

    "Vanishingly small": a number of users small enough to be downplayed, but large enough so that neither an individual approach to the problem would work, nor that the problems could be ignored. Suspected to be a complex number.

  • by stinos on 4/1/23, 6:58 AM

    Anyone knows how this works legally? You buy a service, suddenly without notice the services changes features. Does the small print allow for that? And how is this 'ok' in software but probably not anywhere else (pretty sure a service contract for an elevator doesn;t allow the service company to just say "we're going to limit the ?amount of times your elevator goes up and down to 100 times a day now")

  • by jmyeet on 4/1/23, 11:29 AM

    Some people will let technical limitations define a product. Others will have the product dictate the technical design. This, to me, is an example of the former.

    I don't know the serverside implementation of Google Drive but imagine the files on your Drive correspond to files on something like an ext4 filesystem. In this scenario, each file has a cost (eg in the inode table) and there is wastage depending on what your block size is. Whatever the case, Drive seems to treat files as first-class objects.

    Compare this to something like Git. In Git (and other DVCSs like Mercurial), you need to track changes between files so the base unit is not files, it's the repo itself. It maps to your local filesystem as files but Git is really tracking the repo as a whole.

    So if you were designing Google Drive, you could seamlessly detect a directory full of small files and track that directory as one "unit" if you really wanted to. That would be the way you make the product dictate the design.

  • by fsh on 4/1/23, 6:40 AM

    Very interesting that google chose to do this instead of fixing the software that caused the limitation. No wonder that their products are seen as a joke in the business world.

  • by harshaw on 4/1/23, 12:54 PM

    The challenge with running cloud storage is that you have to think around the corners for usage and shape customer behavior with pricing. Seems like google didn't want to do this or was too lazy (sorry). Millions of files will always be a problem. The metadata costs more for these users, it's impossible to manage, hard to cleanup, etc.

    The problem with google is if they fuckup their service they make it the customers problem. Other places if they fuckup, its more viewed as a one way door. You can sunset old products with (in this case, unlimited files), but you never put in a new restriction.

  • by burnished on 4/1/23, 7:10 AM

    The limit is 5 million files. The article feels a little overblown in its reaction.

  • by topicseed on 4/1/23, 6:49 AM

    We need a .googledriveignore file then.

  • by ourmandave on 4/1/23, 4:23 PM

    Initial thought is the ones who are surprised are the ones who sending a notification email to wouldn't have been noticed.

    Like my dad has 300+ unread emails with who knows how many gigs of attachments.

  • by lopkeny12ko on 4/1/23, 4:56 PM

    I wonder if you could create a block-level virtual filesystem backed by Google Drive so that you could store many small logical files in one physical remote "block" (file).

  • by Reptur on 4/1/23, 5:25 PM

    Seems like an Engineering issue more than a User issue. They could just take the node_modules folders and zip them up behind the scenes without changing the user interaction.

  • by squokko on 4/1/23, 6:41 PM

    This is why despite G Suite being in many ways a superior product, it's made almost no inroads in Corporate America vs. Microsoft Office. Enterprises need to be able to specify a business workflow and depend on it, and if there are nasty surprises it fucks with their money.

    Microsoft software is much worse than many competitors but it's documented, the behavior doesn't change suddenly, and it's backwards compatible.

  • by nickcw on 4/1/23, 7:22 AM

    Rclone users noticed this new limit back in February.

    Here is a thread discussing it on the rclone forum:

    https://forum.rclone.org/t/new-limit-unlocked-on-google-driv...

    It would be nice to have official confirmation of the limit rather than relying on speculation.

  • by SMAAART on 4/1/23, 1:19 PM

    "Don't be evil" ---> You either die a hero, or live long enough to see yourself become the villain.

  • by LightBug1 on 4/1/23, 6:46 PM

    I know Google employees are reading this ... don't you recognise this sh!tshow and communicate it internally?

    I'll never understand how such a large organisation can let this kind of stuff happen.

  • by thrdbndndn on 4/1/23, 10:10 AM

    Wait, is it new?

    I believe Google Drive for Workspace always have a file count limit, and IIRC it's as low as 500k or something, despite having "unlimited" capacity.

    To be totally fair to Google, I know this precisely because there are communities of data hoarders that actively abuse various cloud storages. In Google Drive's case, They have ways to create "free" Google Workspace accounts via registration exploitation from various institutions. People use them to store PB-level data.

    (For the interested, there are also ways to apply free MS developer accounts that are supposed to expire in 3 months but can be re-refresh indefinitely. This comes with 5TB "free" cloud storage x 5 (10?) separate sub-accounts.)

  • by SergeAx on 4/1/23, 11:09 AM

    I wonder what jury-rigged solution may lead to breaking into 5M limit? I can't believe it was just digital hoarding. In the end, hoarders know better to keep things in zip-archives.

  • by nashashmi on 4/1/23, 11:40 AM

    I wish there were a "zip" standard for storing a million small files in one package. NPM and other open source programs badly need this.

  • by Overtonwindow on 4/1/23, 10:54 AM

    Oh good lord, I have so many files spread across a dozen drive accounts. It's free storage, I'm going to use it.

  • by anothernewdude on 4/1/23, 8:14 AM

    Google: "We are too fucking useless or lazy to make file count not an issue since we already limit you by space."

  • by eviks on 4/1/23, 8:19 AM

    On a tangent and out of curiosity, which filesystems are great at working with a gazillion of tiny files?

  • by slackfan on 4/1/23, 2:42 PM

    Just another unlimited(*) with the asterisk in giant bolf and the explanation text in 2point type service.

    Yay.

  • by benhurmarcel on 4/1/23, 1:22 PM

    I wonder how that works for companies using Google Workspace. My company has Workspace users close to 6 digits I believe, I'd think we collectively store way more than a few million files.

  • by jeron on 4/1/23, 7:21 AM

    Timely reminder of this great GitHub repo:

    https://github.com/awesome-selfhosted/awesome-selfhosted

  • by sqldba on 4/1/23, 6:49 AM

    So just don’t be one of the vanishingly small paid users who this affects. Easy.

    I wonder what vanishingly small is. 0.001% of a million is still thousands.

  • by nathants on 4/1/23, 6:53 AM

    hello rclone and s3.