by jaffee on 8/6/18, 9:21 PM with 39 comments
by kodablah on 8/6/18, 10:48 PM
Also, devs should be aware of some of the concurrency limitations of Sqlite. Devs need to essentially share a single connection across the app, and serialize the writes.
Also, side note, I've found one really neat use case for Sqlite over other DBs: simple disk encryption. I wrote a Go wrapper for one of the popular ones [0], but in general if you distribute your app in other ways you may want to offer your users encryption that you can't easily with other DBs. Sure you lose some performance, but sometimes it's worth it (again, less in the server case where you control it and/or can do fs-level encryption, but definitely in the distributed app case).
by pavlov on 8/7/18, 8:28 AM
I've been following this principle for a decade, and 98% of the services I've built have never needed to grow beyond one instance. The ones that did were solved by breaking out specific functions into their own managed services, not by applying a wide horizontal scaling solution.
If you build for scaling before you have anything running that can be tested against, it's highly likely that you're actually scaling the wrong thing. It's like optimisation: to get any meaningful benefit, you'll want to surgically solve the actual bottleneck rather than sprinkle hope-based gestures all over the place. (I'm reminded of the article that was posted the other day, where the writer experienced a Firebase slowdown and tried to solve it by upgrading hundreds of npm packages to their latest versions. That's what I mean by hope-based gestures. Premature "web scale" architectural tripping is essentially the same kind of shotgun approach.)
by pjc50 on 8/7/18, 9:39 AM
Two big pieces of software I worked on used this philosophy. One was https://en.wikipedia.org/wiki/Zeus_Web_Server - nowadays its niche is occupied by nginix, but back then its advantage was being a one-process-per-core select()-based solution, compared to Apache's process-per-concurrent-request solution. It was also absurdly, unnecessarily portable, being written in vanilla POSIX. Ran on a dozen different flavours and architectures of UNIX almost all of which are now dead.
Another was chip design software for a startup. Eventually it would have been useful to parallelise computation, but in the meantime it was so much faster to develop it and just buy big machines and leave it running overnight.
See also https://adamdrake.com/command-line-tools-can-be-235x-faster-...
by interfixus on 8/7/18, 5:16 AM
Developers of would-be federated services to decentralize the web, please take note. This is how you go about ensuring wide deployment. Simplicity, if anyone remembers the word.
by oavdeev on 8/6/18, 10:47 PM
To the larger point, while I totally agree with the premise -- most apps will never need to scale beyond one large instance, I'm not exactly sure what's the actual tradeoff is. If you're writing a simple CRUD app, it is not really controversial that it shouldn't need any complex distributed stuff anyway, it is just a simple python/Go app and a data store.
Most "fancy" things outside that single process paradigm, such as avoiding using local disk and using S3/RDS/document stores, using containers for deployment, etc. usually have more to do with making the app operationally simpler and easy to recover from instance failures than scaling per se.
by erdaniels on 8/7/18, 8:56 AM
It could be worth noting the thread concurrency limit imposed by cgo calls locking the OS thread to the calling goroutine. For example, 128 concurrent requests running long statements would take up 128 OS threads.
Of course any language not using a lightweight threading model has this issue either way if each request is mapped to one thread but it seems worth it to point out that you may want to limit your own concurrency at the request level or with properly sizing a connection pool (like the one present in the sqlite driver).
Edit: whatever the solution to concurrency is, benchmarking it is a good idea for any sized product going into a production environment. There's a good article about this by the people at Cockroach labs: https://www.cockroachlabs.com/blog/the-cost-and-complexity-o... (see Cgoroutines != Goroutines)
by hathawsh on 8/6/18, 10:26 PM
- No need to link with C code. There's a pure go driver for Postgres.
- You get the 'psql' command to inspect and change the database, even while the web app is running.
- You get a lot more database features.
- You can keep the deployment simple by running Postgres on the web app server until you need to scale up.
- Conversion to a full blown cloud database and horizontal scaling is just a matter of configuration.
I understand the desire to keep the stack as simple as possible, but it looks to me like Postgres would actually be simpler than SQLite for the server side.
by chj on 8/7/18, 1:15 AM
Some may say, well, sql servers are not that hard to learn. Well, that is because you live with it day and night. Try going back to it after 1 year of break.
by cabaalis on 8/7/18, 1:11 AM
This seems to be the main push of the article, and I for one agree with that statement.
by justinholmes on 8/6/18, 10:07 PM
Bitbucket pipelines that create an image of the app, create instance template with a container "gcloud beta compute instance-templates create-with-container", create instance group, create DNS, global load balancer, SSL certs and rolling update of the instance groups with a new version (if applicable).
Using preemptible g1-small to keep costs down but deployed in at least two zones per region.
by bvinc on 8/7/18, 5:59 AM
This always seems to happen when I write something.
by mikhailfranco on 8/8/18, 8:58 AM
Single process performance and the COST principle, from the iconoclastic (some might say bombastic) McSherry:
[1] blog http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
[2] pdf https://www.usenix.org/system/files/conference/hotos15/hotos...
by TooBrokeToBeg on 8/7/18, 11:20 PM
by gf263 on 8/7/18, 7:25 PM