by Isofarro on 9/27/19, 6:56 PM with 40 comments
by jedberg on 9/27/19, 8:30 PM
Gotta be super careful with this one. We did this at reddit and it bit us bad. The problem was as soon as the load on a machine went down it got pounded with new requests and the load shot up, but it takes a few seconds for the load number to react to all the new requests. So we saw really bad see-saw affect.
We had to add extra logic to mark how long a machine had beed at a certain load and also randomly send requests to slightly more loaded machines to keep things even.
The moral of the story here is make sure you pick a metric that reacts to the change in request rate as quickly as your request rate changes!
by tpmx on 9/27/19, 9:20 PM
I managed a team that built a 5x 1000 node distributed setup 10+ years ago.
We ended up going with
a) short DNS TTL + a custom DNS server that sent people to the closest cluster (with some intra-communication to avoid sending people to broken clusters)
b) in each cluster; three layers: 1) Linux keepalived load balancing, 2) Our custom HTTP/TLS-level loadbalancers (~20 nodes per DC), 3) our application (~1000 nodes per DC)
A typical node had 24 (4x6) CPU cores when we started and 48 (4x12) towards the end.
These were not GC/AWS nodes, we were buying hardware directly from IBM/HP/Dell/AMD/Intel/SuperMicro and flying our own people out to mount them in DCs that we hired. Intel gave us some insane rebates when they were're recovering from the AMD dominance.
Load-balancing policy: we just randomized targets, but kept sticky sessions. Nodes were stateless, except for shared app properties - we built a separate globally/dc-aware distributed key-value store - that was a whole new thing 12 years ago we built based on the vague concept of AWS Dynamo. App nodes reported for duty to the load balancers when they were healthy.
We had a static country-to-preferred-DC mapping. That worked fine at this scale.
This setup worked fine for a decade and 250M+ MAUs. We had excellent availability.
At some point like 10 years ago a kinda well known US-based board member really, really wanted to us to move to AWS. So we did the cost calculations and realized it would cost like 8X more to host the service on AWS. That shut him up.
Different times. It's so much easier now with AWS/GC to build large-scale services. But also so much more expensive - still! I wonder how long that can last until the concept of dealing with computation, network and storage really becomes a commodity.
by jedberg on 9/27/19, 8:36 PM
by twotwotwo on 9/27/19, 8:31 PM
True your 99th percentile slowest requests won't hit the cache, and certainly that caching won't solve all your scaling difficulties.
However, keeping requests for commonly-needed data away from (say) a DB cluster decreases the load on it at a given level of throughput, and that can be good for P99, and (as the post notes) caching can specifically help with super-hot data which can cause problematic hotspots in some sharding strategies.
Obviously situations vary and there're limits, but a cache seems like a legit tool, not just a band-aid, for a decent number of situations.
by plandis on 9/27/19, 8:24 PM
by oweiler on 9/27/19, 7:11 PM
by gfodor on 9/27/19, 7:36 PM
by speedplane on 9/28/19, 8:21 AM
by prostodata on 9/28/19, 10:23 AM
It seems that in both cases the idea is to distribute (supposedly independent) requests between workers and one of the main difficulties is that requests might not be independent either within one stream (say, in the case of sessions) or between different streams (say, if they need to use one common state).