from Hacker News

Comparing Amazon EC2 to Joyent SmartOS

by xachen on 11/19/12, 4:52 AM with 56 comments

  • by ChuckMcM on 11/19/12, 5:58 AM

    From the article (its conclusion) :

    "Amazon need to lift their game in terms of CPU performance. They offer a great service that obviously extends far beyond a simple CPU benchmark. But when you can get the same work done in Joyent significantly faster for the comparable price, you’ll get far more mileage per instance, which is ultimately going to save the dollars."

    Or, "Wow, Amazon sure makes bank on this EC2 stuff, they are selling kilo-core-seconds for a bunch more than Joyent is and they are still the most widely used provider."

    There have been a couple of good benchmarks around which instances get you the most 'bang' for the buck, but its pretty clear that Amazon is making a ton of money here.

    A good comparison is a rack of 40 SuperMicro 1U dual socket servers with 192G of ram in them. You can create 480 dedicated instances with 16 GB of RAM each, and run the whole thing in a colo facility for $5K/month (power, networking, and floor space). That is 0.2 cents per second, or a bit more than $10/month running them 100% of the time.

    Not a bad business to be in.

  • by latchkey on 11/19/12, 7:33 AM

    I recently tried out Joyent SmartOS for a NodeJS consulting gig I'm doing. After about 30 seconds, I killed the vm and switched to Ubuntu 12 server.

    I don't care if SmartOS is 'faster' or has dtrace. The fact that it is missing basic things like top (no prstat isn't anything close to top) and 'standard' things like upstart made me want to run for the hills. The Joyent wiki documentation on SmartOS is abysmal at best. Even the pkgsrc repository system feels ancient and pales in comparison to what apt is. I have no desire to spend time re-inventing basic tools that linux has had for ages.

    The image version of SmartOS w/ NodeJS isn't up-to-date with the latest Node release and it starts up MongoDB by default. While this may be good for just getting things running quickly, what happens if I don't want Mongo running at all? Generate my own images? No thanks.

    I know, lots of hate... but it isn't all bad. Joyent as a cloud platform is great. Basic ab testing against NodeJS had me serving 9900 concurrent connections without any issues on their least expensive vm (which runs on a blazing fast network at 10gb/s). We were on Rackspace before and with a network limited to 20Mbps/s (at the lowest tier) we just couldn't get the throughput we needed.

    tl;dr: SmartOS is a train wreck. Don't waste your time. Joyent as a platform is great. Why bother with EC2 when Joyent is faster and less expensive?

  • by malkia on 11/19/12, 5:59 AM

    Sorting 5 million random numbers takes ~20secs - and this is the fastest benchmark?

    Wait there buddy.... First of all, why would OS choice be of so much significance here, after all this is all executed mainly on the CPU (not much IO). The only difference would be that the random() function might return different things on different systems (libc difference?)... or maybe not (I'm not familiar with ruby's internals).

    So I'm not sure what this measures at all...

    At least you can run the benchmark shootout (CPU), or something that can test the I/O.

  • by twotwotwo on 11/19/12, 5:48 AM

    - No one pick a host based on a single microbenchmark, please. Game out what your app would need to achieve your target performance level with what each host offers. Amazon still might lose (sounds like that happened with TripAdvisor) but at least you've made your decision a sane way.

    - For the best price, you'd want to use a reserved instance if this is a webserver, and perhaps a spot instance if it's a batch/HPC task. You'd want to use high-CPU instances if you really are all about quickly sorting 5M numbers. If your app is or might be come database-bound, DB perf's super key.

    - I'd love to hear more broadly about the practical differences between the clouds--whether anyone seems to be better at managing noisy neighbors than the next guy, network and I/O performance, etc. One CPU-bound task can't really be the end of it.

  • by ksec on 11/19/12, 11:55 AM

    I have been asking, and wondering why would anyone want to use AWS. It is insanely expensive. It is Slow on CPU, Slow on I/O, Low on Bandwidth, Low on memory, and Zero Support. Compared to many other alternatives that gives you 3 – 4 times the on all the above plus better support for the same price. Even their High CPU and Memory plan is still on the expensive side, only the High I/O SSD offers some value. AWS works great when you are in the mid range of expanding, then you can calculate your usage and book their server in advance using Reserved Instances. Instagram and Netflix as example. But that is hassle you have to put though, there are alternatives that are priced and perform just as well without using Reserved Instances. And once you got past the mid range, a managed dedicated server and some instant cloud severs should be much more cost effective.

    Apart from being forced by VC ( because they are heavily invested in Amazon ) i could not understand why startup are choosing AWS at all.

  • by moe on 11/19/12, 11:54 AM

    I don't understand why Joyent clings to a dead operating system. Are they planning to single-handedly keep solaris alive?

    Cloud computing is a commodity. Many (most?) customers use multiple clouds in parallel. Learning and maintaining the intricacies of a niche OS in this environment is a nonsensical proposition.

    And for what? For a few percentage points on a CPU benchmark? Seriously?

  • by 4ad on 11/19/12, 11:02 AM

    If you think this benchmark is impressive... just do some I/O benchmarking. It blows EC2 away much harder than it does for CPU. ZFS on local disks with SSDs for L2ARC, and no virtualization overhead (this obviously applies to SmartOS). Oh, and the performance is predictable and doesn't vary in time. And you have DTrace to understand your application.
  • by tszming on 11/19/12, 12:57 PM

    For a more detailed comparison, see: Is Joyent Really 14X Faster than EC2 and Azure the "Fastest Cloud"? (tl;dr - EC2 isn't that bad) http://blog.cloudharmony.com/2011/11/many-are-skeptical-of-c...

    To me, the weakest part of AWS is always on the technical support - you have no right to create a ticket to them unless you pay for the support fee (as contrast to Joyent /Rackspace/Linode), even you have discovered a bug in their system!

    On the other hand, I also agree with @latchkey that SmartOS /Ubuntu is an important factor when we are choosing the platform - for a lot of people, ease of use and larger community is more important than cutting edge features like dtrace and zfs.

  • by nickporter on 11/19/12, 5:49 AM

    Anybody else want to share their experience with Joyent versus EC2? I'm considering doing the switch as my EC2 bill is getting a little expensive.
  • by sonier on 11/19/12, 5:58 AM

    After seeing this, I thought I would spin up a server with Joyent, just to give it a try. After 10 minutes, the server is still not marked as "provisioning," which seems like an excessive amount of time.
  • by feniv on 11/19/12, 7:35 AM

    Running a benchmark on services like this can be really tricky.

    It really depends on which cluster you were allocated and what other applications were running at that time. If someone else was running a heavy process at the same time, then that could explain some of the sluggishness..

    Even still, the numbers are so drastically different that this question does deserve a second look. Someone from Joyent or EC2 should really put together a solid benchmark to prove once and for that their service is indeed better in a wide range of compute and networking categories.

  • by kdsudac on 11/19/12, 6:42 AM

    I haven't tried out Joyent, but I've tested out Linode and Storm-on-Demand and they easily beat Amazon EC2. My benchmark was running our production image processing algorithm that utilizes OpenCV via python on Ubuntu Server (definitely CPU-limited).

    FWIW, here are my benchmarks, along with pricing for Northern California regions:

    Linode 512: 15.2 seconds, 65 cents per day

    Storm-on-demand 1 GB: 13.05 seconds, 5 cents per hour

    EC2-micro: 168 seconds, 2.5 cents per hour

    EC2-small: 63 seconds, 9 cents per hour

    EC2-medium (high CPU): 12.13 seconds, 18.6 cents per hour

  • by adamlindsay on 11/19/12, 11:52 AM

    An AWS micro is designed for burstable CPU loads, i.e. Websites. Performing a CPU intensive task will never do well. Amazon even says this in their own documentation.

    The second fault I have is that articles like this, is that they only every look at the OnDemand price, which most people using AWS seriously don't pay. AWS has reserved pricing and spot both of which will dramatically drop the pricing. Maybe not as inexpensively as Joyent, but much much closer.

  • by ComputerGuru on 11/19/12, 7:09 AM

    For Windows servers, Joyent is prohibitively more expensive than EC2... though their Linux prices trounce AWS' significantly. I wonder why that is. They also don't offer anything smaller than "medium".

    Does anyone know a good cloud-based Windows host as an alternative to AWS?

  • by kristianp on 11/19/12, 9:32 AM

    I tried out Joyent back when they did a year's free VM promotion with Facebook. I found Solaris to be very nice, but it required a little work to get software built for linux to build on it. Different C compiler for one thing. Usually for common pieces of software, like ruby there is a guide on the Joyent website for compiling.
  • by rplnt on 11/19/12, 11:22 AM

    I never used any cloud computing but hell that's slow. Just tried the "sort five million numbers" on my desktop and it took under 5 seconds (something above 5 in VM). I tried it in python though.. but I imagine ruby's performance would be about the same in this regard.
  • by nodesocket on 11/19/12, 10:51 AM

    The `ruby` benchmark on my iMac i3 3GHZ took `14.5` seconds. Inside of a CentOS 5.3 x64 virtual machine on Softlayer with 2 cpus and 1GB of memory got `15.6` seconds. Not sure how Joyent and especially AWS performed so poorly on that benchmark.