by bittersweet on 11/21/12, 3:36 PM with 101 comments
by whalesalad on 11/21/12, 4:45 PM
I call BS. It's either a smoke screen for some other issue, or their app was built back asswards. Either way, this post and the circumstances are indicative that Amy and/or the Charm leadership has no idea what they are doing.
And they were not even launched? Beta or something? The comment about opening to the general public implies that they weren't getting that much usage. These days it's hard to build something with a modern framework and decent hardware that does not at least work for a closed-beta period. Add 100k to the mix and honestly what the hell.
Amy/Tom run Freckle and do some other cool shit so why are they dropping the ball on this? It's like getting a paper cut on your finger and going home for the day because it's just too much to bear. Cry me a fucking river.
Why does this keep happening?! People are throwing cash into the dumbest shit these days.
by jtdowney on 11/21/12, 4:27 PM
You probably noticed Charm had some nasty downtime a couple weeks ago.
Service quality is very important to us. If we didn't think we could do better, we wouldn't do it at all.
We've spent very generously on sysadmin services and infrastructure (nearly $100k of investment on sysadmin services/infrastructure alone). We hired the best possible, and we splurged on a redundant, powerful, and expensive server configuration from the beginning.
Now we've discovered that there's some kind of base incompatibility with Ubuntu, which is giving us kernel panics which nobody can track down. Charm has been plagued by mystery technical problems from the beginning, when we had to backport from Rails 3.x to 2.x because of massive performance slowdowns which even Rails Core members couldn't identify.
What this has really shown us is that, if we open Charm to the general public, we won't be able to provide you with the kind of service you deserve. We are a tiny team, and so far, we've had zero luck in our attempts to grow by hiring developers. Problems which are small now will only get bigger.
There are a lot of things I'm willing to take risks with, but not with your ability to provide support to customers for your business.
And so it is with a very heavy heart that we will cease operating Charm from Dec 15, for the foreseeable future.
You won't be billed again, and we'll refund your last payments.
We will gladly help you migrate your data out of Charm. Please contact us directly (support@charmhq.com) for help.
Thank you so much for taking a chance on us, and sharing our dream for a superior email interface.
I'm truly sorry to disappoint you.
Best wishes,
Amy
by sudhirj on 11/21/12, 4:55 PM
1) Bought a ton of Heroku dynos with every single addon enabled. That would have given them any version of Rails they wanted with Ruby 1.8, 1.9 or even 2.0; Postgres, MySQL, Redis, MongoDB, two different kinds of Memcache and three different kinds of asynchronous processing.
2) Gotten fully managed servers at Rackspace with pro support for pretty much any setup they wanted.
How is this even possible? The same application happens to hit bugs in the Ubuntu kernel and Rails core that haven't been fixed all the way to 3.2.9? And they have 100k to spend but couldn't be bothered to try different kernels and distros? Really? Have they seen the AMI launch screen on EC2?
by cheald on 11/21/12, 4:47 PM
Lots and lots of people run Rails apps on Ubuntu. Many more run them on non-Ubuntu distros. There is not an epidemic of "welp, Rails magically crashes the kernel, time to pack up and go home", and citing that as the reason to close down after spending over $100k on infrastructure smells really, really funny.
"A poor workman blames his tools."
by dazzawazza on 11/21/12, 4:01 PM
$100,000 and no one said "hey what about running it on <insert other os>"?
by davidw on 11/21/12, 4:08 PM
Maybe if they were a bit more familiar with Linux they would have done better, or at least have saved some money figuring out what went wrong:
by facorreia on 11/21/12, 3:51 PM
Anyway, kudos for not submitting their users to a service they're not confident they would be able to provide with adequate quality and better luck next time!
by ivix on 11/21/12, 4:17 PM
Blaming this on Linux or Rails is ridiculous. And as for 'mystery technical problems'...
I think the reality is more like they got in over their head, decided it was all a bit too difficult, and gave up.
by electic on 11/21/12, 4:35 PM
Probably a bad idea to advertise you can't keep a simple webserver up. Probably won't get funded ever.
by blrgeek on 11/21/12, 5:27 PM
1. Open Source OS
2. Open Source web server
3. Open Source programming language
4. Open Source web framework
5. Some (relatively) simple customer support ticketing app [not likely to be a system pounding Gorilla]
And they claim that they've not been able to root-cause a Kernel panic and a framework slowdown?
And they've spent $100K on that with no results to show for it?
That sounds really strange to me.
[I've debugged Linux kernel panics on custom stacks, with no disk/logs, only console, 32MB of RAM, PPC cross-compiled - and never run into a dead-end like this]
by pixelbeat on 11/21/12, 3:48 PM
by axlerunner on 11/21/12, 4:39 PM
by nateberkopec on 11/21/12, 3:57 PM
100k for "infrastructure alone" seems like a massive investment for a startup. How far along were they?
by rlpb on 11/21/12, 4:01 PM
by twp on 11/21/12, 4:40 PM
Googling for "rails charmhq" doesn't reveal much, just that one of the Rails contractors was a company called Queve: http://www.qurve.com/clients/charm/
by raverbashing on 11/21/12, 4:18 PM
You can open a bug with distros and try to work around it. It is usually doable
"unfixable kernel panics" don't seem something a person familiar with Linux would say. Also, there are several distros, kernel versions, and Ubuntu isn't my first choice for a server.
And the gist is over the rate already.
by runjake on 11/21/12, 5:04 PM
by zdw on 11/21/12, 4:08 PM
It's better to go conservative on your infrastructure, get it stable, then experiment if required.
by jondot on 11/21/12, 4:51 PM
I'd love to see an Ask HN with _proper_ technical details. Who knows, maybe they'll get better ways to solve this rather than close shop.
by citricsquid on 11/21/12, 4:28 PM
Found this so far: http://charmhq.com/Charm_Bootcamp.pdf
by northisup on 11/21/12, 4:33 PM
by jbigelow76 on 11/21/12, 3:40 PM
by dantiberian on 11/22/12, 8:03 AM
by krsgoss on 11/21/12, 4:41 PM
by cpeterso on 11/21/12, 4:06 PM
by krobertson on 11/21/12, 4:07 PM
by sdrgalvis on 11/21/12, 3:48 PM
by ahoyhere on 11/22/12, 6:00 AM
The email isn't "Wah we hit a server problem, bye bye."
The point is:
"What this has really shown us is that, if we open Charm to the general public, we won't be able to provide you with the kind of service you deserve. We are a tiny team, and so far, we've had zero luck in our attempts to grow by hiring developers. Problems which are small now will only get bigger."
If you've never run a serious product, or a real business, or tried to hire for technical positions, I can understand why you'd zero in on the "facts" about the technical situation and ignore all the "foofy personal window dressing," and write things like "I call BS! It's a smoke screen!" or "Why didn't they just try BSD?"
And yet I addressed the actual problem in very clear terms in a paragraph you can't possibly miss.
Next time somebody makes a hard business decision and you hear about it on HN and come out, irony guns blazing, may I humbly suggest you read more than the subject line written by the unrelated HN submitter?
As for any poor silent, lurkers who wonder if this is how they will be treated if they -- gasp! -- ever find themselves in over their heads, or in a business they realize they don't actually want to be in… our customer reactions have been uniformly:
"Aw, I'm so sorry... Charm is such a nice piece of software... your email was so touching."
Why? Because we've always shown our customers respect by creating great software, and we're showing them even more respect by ensuring we do not make promises we can't keep.
Our friends and technical acquaintances have been full of nothing but sympathy, understanding, and for those closest to the situation, praise for making the right, hard decision.
Yep, it sucked. Yep, we poured something like $200k into development, redevelopment, and infrastructure all told. Yep, it is a fucking amazing piece of software and the best thing I've ever designed.
But is it worth the constant heartache of the impossible task of finding people equipped to work on it? Of having it stuck in some kind of product half-life because of that? Of feeling responsible for, but incapable of, being "on call" in the middle of the night?
Of feeling guilty because, unless we can somehow suddenly be great at those things, we're taking money for a service which might let our customers down?
Nope. It's not worth it.
And boy do I feel lucky and privileged that because we spent nothing but our own money on it, we are free to decide to do whatever we think is right.
See also my principles 10 and 11: http://unicornfree.com/2011/lessons-learned-from-16-years-of...