from Hacker News

Cron in production is a double-edged sword

by neo2001 on 10/16/15, 9:49 PM with 57 comments

by vidarh on 10/17/15, 1:36 AM
Most of what he's writing about, and much more, is made substantially easier with systemd timers.
E.g. want errors to cause e-mails, but everything else to just go to logs? Use a timer to activate a service, and make systemd activate another service on failure.
Want to avoid double execution? That's the default (timers are usually used to activate another unit, as long as that unit doesn't start something that doubleforks, it won't get activated twice).
(Some) protection against thundering herd is built in: You specify the level of accuracy (default 1m), and each machine on boot will randomly select a number of seconds to offset all timers on that host with. You can set this per timer or for the entire host.
And if you're using fleet, you can use fleet to automatically re-schedule cluster-wide jobs if a machine fails.
And the journal will capture all the output and timestamp it.
systemctl list-timers will show you which timers are scheduled, when they're scheduled to run next, how long is left until then, when they ran last, how long that is ago:
```
     $ systemctl list-timers
    NEXT                         LEFT     LAST                         PASSED       UNIT                      
    Sat 2015-10-17 01:30:15 UTC  51s left Sat 2015-10-17 01:29:15 UTC  8s ago       motdgen.timer             
    Sat 2015-10-17 12:00:34 UTC  10h left Sat 2015-10-17 00:00:33 UTC  1h 28min ago rkt-gc.timer              
    Sun 2015-10-18 00:00:00 UTC  22h left Sat 2015-10-17 00:00:00 UTC  1h 29min ago logrotate.timer           
    Sun 2015-10-18 00:15:26 UTC  22h left Sat 2015-10-17 00:15:26 UTC  1h 13min ago systemd-tmpfiles-clean.timer
```
And the timer specification itself is extremely flexible. E.g. you can schedule a timer to run x seconds after a specific unit was activated, or x seconds after boot, or x seconds after the timer itself fired, or x seconds after another unit was deactivated. Or combinations.
by NightMKoder on 10/17/15, 1:16 AM
Something we've found to be fairly lightweight (compared to e.g. Chronos), but incredibly featureful is using Jenkins (the CI server) as a cron runner. We use http://docs.openstack.org/infra/jenkins-job-builder/ to configure it at deploy-time so it lives as part of the deploy rather than system config.
Here's a small list of things we're getting out of it:
- concurrent run protection (& queue management via https://wiki.jenkins-ci.org/display/JENKINS/Concurrent+Run+B... )
- load balancing (e.g. max concurrent tasks) and remote execution with jenkins slaves [sounds complicated, but really jenkins just knows how to SSH]
- job timeouts. No more hanging jobs.
- failure notifications via slack/hipchat/email/whatever. [email only on status change via https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin ]
- log/history management: rotation & compression.
- fancy scheduling: e.g. run this job once every 24h, but if it fails keep retrying in 5 minute increments (https://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin ). You could also use project dependencies for pipelines, but we've been staying away from that.
- monitoring: we use the datadog reporter & alert on time since last success. Given how mature Jenkins is, this likely translates to whatever system you're using just as well.
It's worked incredibly well for us. We migrated to Jenkins from crontabs with cronwrap (https://github.com/zomo/cronwrap). We're never going back.
by nathancahill on 10/16/15, 11:47 PM
I've been using Dead Man's Snitch[0] in production for a few years. It's been a life saver. Not affiliated, just a happy customer.
[0] https://deadmanssnitch.com/
by liquidise on 10/17/15, 3:34 PM
This is not an argument against cron. It is a demonstration of people not abstracting code. One of the thousands i've come across.
Take all of the features he mentions, and abstract the to a launch_from_cron.sh file. Make that file accept a script path as an argument and viola! All of the safety added to cron without the need for code duplication or these massive overhead solutions listed in these comments.
by josnyder on 10/17/15, 2:31 AM
I work for Yelp, and we use cron for purposes similar to those mentioned in this article, mostly synchronizing small bits of configuration or data that we want local to the machine. We're heavy Puppet users, and we made a module to assist us in the management of our crons [1]. If you're a Puppet shop, I highly recommend checking it out. It provides answers to each of the problems mentioned in the article, often using the same mechanisms. I especially like its integration with Sensu, which we use for monitoring the jobs.
We've found that deploying cronjobs onto individual hosts is quite powerful, and helps us fill a niche between configuration management tools (like Puppet) and specialized coprocesses (like Smartstack). We have cronjobs for downloading code deploys, showing Sensu state within the motd, reconfiguring daemons (especially the Smartstack ones), and (of course) cleaning up unused data.
Of course, there's also the separate problem of scheduling and coordinating tasks across an entire cluster. In most cases we don't use our cron daemons for this, although we do have some jobs that run on multiple hosts and enforce mutual exclusion by grabbing a lock in Zookeeper.
[1] https://github.com/Yelp/puppet-cron#puppet-cron
by sebw on 10/17/15, 9:22 AM
No one mentionned Rundeck: http://rundeck.org/
I've been using it for two years now. This has replaced cron on about 200 nodes.
Not only it does cron, but also helps deploying artefacts (integrated with Jenkins) through simple forms. We now have ops with 0 experience in Linux deploying code.
by zimbatm on 10/17/15, 11:03 AM
Having local mailboxes in each server is not really useful in a cloud setup with hundreds of machines. But it's not a reason to silence the output; something bad might happen and only stdout/stderr might give you an anwer of what exactly is going wrong.
Instead use https://github.com/zimbatm/logmail. It's a `sendmail` replacement that forwards everything to syslog. Then forward all your syslogs to a central place an you can capture and analyze these messages.
by aidenn0 on 10/17/15, 12:44 AM
I use Jenkins instead of cron. I get an rss feed of processes that exited with non-zero, it captures the output but doesn't e-mail it to me. This is totally not what it's designed for, but it is closer to what I want than cron is.
by INTPenis on 10/17/15, 6:40 AM
The problem isn't cron, cron is just a dumb execution tool.
The problem is that we don't have any way of alerting our monitoring systems from a cron job.
This is exactly what I've been implementing, a simple curl API call to our monitoring system when a cron job has run is all that we need. This puts the monitoring of cron into the same sphere as all other monitoring and puts the alert on a webpage where it can be found eventually by our 2nd line or our on-call personnel, instead of in someones mailbox.
Edit: And you don't need a fancy REST based API for your monitoring system to do this, ye ol' nagios agent could do it with some hacks.
The hard part is having the discipline to fix all your cron jobs in this way, but adding || true is already tantamount to this.
by hendry on 10/17/15, 12:45 AM
Doesn't systemd timers https://wiki.archlinux.org/index.php/Systemd/Timers address these issues?
by zpidey on 10/16/15, 11:45 PM
These all seem like issues you'd run into with any task scheduler. Error emails, overloading a central resource with many tasks. Most of these aren't particular/limited to cron at all.
by ck2 on 10/17/15, 11:47 AM
gee this is a difficult concept
how to get cron to only send important emails and not every time it runs
you think maybe you should have just used
```
    > /dev/null 
```
and not
```
    > /dev/null 2>&1
```
why is this a full blog post?
by psior on 10/17/15, 2:38 PM
Very interesting discussion. I've been doing some related work where I needed to run some tasks in a non-overlapping way and while flock was an initial option I later moved to redis queue (ie rpush & blpop mix) to guarantee a certain (and needed) order of execution. This is mixed with a 'send email in case of error' check and so far is doing fine, though I'll definitely look into Jenkins if I ever feel this current approach proves not to be reliable enough.
by ausjke on 10/17/15, 3:01 AM
Great read and definitely will keep this in my toolbox, the whole article is explaining why the below good when you need use cron:
15 * * * * ( flock -w 0 200 && sleep `perl -e 'print int(rand(60))'` && nice /command/to/run && date > /var/run/last_successful_run ) 2>&1 200> /var/run/cron_job_lock | while read line ; echo `date` "$line" ; done > /path/to/the/log || true
by hbogert on 10/19/15, 8:12 AM
Excessive use of crons is a devops (hate the word) smell. You get reliant on their side-effects and to migrate to other solutions you need enormous amounts of testing and legacy interfaces. The most obvious downside to a cron is the at least 1m interval. On average you are waiting 30s for something which already should be there. Of course it's perfect for things like reporting which make sense for certain intervals. Using it for mail queues and stuff.. bad times.
by rbc on 10/17/15, 2:58 PM
Something else. The cron service is a one hit wonder. All it does is schedule. It places responsibility for handling output and setting semaphores for use by other applications to the person who wrote the command called by cron. You can't really blame cron if the command/script doesn't do these things. You just need to look to another type of scheduler/batch facility that provides a richer feature set for handling workflow, monitoring and reporting.
by pinterface on 10/17/15, 4:56 AM
I've been pretty happy with shush[1], which is a similar script that helps with a lot of this--including randomdelays, locking to avoid overlapping runs, e-mailing only on errors (or other criteria as you see fit), and so forth.
[1] http://web.taranis.org/shush/
by zimbatm on 10/17/15, 11:08 AM
https://github.com/Yipit/cron-sentry is also quite nice as a wrapper to capture failing cron jobs and forward them to https://getsentry.com/
by rbc on 10/17/15, 2:43 PM
CFEngine also provides a scheduling capability that can be used in conjunction with other factors using boolean expressions. Something like "Run at midnight on Saturday if you are a production linux server." The splaytime parameter can spread out the execution of a command across a cluster based on its name hash.
by ilamparithi on 10/17/15, 11:21 AM
We use https://wiki.jenkins-ci.org/display/JENKINS/Monitoring+exter... to monitor the cron jobs.
by impostervt on 10/16/15, 11:50 PM
I just started using AWS Lambda for a cron-like job I needed. It supports scheduling now.
by geggam on 10/17/15, 2:03 PM
Nice blog explaining how someone cant use cron. Doesn't mean the rest of us are that incapable.
by TerryADavis on 10/16/15, 11:58 PM
Linux is designed like a 1970's mainframe.
TempleOS is designed like a C64.
I don't see cron as useful for a C64 user.
by DannoHung on 10/17/15, 12:20 AM
Is there any good open source distributed scheduler that blends both timer based tasks and event based tasks?
Chronos is the only one I'm aware of, but I don't believe it supports event based tasks.
by vacri on 10/17/15, 1:00 PM
Another cron trick is to use chronic/cronic before your command. It silences the command except for error states - cron likes to report any text it sees, which you don't want for non-error states. It also detects errors better than just assuming all errors happen on STDERR.