The problem is that if you're somewhat high profile, and guest post on other higher profile sites you should be okay. Move down the "food chain", and Google doesn't look so kindly at your guest blogging strategy.
*http://www.mattcutts.com/blog/guest-blogging/