by mattrobenolt on 8/25/18, 1:54 AM
Hey, developer of Sentry here. We're submitting a patch that prevents showing any settings on this DEBUG page.
I'd like to mention that we never suggest running Sentry in DEBUG mode, nor do we document how to do this. Sentry does use Django, so it's pretty easy to put pieces together and figure out how to do it.
So while we can help obfuscate this specific issue, running in DEBUG mode always has the potential to surface insecure information due to the nature of it.
Our patch to suppress this information entirely can be found here: https://github.com/getsentry/sentry/pull/9516
by matharmin on 8/24/18, 4:25 PM
This is why you (looking at frameworks) should never use a format that may contain code to store data, especially when the client has control over that data (even if signed). The same vulnerability has occurred in almost every language/framework that does this, including Rails and Java-based ones. Just use something like JSON, which completely avoids code execution vulnerabilities like this. Except of course for the early JavaScript JSON parsers that just used eval for parsing...
by Areading314 on 8/24/18, 2:34 PM
This is a great concrete example why you should never run debug mode on a public server. Django can only do so much for redacting private info. This is also a great example of how insecure pickle is!
by cc-d on 8/24/18, 2:47 PM
Facebook joins Patreon in the "why somebody should make sure our python web framework debug mode isn't enabled in prod" club.
by tekromancr on 8/24/18, 4:51 PM
Some surprising takeaways for me;
Facebook uses a Django app in their infrastructure.
That app was in debug mode, revealing server secrets.
That app was also configured to use the Pickle based session storage, leading to one of the few serious RCE vulnerabilities in Django.
I'll have to remember this next time I think an exploit scenario is too unlikely.
by gandreani on 8/24/18, 2:33 PM
Nice job! I also really appreciate the lack of memes and very concise format of this blog post
by beefhash on 8/24/18, 3:52 PM
>
Quoting the Sentry documentation, system.secret-key is “a secret key used for session signing. If this becomes compromised it’s important to regenerate it as otherwise its much easier to hijack user sessions.“; wow, it looks like it’s a sort of Django SECRET-KEY override!One wonders why that is even there. Was Django's own session code not good enough?
by smsm42 on 8/24/18, 6:44 PM
So, summarily we have:
1. Enabling debug mode in production
2. Running publicly-accessible app with publicly-accessible crash screens without any monitoring system noticing it's happening
3. Relying on auto-cleaning in debug facilities to sanitize security information (never works)
4. Using over-powered serialization protocol which allows for code execution for storing user-accessible data
5. Thinking that merely signing content prevents abuse of (4)
Not bad.
by Grollicus on 8/24/18, 6:27 PM
This is a good example why you need regular pentests in big companies. Everyone (should) know that using pickle is insecure and everyone (should) know that django debug should be False in production. Still, if the numbers get large enough someone will miss something.
by QuadrupleA on 8/24/18, 5:02 PM
Great article. Big frameworks like Django are nice to get an app started quickly, but if you use them long-term it really pays to study how they work under the hood, read some source code, etc. Although "don't run debug mode on production" is probably in the first page of the tutorial :)
by rhacker on 8/25/18, 12:51 AM
Congrats on the bounty payment! That has got to be one of the most concise blog posts too.
by amelius on 8/24/18, 4:23 PM
The fact that the machine has a hostname "*.thefacebook.com" doesn't imply that it also runs software of the "Facebook" social media software. So not sure how much impact this exploit would have had.
by green_on_black on 8/24/18, 4:01 PM
He got $5k for an arbitrary remote execution bug? What a rip-off.
by mandeepj on 8/24/18, 3:32 PM
> scanning an IP range that belongs to Facebook (199.201.65.0/24)
ping -4 facebook.com
results in 157.240.18.35. Maybe, author used some other way to get those IPs. Can anyone throw a light on this?
by Cthulhu_ on 8/24/18, 2:54 PM
Wow, a fix in <24 hours, that's pretty impressive.
by BFatts on 8/24/18, 2:39 PM
Great article...
One suggestion: You use "However" quite a bit. Not sure if you intended to show your thought process as it evolved, but that is the feeling I got.
by nickthemagicman on 8/24/18, 8:33 PM
So they were pickeling EXECUTABLE objects in the session and storing it in the users browser cookie?
Interesting.
Nice find.
by Kiro on 8/24/18, 8:44 PM
> Pickle is a Python module used to serialize data, but contrarily to JSON or YAML, it allows to serialize objects properties but also methods. In most of the cases, this is not a problem, but one can also serialize an object with code in the __reduce__() method, which is called when the object is unpickled.
Why does it run __reduce__?
by srcmap on 8/24/18, 3:01 PM
Other than fixing the Django source code , is there any OS level mitigation techniques that can detect and prevent such security vulnerabilities?
I am thinking something like selinux, docker or chroot - a bit like internal firewall for Django (or any other webapp).
Any suggestions on the links to latest best practices?
by mandeepj on 8/24/18, 4:20 PM
> I found a Sentry service hosted on 199.201.65.36
I do not remember on top of my head now but I think there are few scanning software to find all the running apps on a remote machine. If you are aware then please share
by protondonor on 8/24/18, 7:08 PM
Great hunt!
by jpmoyn on 8/24/18, 3:36 PM
Great, concise article! But the real question is how much did they pay you for the bounty??
by bayesian_horse on 8/24/18, 5:45 PM
I can get remote code execution on an Amazon Server (AWS). Do I get a cookie?
by exikyut on 8/24/18, 2:28 PM
So, this was simply taking advantage of a crash-prone webapp running on a debug-enabled Django instance using Pickle session serialization, and more specifically this was only possible because
_Django didn't redact the stored secret key used to sign serialized inputs out of the crashdump information!_Did the author tell Django about this yet, or is this a (possibly unintentional) 0-day?
Besides the above interestingness, the morals of this story I get are
- Stay persistent and leave your scanners running; you never know what new things will turn up.
- Crashdumps _are_ interesting
- Yay, $5,000!
- Middleware and frameworks will always clash in useful and interesting ways?
by Alex3917 on 8/24/18, 2:32 PM
In contrast, I submitted a bad vulnerability in Facebook’s password reset feature yesterday that lets attacker’s send password reset PIN numbers to email addresses the user doesn’t necessarily control. The security team said to works as designed so they’re not going to fix it.
Basically if someone requests a password reset on your account then the PIN number gets sent to all email addresses associated with your account, not only the primary one. This is an issue because many people have one locked down email address for things like registering accounts, but others they use to talk with people, delegate to their staff, use with CRM apps, etc. (But you still need your everyday email addresses linked to your account so that people can find you by email, see your email on your profile, etc.)
The FB security team just says that delegating your email address isn’t secure so it’s not their problem. Like no shit, that’s why it’s a vulnerability. But for some reason the FB security team thinks it’s a good idea to let anyone immediately bypass 2FA and hijack your account.