from Hacker News

Paypal has not been sending payment notifications to merchants

by emilepetrone on 7/31/14, 10:50 PM with 71 comments

• by thenduks on 8/1/14, 2:05 AM

  You can make an API call to check the status of a transaction very easily.

  When my users are redirected back to my site (thanks page, or similar), I check if their transaction is completed, if not, I kick off an every-five-seconds check while asking the user to hold on while we talk with paypal. I will eventually fail after some number of checks of course, but this means PayPal can stop sending IPNs and everything will just keep going along just fine.

  If the user might not end up back on your site for some reason, run a cronjob that tries to verify transactions created in the past day/hour/whatever.

  An issue like this doesn't have to, and really shouldn't, cripple your business.

• by contingencies on 7/31/14, 11:53 PM

  If you've ever sat down and looked at Paypal's APIs properly, for instance with a view towards a proper risk analysis and/or from-scratch integration, then this should not surprise you. I did this for the first time perhaps ten years ago, and here's what I found.

  Firstly, the degree of marketing bullshit that you have to push through just to get to the truth of the APIs is staggering.

  Secondly, they perform backflips to avoid telling the truth about their integration options, which is to say that they're all fundamentally insecure for real time digital goods and services unless you implement IPN plus polling plus duplicate detection / additional round-trip validation calls, ie. you can't trust an IPN notification. The degree of complexity (order state tracking, IPN state tracking and duplicate detection, retry support, running your own IPN-receiving server) and latency requested here of client businesses is immense.

  Thirdly, their idea of international support is pathetic. It seems that they've basically duplicated their entire business process to other countries, translated it, and then assumed that everyone in that country requires only one human language in all of their interactions: documentation, support, emails, etc.

  Finally, as is widely known throughout the industry they have a shocking reputation for the arbitrary suspension and seizure of accounts and assets, with little to no recourse for those affected.

  I am not surprised that Stripe has taken off. Unfortunately, that's still lipstick on a pig: fundamentally, the settlement, risk, transparency and government interaction model of credit card networks means they are unsuitable for an increasingly large volume of business around the world.

• by erikcw on 7/31/14, 11:15 PM

  Not sure if this is related to this particular case. From previous experience, PayPal will silently stop delivering IPNs if your webhook URL ever returns a non-200 status code for an extended period of time.

  It took us a fair amount of time to figure out that this was the cause of our issue -- the PayPal UI didn't indicate any problems and PayPal business customer support was unable to see the source of the problem either.

• by driverdan on 8/1/14, 3:36 AM

  Here's an idea, stop using shitty services like PayPal. Today it's IPNs failing, tomorrow they're locking your account and holding your money for 180+ days with no justification.

• by jay-saint on 8/1/14, 2:31 PM

  We have been experiencing sudden weirdness with out PayPal Pro payment processing integration with our Magento cart over this time period. I have spent hours looking over logs and on support call with PayPal integration support. You would think that since PayPal and Magento are both part of eBays X.commerce that they would work together well.

  I received a lot of "hmmm that's weird" and "well i've never seen the API do that" comments from the integration support specialist at PayPal."

  What is happening for us is that Paypal is simultaneously sending a authorization and then a second later a capture. the results are weird some transactions process and other get a duplicate invoice it error.

• by myddryn on 8/1/14, 12:09 AM

  This has always been a problem with Adaptive Payments, just not always so severe. Paypal's Express Checkout product has the best solution to this in that a payment cannot be completed without an explicit API call by your application. This makes the payment confirmation IPN redundant.

  It is a more complex integration, but our product has been much more stable since we made the transition to Express Checkout.

  This does not however prevent other problems with IPN delivery on refunds and other post order activity. Paypal's IPN infrastructure is pretty weak in general, very poorly documented and completely untestable.

• by emilepetrone on 7/31/14, 11:02 PM

  This is a mission critical bug for any other startups that depend upon the Adaptive Payment API. That may only be a subset of Paypal merchants, but for those that do - you need to know this ASAP.

• by csomar on 8/1/14, 4:40 AM

  I don't remember accurately the documentation URL but there was a mention (I did lots of PayPal API development) that IPN cannot be always trusted and you should instead query the PayPal API (in a cron job fashion) to check if the transaction went through.

  There are many reasons why an IPN will fail. Like having your firewall settings messed up, network issues on your or PayPal's side, PayPal actually sending an IPN but your infrastructure not making use of its information...

• by WatchDog on 8/1/14, 12:12 AM

  If your business process relies completely on a callback that can fail for plenty of different reasons, then your business process is broken.

• by pbreit on 8/1/14, 2:48 AM

  I'd be very surprised if IPN was down for several days, much less 9. Can anyone confirm such a thing?

  IPN hasn't gotten much love in years but still pretty cool that it was (one of) the original webhooks back in 2001 and has worked decently for the past 13 years. In fact, an integration from 2001 should continue to work OK.

• by d_runs_far on 8/1/14, 1:41 PM

  I organize a bunch of trail running events, our registration page has both a stripe and paypal option; 85% use stripe, but the other 15% I don't want to give up on. Oh, and no IPN glitches here in the last 2 weeks (or ever actually - knock on wood).

• by abritishguy on 8/1/14, 9:24 AM

  a) Stop using paypal (or at least make it an option but recommend stripe or similar)

  b) IPN is clearly going to be flakey, there are several reasons why IPNs might not be delivered and they are not all paypal issues. In the paypal docs it says that if your webhook doesn't return 200 to some IPNs then it will stop sending them. For my new stuff I don't use paypal at all (stripe is just too good) but when I did I had a cron job running every 5 minutes that checked every transaction that had been created but no IPN had come through for. Over the cause of a couple of years I caught several transactions where the IPN had been lost.

• by kawsper on 7/31/14, 11:15 PM

  What is up with that blog title?

  > Disbursements through US Bank Accounts or Debit Cards

  Is it only related to US bank accounts and debit cards? Or is it an IPN issue affecting all card?

• by Xorlev on 8/1/14, 12:46 AM

  This explains a lot... my IPN handler had been awfully quiet, but thankfully it was only used for automatic thanks.

• by mgkimsal on 8/1/14, 12:18 AM

  had this problem for years, along with google checkout's callback process. google was worse. they seemed to get to a point where if/when they'd ping, if the request took longer than a few seconds to reply, it would stop, then never retry.

• by emilepetrone on 7/31/14, 11:21 PM

  BTW this post was #7 on HN, and then the HN gods flagged it for banishment. Not sure why...

• by kevin_thibedeau on 8/1/14, 3:05 PM

  Not to worry. Just the NSA deploying a MITM attack. The bugs will be ironed out soon and your transactions will proceed smoothly as before.