PayPal IPN random fails

Hello,
Over the last week I've had a handful of failed PayPal IPN notifications. They seem to be random as there are other IPNs coming in around them with no problems. 

In PayPal, it states an Error 500 for these failures, however the site has been up and there are plenty of other IPNs going through. I've set the filed IPN to resend and they are retrying. I'll give it a day or so. In the meantime, I've given the users access manually, and sort out duplicates later. 

I'm not sure why this may be happening, and if I'm the only one with this issue?

I've taken the notification URL (not sure if this is a good idea) and entered it on a clean browser. 
The URL is something like  mysite.com/index.php?option=com_comprofiler&view=pluginclass&plugin=cbpaidsubscriptions&cbpgacctno=1&cbpbasket=xxxx&do=gw&result=notify&cbpid=cbpxxxxxxxxxxxxxxxxxxxxxxxx&format=raw
The result was the Joomla error page with the error "1062 Duplicate entry '12' for key 'jos_cbsubs_notifications.PRIMARY'" 

Not sure what to do with this. It doesn't help that it's Friday the 13th, in October no less. 
Thanks,
Tomás

Are multiple notifications coming in at the same time? It's possible it's trying to process and save 2 notifications at the same time resulting in that issue. Normally they're at least a second apart. PayPal retries notifications though so it should be retrying shortly afterwards and process fine. Is it not retrying?

I cannot make heads or tails of this one. 

• I have 2 sites on this server (dedicated server, fully controlled by me).
• Both sites are on Joomla 4, CB 2.8.2 & CB Subs 4.6.2, PHP 8.1
• Both sites only have PayPal and Stripe as payment options. 
• Both sites have Gateways with secondary “owners” (Going directly to other PayPal accounts I don’t have access to). The secondary gateways are only PayPal (not Stripe)
• Over the last couple of weeks, about a dozen IPN notifications failed from PayPal
• I was notified a few days ago that some subscription payments were not showing on the site
• Other IPNs which came before, between and after those that failed, went through OK.
• I get about a dozen IPNs a day, all going through fine, except for the aforementioned.
• As far as I can see, all the failed IPN included members who also pay to secondary gateways (meaning gateways with other owners). Obviously, the payments to different Gateways (and owners) are completely separate and independent transactions, the setup won't allow combining them. 
• Members that had a secondary gateway payment who’s IPN failed, also had their payment to the master gateway IPN fail, even as these were completely independent payments. Making it highly coincidental.
• This is making very few users have multiple IPN failures.
• 2 members, who were unaware their payment had gone through, but the IPN failed, when they couldn't access the site, they paid again, and the IPN failed again. Again, highly coincidental.
• All retries of the failed IPNs (16 retries for each IPN) failed. This is while other IPNs go through OK.
• I manually forced the failed IPNs in PayPal to retry, but all failed again. 
• I didn’t see any stepping on each other (the same second), but I’m not sure I would see this as the IPN’s failed and some don't come from my PayPal. Seems very unlikely though.
• To my knowledge, the site has never been down showing a 500 error
• There have been no issues with Stripe, but there are no secondary gateways with Stripe.
• I migrated the first site with less traffic to Joomla 4 on September 10. 
• I migrated the second site to Joomla 4 on October 4. 
• The first IPN fail was on October 1st, for the first site. I'm not aware of these IPN failures before the upgrades. 
• Today, there was another IPN fail (PayPal states Error 500) although the site has been up and there were successful IPNs and webhooks before and after (spaced by 20 minutes or so).

The only common thread that I can see is the members paying to the Gateways that have an alternative owner, but there are plenty of members that have payments to the other Gateways that don’t fail. 
It would be quite involved, but I could try to find out if the failed IPNs come after the 2nd payment to a Gateway with a different owner, since the last upgrades. The hard part is that I don’t have access to those PayPal accounts of the other Gateway owners, but I can ask and try to put it together. 

Any ideas?
Thank you,
Tomás 
 

CBSubs should be logging those failed IPNs within CBSubs > Notifications as well as within CBSubs > History Log. Are there no failed notification entries and no errors logged for those failed IPNs? That should tell you exactly what the error is. If nothing is logged it's possible something installed is causing an error before CBSubs processes anything. In which case are there any errors in your server error log around the time of the failed IPNs?

krileon wrote:

CBSubs should be logging those failed IPNs within CBSubs > Notifications as well as within CBSubs > History Log. Are there no failed notification entries and no errors logged for those failed IPNs? That should tell you exactly what the error is.

There's nothing there. None of hte failed IPNs show on either of those logs. The only errors I see there are "stripe Webhook version mismatch", but those are still going through OK.

krileon wrote:

 If nothing is logged it's possible something installed is causing an error before CBSubs processes anything. In which case are there any errors in your server error log around the time of the failed IPNs?

There's nothing in the /administrator/logs and nothing in the Apache error or access logs, but there's this in the other_vhosts_access.log 
The first IPN for basket 3336, went through fine, but the next entry in the log, 2 seconds later gave a 500 error for basket 6521, which re-tried another 2 minutes after and also returned a 500 error. 
This second payment, was from an existing monthly recurring payment that had successfully gone through for several months before, and obviously one of the new failed IPNs in PayPal. 

¯\_(ツ)_/¯
Tomás 
 

CBSubs not logging anything is concerning. That usually means CBSubs wasn't loaded yet since our logging hooks directly into PHP error handler. It's likely there's something else on the site possibly causing your issue. You mentioned both sites. Is this happening on both sites or only 1 of them?

There's also a potential concern in your access log. Notice the one that succeeded has &cbpgacctno=2 and the one that failed has &cbpgacctno=1. cbpgacctno is the gateway id. So the one that succeeded has a gateway id of 2 and the one that failed has a gateway id of 1. Please check your payment gateways within CBSubs > Gateways and review your settings for the gateway with an id of 1. Seams like you have 2 PayPal gateways?