A breakdown of our diagnosis of recent Yahoo Contact API failures.
About a month ago I started to observe an annoying error with our Yahoo! imports. There were periodic failures at a very low rate. The failures were all identical; they looked like this:
{"error":{"lang":"en-US","description":"Please provide valid credentials. OAuth oauth_problem=\"signature_invalid\", realm=\"yahooapis.com\""}
It was very hard to reproduce the issue in development, although it did happen periodically. When I was able to reproduce it locally, I found that I would get a successful response if I resubmitted the same request, the only difference being that the random OAuth parameters were regenerated: nonce, timestamp and signature. Because of the visibility and the challenge in reproducing, I just followed my heuristic and built the retry logic into our application. For a while that was mostly successful, lowering the daily average failure rate well below 1%.
Over the past two weeks the failure rate started to slowly creep up again and I tried my best to ignore the problem. I did extend my retry logic to try a second and third time before giving up, producing some diminishing returns.
Finally, I could hide from the issue no longer yesterday when 100% of our Yahoo imports started to fail with the same issue.
Exactly what was happening on Yahoo’s side during this time is anyone’s guess. Maybe some bad code had been isolated to a portion of their servers managed to infect the rest of the farm. Maybe it’s just Seasonal Jitters. Whatever the cause, we were unable to connect even once to Yahoo between 2PM yesterday to 6AM this morning. Then even more suddenly, everything started working again. Here’s what the failure rates looked like:
This is not the first instance of this problem with Yahoo! In December, 2011 we had a 13 hour period of disruption with the same unexplained symptom: oauth_problem:signature_invalid. And we’re not the only integrator who noticed the issue. At least 2 threads on developer.yahoo.com point out the issue with the Profiles API and the Contacts API. There’s been no verbal response from Yahoo! but at least we know they found the fixed the issue for now.
