Posted 3 years ago
Problems with feeds from Yahoo! Pipes
The past few days our users that have Yahoo! Pipes feeds added to their Facebook Profiles and Fan Pages, started reporting that their Yahoo! Pipes feeds suddenly “stopped working". Their feeds stopped posting for a few hours then some of them posted some entries and then paused again for a few hours and so on.
Looking into this, we discovered something we probably should have thought about earlier: Yahoo! has a hard rate limit imposed on Pipes. This means that Yahoo! only accepts to execute a Pipe just a given number of times per hour. If that limit is exceeded then the Pipe is blocked for an hour; and so on.
The limits of Yahoo! Pipes are listed in their FAQ and seem to be as follows:
- 200 runs in 10 minutes for a given Pipe.
- 200 runs in 10 minutes for any Pipe from a given IP address.
Limit #1 is not a problem because RSS Graffiti runs each Pipe roughly every 4 minutes. So we never hit the limit #1. But limit #2 is indeed a problem because RSS Graffiti runs much more than 200 Pipes every 4 minutes. So since all runs come from the IP of RSS Graffiti’s server, the limit is exceeded and Yahoo! stops running any Pipe for RSS Graffiti for an hour after the limit is hit.
Before version 1.8.5 Beta, feed processing was not so often (every pipe was run once in every 30-40 minutes), so we did not face this problem. But now that we are able to make such frequent updates, we need to find other ways to overcome these limits.
Here are some workarounds we already have in the works:
First of all we already sent a request to Yahoo! for whitelisting RSS Graffiti’s server IP addresses so that the limit is raised to a more reasonable lever for our needs.
Another thing we are preparing for release ASAP is "per feed scheduling”. This will allow our users to set the number of minutes between updates to each feed. This way, users that really need Yahoo! Pipes will be able set their Yahoo! Pipes feeds to update less often than the rest of their feeds, effectively avoiding to hit the Yahoo! Limits.
We also noticed that most of our users that bring Yahoo! Pipes feeds to Facebook, actually do it to bring in their twitter feeds, after trimming their twitter usernames from the feed entries by using a Pipe. So we are already preparing a solution for those users: the next version of RSS Graffiti is going to trim the twitter username automatically from twitter feeds so there is not going to be a need for these users to turn to Yahoo! Pipes to do the trimming.
We will keep you posted on this. Our efforts will concentrate on having these alternatives available for you tomorrow.
- rssgraffiti-status posted this