Posted 2 years ago
Updates in the processing engine
We just released version 1.8.5 Beta of RSS Graffiti.
The focus of this version was the processing engine. What was wrong with it? It inexplicably slow.
RSS Graffiti started on September 2009 and has been working like a charm since then. Of course there are always things to improve or problems to solve. But mostly everything was smooth as far as the application itself was concerned. Until, one day, growth caught up with us. Since the beginning we had set the background processing engine to adjust its performance automatically so that all feeds are processed every 10 minutes. That seemed like a reasonable number given the circumstances.
But since RSS Graffiti has been doubling its users every month or so, at some point we realized that the processing engine could not keep up, So the 10 minute update cycle started getting bigger and bigger. On the 1st of December 2009, RSS Graffiti processing engine was running at 10 minute cycles and was processing all the 5.000 active targets (roughly 8.000 feeds) that were using the application at that time. (In RSS Graffiti terminology a “target” is the equivalent of a “wall” in Facebook).
By the end of 2009 the active targets were 7.000 and the processing cycle had climbed up to 20 minutes. This did not make much sense. We tried to adjust the various settings we had in place for that but it was obvious there was a bottleneck somewhere. Christmas holidays setback any efforts for 3 weeks and by the end of January 2010 RSS Graffiti has grown to 13.000 active targets (over 20.000 feeds). Processing cycle had climbed up to 40 minutes. 40 minutes was still much less than the usual hourly checks other similar applications do, but still it was way to far from our 10 minute goal.
We had to do something about it fast. It has been months since we were discussing improvements on the processing engine but we now had to go about them fast and figure out where the bottleneck was.
To cut a long story short, the problem was in the database server. We tried quite a few tricks for 6 days (nights actually) and we came up with a solution that was released today in version 1.8.5 Beta.
Processing cycle is now back down to 4 minutes and we know we can maintain our 10 minute performance goal without adding any new hardware until RSS Graffiti reaches roughly 50.000 active targets. After that our current hardware will run out of capacity, but don’t fear. The application is designed to run on an infinite number of servers which can simply be added to the data center and just play. We estimate that if we don’t do any further performance optimizations in the processing engine we will need a server for every 50.000 active targets.
So, for the moment this issue seems to be behind us. It’s now time to focus on more exciting things! We will first fix some pending issues that have been reported by our users in the past weeks. Then we move on to add a Facebook Tab feature to RSS Graffiti. We really have big expectations from the RSS Graffiti tab and we are eager to start working on it. But we will talk about these plans as the time comes.
We’ll keep in touch!