Audioboo goes on a Bank Holiday
A big massive apology to all our community for the failures over the weekend when boos and activation emails got lost in some weird 5th dimension space for 24 hours or so (not the best thing to happen when Guardian Tech says we’re the new talk of the net .
And probably not the best time for the core team to be either away in Cyprus for a week, having a quick break in an internet challenged Norfolk or without any access at all in Devon.
So what happened?
A small server error which is usually picked up manually caused a back-log of over 30,000 comment notification jobs to be created & internally posted to the main database. The ‘job worker’ then re-attempted to re-submit these, which delayed all the other server jobs to process the audio clips, to deliver authentication emails for new accounts. Etc, etc…
Essentially, the server thought it was far too busy to even think about posting new boos.
Currently, the audioboo server (yes, there’s just one) is carefully monitored on a daily basis to make sure it stays up. I think we’ve done pretty well so far with minimal resources and an excellent network architecture. We know we need to move to a more robust scenario (which we’ve designed on paper but not managed to find the cash for) but until then, the whole thing is kept going by ongoing manual interventions and tweaks . With the main team being out of contact for a few days (heck, we need a life) means the whole damm thing is very susceptible to failure.
So apologises to all, buts it’s been an eye opener for us. We’re going to focus in the next few months on raising some significant finance for audioboo (to be frank, it’s a bit of a side project at the moment) in order to enable it to perform on a more stable infrastructure, to exist on many more platforms than the iPhone and to develop some quite interesting API’s above & beyond the read/write ones we’ve already made.
Keep you posted!