Audioboo goes on a Bank Holiday
A big massive apology to all our community for the failures over the weekend when boos and activation emails got lost in some weird 5th dimension space for 24 hours or so (not the best thing to happen when Guardian Tech says we’re the new talk of the net 
.
And probably not the best time for the core team to be either away in Cyprus for a week, having a quick break in an internet challenged Norfolk or without any access at all in Devon.
So what happened?
A small server error which is usually picked up manually caused a back-log of over 30,000 comment notification jobs to be created & internally posted to the main database. The ‘job worker’ then re-attempted to re-submit these, which delayed all the other server jobs to process the audio clips, to deliver authentication emails for new accounts. Etc, etc…
Essentially, the server thought it was far too busy to even think about posting new boos.
Currently, the audioboo server (yes, there’s just one) is carefully monitored on a daily basis to make sure it stays up. I think we’ve done pretty well so far with minimal resources and an excellent network architecture. We know we need to move to a more robust scenario (which we’ve designed on paper but not managed to find the cash for) but until then, the whole thing is kept going by ongoing manual interventions and tweaks . With the main team being out of contact for a few days (heck, we need a life) means the whole damm thing is very susceptible to failure.
So apologises to all, buts it’s been an eye opener for us. We’re going to focus in the next few months on raising some significant finance for audioboo (to be frank, it’s a bit of a side project at the moment) in order to enable it to perform on a more stable infrastructure, to exist on many more platforms than the iPhone and to develop some quite interesting API’s above & beyond the read/write ones we’ve already made.
Keep you posted!
I LOVE AUDIOBOO
I think you guys deserve nothing but praise for what you’ve achieved so far!
Your open communication explaining the issues is very much appreciated and highly commendable.
I’m certain we’re all very easily able to forgive a small blip, which frankly I’ve been surprised is the first in an excellent, uninterrupted service.
Hope you all had a restful weekend and thanks for the blog!
Regards
Mark
First I was annoyed. But I recognized that so far you do it all on a free base. Offering an API together with a pro version sounds good.
May be you could work together with other location based services. For example I use Brightkite (I really love it!) and it would be great if I could join bkite posts with boos – or even post with my iPhone app from Brightkite to AudioBoo and vice versa.
So I’m looking forward to you next steps
[...] with lengthy delays over much of the holiday weekend in audio content appearing on the website as CEO Mark Rock explained in a blog post last night. A big massive apology to all our community for the failures over the weekend when boos [...]
Thank you for your blog explaining the problems you experienced over the weekend.
Two suggestions:
1) Please can you provide a user search facility so we don’t have to memorize user’s names from the iTunes feed and put their names in the URL to find their boos.
2) May I suggest you organize a “Donation” button on your pages? PayPal would be a good way to start, don’t you think. I’d be willing to donate a small amount and I think other regulars would too, to help fund upkeep or another server etc.
Thank you for a great service. I felt bereft when my friends’ boos were missing though! =D
Sair
How about a note about this on the website someplace where we can see it?
Yes, it’s great that you guys do this service for free and I definitely appreciate it, but knowing that the server is screwed for a while and not warning us is seriously negligent and has caused a lot of us to waste time, 1st creating content that may be lost forever and second trying to work out why new posts haven’t shown up in the last 8 hours.
Not cool. Do whatever you need to, take the site down, take hours to process clips, whatever, but tell us what’s up. I had to go to help, then click on a forum post and then click on a link to the blog, which I didn’t know even existed. Keep your community in the loop and we’ll come to feel like we’ve invested in the site and be more likely to stay productive members of your community.
[...] how it goes, as well as keeping our fingers crossed that our servers don’t take yet another weekend off. This entry was written by Mark, posted on June 19, 2009 at 12:21 pm, filed under Uncategorized. [...]