Feed: Backflip | Site Maintenance - AggScore: 17.9

[ October 13, 2009 - 7:00pm PDT ]

A power outage at our hosting facility took out all of our servers for several hours starting around 7:00am PDT. Once the power came back on we began to bring the servers up remotely. Unfortunately, the database server is refusing to start up. Surely it's nothing terminal but will require human intervention and there is no one available to head over to the colo facility at the moment. Please check our twitter microblog on the left for the latest updates.

[ January 5, 2009 - 11:08pm PST ]

We're up and running now however we're not completely done with the recovery and there are parts of the site that will not work. We appreciate your patience while certain parts of the site come back over the next week. You do not need to be worried about making changes to your account while recovery continues. Our processes account for new changes.

Here's a short list of things that you might run into:

• Recently backflipped pages that don't have a url - "internal error" when you click on the link (this is the #1 problem we're looking at right now)
• Internal errors on some public folders - fixed jan 6 @ 1:36am
• Internal errors on the "my sites a-z" tab - fixed jan 6 @ 1:48am
• New users unable to sign up - fixed jan 6 @ 11:40am
• Incorrect folder counts, overall page counts
• Dates may look strange - fixed jan 6 @ 11:40am
• Top Ten lists may be messed up (wrong order, missing entries) - this is most likely not fixable through our recovery process

Now for the bad news. We lost some data, most likely irrevocably. There was a long list of causes: hardware errors, software errors, and some human errors which lead to the loss of a data file which contained activity between March and October of 2007. User accounts created during this period may be gone completely. Pages that were backflippled, folders, shares, public folders, etc. created during this period may be gone. Some pages that you may have moved into folders created during this period may be back in their previous folder. It's unfortunately difficult to measure the extent of the data loss. Overall it accounts for less than 5% of the backflip database, but for some users it may be much worse, while for others unnoticeable. We haven't given up completely on recovering this missing data file, there's still some long shots that we will attempt. Our first priority, however, is to move forward, keep the site running, and stabilize it.

Thank you for hanging in there with us. Thanks to the well wishers and especially those of you who donated which helped offset the cost of the new hardware.

[ December 27, 2008 - 12:19am PST ]

More snags, more postponements. I'm not going to make any more updates here until we have solid information for you. See the our Twitter microblog for the most up to date info.

[ December 26, 2008 - 12:57pm PST ]

Unfortunately we have to move the ETA for site return to Saturday, December 27th. The new machines are getting installed at our facility in San Francisco at around 11pm today. The data will then need a prolonged restore procedure from our backups. Ballpark estimate is 24 hours. We will update you via the our Twitter microblog as we proceed. Thank you for your patience.

We'd also like to take a moment to thank all of our loyal users for the donations we've received in the last 48 hours. We are truly grateful.

[ December 24, 2008 - 10:59am PST ]

We are scheduling the database rebuild for Friday, December 26th. The site will remain down until that point.

[ December 23, 2008 - 2:37pm PST ]

Bad news. Our database server suffered a fatal hardware failure. All the data is backed up so we're not worried about getting your data back, however due to the holidays we currently do not have the manpower to put a new server in place for a few days.

As you may or may not be aware (well, I guess you are now :D), this site is run by volunteers and has no real income except enough to cover the costs of the bandwidth, space, and power in our hosting facility. Hardware is purchased from our own pockets so if you ever considered donating to keep Backflip running, now would be a really good time.

We are going to try and get the site up as soon as possible. Check this page for big updates but for the most up to date information always check our Twitter microblog. We thank you for your patience and wish you Happy Holidays and hope to update you with good news soon.

[ December 23, 2008 - 8:56am PST ]

We're experiencing some database errors this morning. Please stand by. For the most up to date information remember to check our Twitter microblog.

[ October 29, 2008 - 4:51pm PDT ]

In addition to this status page we will now also post quick status updates via Twitter. Use this twitter link to check on the latest status and sign up for the rss feed or get a twitter account and follow backflip_status.

[ October 11, 2008 - 8:11am PDT ]

This is a followup entry on the Denial of Service attacks we have endured in the last few months. Why would someone target Backflip? What you need to understand in that these DoS'es were not specifically designed to ruin the backflip service. Quite the opposite, the vast majority of the "attacks" were side effects of a variety of bots attempting to automate the use of Backflip for the purpose of raising their customers' page ranks.

Even before I went to try and identify the sources I suspected that our attackers were primarily black hat SEO providers. The way these outfits work is that you pay them to get your business page to appear higher in a search engine's results. Backflip provides a perfect target for a black hat SEO outfit.

I've spent several weeks now identifying the sources of these attacks. In general, many of the attacks came from within server hosting facilities. They are not themselves responsible, they simply host a black hat CEO client. Fortunately these are typically upstanding facilities with an abuse department. They have been notified.

Amazingly, however, by far the most nefarious source of the attacks (responsible for almost 70% of the attacks) turns out to be a specific company: l o g i c a l   s o l u t i o n s. The reason why I'm not listing a url for them or even writing their name normally is because I do not wish to give these unscrupulous spammers (yes, SEO can be a form of spam). Their primary business? Search Engine Optimization!

What are we doing to stop these spammers? For one, we're blocking their ips at the firewall. The majority of our users are not using servers inside a hosting facility to surf backflip. We've put into place code that prevents the abuse of the site and to alert us of any further attacks. These are in no way perfect solutions and a particularly stubborn spammer may still be able to abuse us but never again on the scale that we've experienced in August and September.

Amazingly, as I was writing this, an obvious SEO spammer actually emailed me asking me why the site wasn't working for "him". I searched for his username on Google and almost fell out of my chair laughing. The same username appears on every imaginable bookmarking website and containing the same exact links he is trying to peddle on backflip. I didn't write much of a reply. I simply sent him a link to this status page :-) Ya, I'm talking about you, "Mak"...

[ October 10, 2008 - 10:30pm PDT ]

Today's outage had nothing to do with the previous problems and denial of service. The primary webserver crashed and had to be rebuilt. Apologies for the prolonged outage especially considering all the other problems we've recently experienced.

[ October 4, 2008 - 4:25pm PDT ]

Believe me folks, I more than understand your frustration with the site performance. I'm a backflip user too! I've been working for a few weeks now on a solution to this denial of service problem and I think I finally have a way to fight these *idiots* ruining the site for everyone else. I will not be going into any details but am hoping that everyone will notice the site performance getting better in the next few days. As to the script kiddies - we know who you are...

[ September 8, 2008 - 1:21pm PDT ]

We are still having massive problems with performance. In the last 2 weeks it's come to light that we're under a low key denial of service attack. Unfortunately it's very difficult right now to differenciate real users from the attackers (who are hitting us from several dynamic ips). We ask for your patience as we try and figure out how to stop the attacks from continuing to degrade our service.

[ August 23, 2008 - 9:28pm PDT ]

Yes, we are very aware of the decreasing performance of backflip over the last month or so. We've got a bunch of ideas on the table on how to fix the situation. We're optimistic that this fall will see new hardware and improved overall performance for backflip. So hang in there and thank you for your patience.

[ July 7, 2008 - 1:30pm PDT ]

Sorry about the prolonged outage this morning (started around 9am PDT). We had some storage issues that have now been resolved.

[ June 26, 2008 - 7:00pm PDT ]

Our webserver crashed this morning at 11am PDT and needed minor (but time consuming) repairs. We're back now and no data loss occured. We appologize for this prolonged outage.

[ May 13, 2008 - 2:47pm PDT]

Some of the slowness of the system in recent days has been due to a failing piece of hardware. We hope to have it replaced soon. We're also bringing online a new web server which will also help make the site speedier.

[ November 16, 2007 - 5:45am PDT]

Wow. What a morning. First off, we have a database glitch that needed resolving. Second, for some reason the domain backflip.com didn't renew automatically and networksolutions put up one of those hideous landing pages for the domain. If you're reading this, you probably missed the terrifying feeling when you go to backflip.com and see a stock photo and a bunch of networksolution ads! Also, if you are reading this be assured, it was just a clerical error, we are not going away! :)

[ September 3, 2007 - 10:02pm PDT]

The service was flaky all weekend and we appologize. I think we've got things stabilized now.

[ August 26, 2007 - 8:56am PDT]

A reconfiguration issue caused the web server to fail restart after a critical server update required a reboot. The configuration has been corrected. Big apologies for a totally avoidable outage.

[ August 22, 2007 - 8:45am PDT]

Well we went 2 and a half months without problems coming up but the dreaded database issue happened again this morning (it's basically related to the disk layout and space available and how we manage it). This time it was purely human error on our part. The issue is fixed and everything should be back to normal now, sorry for the inconvenience.

[ August 4, 2007 - 12:17pm PDT]

We've been monitoring this website using Wormly for several months now. To keep us honest, we've installed a "Wormly certified" uptime meter at the bottom of this page. It shows our uptime over the last 30 days and updates automatically.

[ July 10, 2007 - 2:15pm PDT ]

One month without those db outages feels good. I think we have that one figured out now.

[ June 11, 2007 - 11:30pm PDT ]

If you think you're tired of these outages think about how we feel :D - another minor db issue that's causing an extended outage, i'm afraid. Back as soon we can.

[ May 21, 2007 - 08:02am PDT ]

Regrettably our database problems are back again! We are very sorry about these sudden outages and are working hard to correct them. No data has been lost so hang in there please.

[ May 18, 2007 - 11:22pm PDT ]

The database problems went away. Appologies for the intermittent outages.

[ May 18, 2007 - 5:24pm PDT]

We're having some minor problems with the database today that unfortunately caused site wite outages twice today. These should be fixed shortly.

[ February 20, 2007 - 9:42am PST]

The webserver got into a strange state this morning. No idea (yet) what caused it but we will be investigating. Site is to normal again now.

[ February 15, 2007 - 10:02pm PST]

We're experiencing intermittent problems with our database servers today. We're working on a fix sometime tomorrow morning.

Update February 16th, 2007 - 12:41pm PST: Sorry for the outage. We're back and no data was lost.

[ February 1, 2007 - 7:22am PST]

The January 31st outages are over. The morning outage was actually over at 5:02am so 4.5 hours ahead of schedule. However we made up for that situation by having an unplanned 1 hour outage at 11pm :) Sorry folks! But we are on a solid electrical circuit now at least.

[ January 30, 2007 - 10:09pm PST]

We will have an outage from 4:30am to 9:30am PST on wednesday, January 31st. Apologies in advance, but this is necessary as we are moved to a stable electrical circuit to prevent the sort of outage we saw on January 20th.

[ January 20, 2007 - 3:07am PST ]

Another power outage in our data center. We are awaiting an explanation from the providers. We're, once again, very sorry about this. Backflip was down for about 2 hours.

[ January 19, 2007 - 11:16am PST ]

A major power outage in our data center knocked out some of our servers at approximately 6:30pm on January 18th. Two of our servers failed to come back online and had to be restarted manually. This was done at 7:30am and 9:30am today. We are discussing the cause of the outage with our facility. Our apologies for the length of this downtime.

[ December 14, 2006 - 10:30pm PST ]

We had another web server related outage between 4:15pm and 9:30pm PST. Things should be back to normal now.

[ November 13, 2006 - 8:40am PST ]

The service is back after more than 24 hours of downtime. We're apologize for the length of this outage. The problem was caused by human error in the configuration of the web server. Thus we truly apologize as it could and should have been avoided.

[ October 14th - 12:15pm PDT ]

Our database experienced a temporary problem which caused all logins to fail as if your account wasn't there. This problem has been fixed. No data was lost and you have our apologies for the inconvenience.

[ May 21, 2006 - 10:22am PDT ]

Yesterday around 5:30am PDT the webserver croaked in a way that couldn't be fixed remotely. As all the backflip volunteers were out of town it wasn't until this morning that we could run in and fix the problem. The problem is now fixed. There was no data loss.

[ April 20, 2006 - 7:46pm PST ]

Today's outage was caused by a faulty disk in the primary web server. The disk has been replaced, no data loss occured.

[ March 23, 2006 - 10:28am PST ]

We experienced an outage that last from March 22 @ 5:30pm PST until March 23 @ 10:30am PST. This outage was caused by human error during reconfiguration of the primary web server. We apologize for the downtime. On a positive note, we were able to fix the problem 8 hours ahead of our initial ETA.

[ February 12, 2006 - 11:22am PST ]

Sorry about the short outage this morning. Everything is back to normal now.

[ February 3, 2006 - 11:04am PST ]

We're back, sorry about the outage.

[ February 3, 2006 - 4:10am PST ]

We are temporarily down while investigating a problem with our database server. We should be back in a few hours.

[ December 25, 2005 - 4:29pm PST ]

We're back after a 12 hour outage. Sorry about the inconvenience. All of us happened to be away from our computers (a rare treat) and thus unavailable to fix the minor problem that caused the system errors in short notice. Happy Holidays to all!

[ June 24, 2005 - 11:55am PST]

We're back after an approximately 27 hour outage due to a serious hardware failure on our webserver. Thank you for your patience. We hope you've noticed that the site has gotten significantly quicker in the last week. The Database Human has been doing some wonderful optimization work. We should see further improvements throughout the summer. Thank you again for your continued support.

[ March 27, 2005 - 1:18am PST]

A fire in San Francisco caused a power failure outside of our control around 7pm. Backflip was back 30 minutes after power was restored however a short brownout took down one of the backflip servers around 12:30am. Service is now back to normal.

[ February 23, 2005 - 6:41am PST]

We are aware of the problems with public search results being full of holes and are working with Google to understand why so many pages that exist in their index are not showing up.

[ February 10, 2005 - 10:55pm PST]

We're back after one of the worst backflip outages in years. The service went down on February 5th when our primary web server failed. As we were building a replacement server Murphy's Law struck us hard and the replacement server failed in exactly the same way... We finally got a third server built from scratch up and running last night, but it took until this evening to get all the software kinks straightened out. Thank you for your patience.

[ December 29, 2004 - 1:09pm PST]

Found a bug that was causing pages to load much slower than necessary. Most users should see a dramatic performance increase when using their account.

[ December 28, 2004 - 11:15pm PST]

The Backflip Buddy is back. It is fully operational in Internet Explorer 4.0+ and Mozilla based browsers such as Firefox. That's right, even the Daily Routine is now supported in Firefox.

[ December 28, 2004 - 1:51pm PST]

Public search is now working using the incredible Google search engine.

[ November 15, 2004 - 2:41pm PST]

This afternoon we were forced to repeat the operation from November 10th. During the time it took to do this (around 2 hours) Backflip was down. We should not have to perform this sort of maintenance again.

[ November 10, 2004 - 12:41pm PST]

This morning at 8:45am we needed to perform an unplanned database operation. Backflip was unavailable during this time. The operation was a success and patient is doing great :)

[ September 24, 2004 - 10:43am PST]

Yesterday and today our server facility upgraded their power grid configuration. This unfortunately meant that our servers had no power 4 different times in the 24 hours from 9:30am Pacific on the 23rd to 9:30am Pacific on the 24th. We're back now. We apologize for the prolonged outages.

[ July 17, 2004 - 11:06am PST ]

We are aware of the "blank page" issue with Internet Explorer users on PCs. I think one of the recent Microsoft security upgrades changed the way IE determines when it's done loading the page. If you see just the top banner and the browser says "done" just wait a little longer, the page *will* eventually draw. The real problem here is that we need to make the site faster, and we're planning to make some upgrades by the end of the summer. Please bare with us.

[ July 10, 2004 - 10:12am PST ]

I've made some changes with the export tool that should fix a problem a lot of users have been seeing where not all their pages were being exported correctly.

[ February 9, 2004 - 6:00pm PST ]

All is well again. Sorry about the brief interruption.

[ February 9, 2004 - 5:30pm PST ]

Our database has experienced a temporary error. We should be back online shortly.

[ January 25, 2004 - 12:12pm PST ]

This morning's downtime was unrelated to the work yesterday. Our co-location facility had power problems starting at around 5am PST. Sorry for the inconvenience.

[ January 24, 2004 - 5:00pm PST ]

We're back. Upgrade is now completed.

[ January 24, 2004 - 1:00pm PST ]

Greetings and Happy New Year. We are in the process of upgrading our database server. This process should take around 6 hours during which the backflip site will not be available. Thank you for your patience.

[ December 21, 2003 - 5:56pm PST ]

A major power outage in San Francisco brought down countless web sites, including Backflip, last night at 6pm PST. A fire in a critical grid distribution point was the apparent cause of the outage. Backflip experienced no data loss, however we were unavailable for the full 24 hours.

[ November 5, 2003 - 4:10pm PST ]

A primary power supply failed in the backflip system last night and it wasn't until this afternoon that this problem could have been corrected. Sorry about the downtime.

[ October 25, 2003 - 12:16am PST ]

Top Ten lists are back. Sorry about the outtage. We were working on a new feature and a piece of the dev code accidentally snuck into the production system. The offending code has been found and removed.

Our support system has been slammed by a sort of denial of service in the form of an unbelievable amount of spam. As a result we are forced to close all existing tickets. If you are having a problem, please open a new ticket (by sending email to support@backflip.com). Sorry to all the people who may have had open tickets, there's over 10,000 tickets, 99.5% of which are spam. The Support Human just doesn't have the bandwidth to weed out the spam from the real tickets.

To prevent the support queue from getting slammed again, we've installed a spam filter called Vipul's Razor. Hopefully this will keep the spam in check.

[ August 26, 2003 - 8:51pm PST ]

The database got into a strange state today and for a period of a few hours you may have experienced errors logging into the website. Not to worry, all your data is still here and you should no longer have problems logging in.

[ July 15, 2003 - 9:30am PST ]

Our primary network switch gave up the ghost early this morning (around 2:10am). We had to wait until the local computer store was open to replace it. We now have a new switch and the service is stable again.

[ July 11, 2003 - 6:17am PST ]

We're down this morning for a few hours fixing stuff. Sorry for the inconvenience.

[ May 19, 2003 - 7:07pm PST ]

One of our advertisers pulled a fast one on us and managed to serve pop-up ads as you use the Backflip website. Please believe us this was totally outside of our knowledge. A couple of users complained and then I finally got one myself. I am furious at our advertiser. But I also want to apologize for any inconvenience. I figured out a way to block the pop-ups regardless of what our advertiser is doing. -- Support Human

[ May 18, 2003 - 4:27pm PST ]

Changes in our Domain Network Server configuration caused the web server to lose connectivity to the database server for several hours yesterday (May 17th). The symptoms included "internal errors" and backflip claiming not to know you at login time. This was corrected and no data loss occured.

[ April 17, 2003 - 8:36am PST ]

A misconfiguration of our mail server caused problems for users sharing links and folders with other users (mail wasn't getting sent) between April 4th - 15th. This has now been corrected.

[ February 24, 2003 - 01:58am PST ]

We are proud to announce that we now have a much more powerful, semi-automated support system in place (support@backflip.com). Until this day we basically been answering email using a regular 'ole email client. From now on all support tickets will be tracked and monitored. Hopefully this will be of great benefit to our loyal users a lot of whom were probably getting frustrated at the lack of support in recent weeks.

[ January 25, 2003 - 4:45pm PST ]

Our web server got cranky today making it impossible to log in to fix a very simple problem. Everything should be working normally again. We're investigating how to prevent this sort of silly issue in the future.

[ January 18, 2003 - 3:18pm PST ]

A small database glitch similar to what happened on January 7th caused the site to behave strangely as if user accounts were no longer available. We've resolved this and you should be able to log in again. Thanks for your patience.

[ January 7, 2003 - 5:20pm PST ]

We experienced a small glitch today that was quickly resolved. It caused the site to behave as if user accounts were not being accepted. No data was lost. Sorry about the inconvenience.

[ December 14, 2002 - 2:13pm PST ]

Small glitch today. Nothing to worry about, we are 100% operational again.

[ December 3, 2002 - 5:18 pm PST ]

We are aware of some discrepencies between your folder counts and the number of marks inside your folders. This will be corrected.

We fixed some bugs recently:

• You should no longer get internal errors when navigating inside the Everything Else folder
• Search results can now be sorted by Relevance, Title, or Date
• What's Hot on Backflip now gets updated daily

[ November 23, 2002 - 00:49 am PST ]

After 10 days of downtime we are back! The database crash damage has been undone. Please note, because of the serious corruption we encountered (hardware related) we had to restore from our last known good backup, and rebuild the more recent data. We concentrated on restoring all users, bookmarks, folders, and descriptions. However, because we had to piece the data together from a corrupt hard disk don't be surprised if you see small weirdness like a bookmark you deleted within the last few weeks having returned, or changes you made to a bookmark description recently being undone. We decided it was more important to have the site back up and running than worry about such details. Please let us know if you run into more serious problems.

Give a big cheer to our database engineer. He worked pretty much straight for 10 days putting this baby back together.

On a side note, while the site was down we took the opportunity to eliminate a bunch of problems a lot of you have been seeing when using Backflip from behind corporate and personal firewalls (images not loading, buttons not working, etc.) by moving our "static" objects (images and such) to a separate server.

[ October 1, 2002 - 4:00pm PST ]

Long overdue, our search feature now works again. You can currently only search your own folders (not public ones, that is still coming). Also, shared folders that are not your own (but attached to your account) will not show up in the search as well (this functionality will also be introduced in the near future).

Thank you for your patience

[ September 12, 2002 - 9:09pm PST ]

It's been one of those days. In the morning, our network provider had major problems with a critical router. This was resolved around 3pm Pacific time. Shortly afterwards (around 5pm Pacific), the web server crashed and had to be fixed. Just when it looked like we were home free, the database server crashed at 8:30pm Pacific time. Argh! We do not expect the site to be up until around 9am Pacific time on Sept. 13th. We appologize for this prolonged outage.

[ May 26, 2002 - noon ]

We just want to take the time now to apologize for the myriad of downtimes in the last month. Murphy's law hit us hard. Everything that could go wrong pretty much went wrong. We lost our primary web server. Our domain information got messed up causing the site to appear as if it was gone. Power failures outlasted our batteries, one of us got into a motorcycle accident and broke his arm, etc. etc. etc.

So before you jump off this ship, we just want to say: This doesn't mean we're going away!. Quite the opposite. We have just completed moving Backflip into a new server room in San Francisco with much better bandwidth (you should be noticing a significant improvement in performance), much better power grid, 24/7 staff, etc (in fact, one of the people running the new server room is yet another ex-Backflip employee!). We are also planning on putting up some redundant web servers in the very near future as well as finally getting search working.

So stick around. Backflip is here to stay.

[ February 18, 2002 - 9am PST ]

Many of you (who are IE6 users) have noticed that despite clicking on the "Sign in automatically" checkbox at login time you would be asked for your username and password again. This has been corrected. We have now implemented the P3P standard that IE6 complies with to further assure your privacy and describe how data we collect from you is used. You can review this policy now (in IE6) by going to the View menu, selecting Privacy Report..., clicking on http://www.backflip.com/status.html and clicking on the Summary button.

[ January 8, 2002 - noon PST ]

Good news IE6 users, we now support IE6 favorites import.

[ December 26, 2001 - noon PST ]

We've made a slight change as to how the My Top Ten list gets generated. Links that you haven't clicked in 3 months or more will no longer appear in the list.

FOR YOUR INFORMATION... In September of 2001, Backflip was acquired by a small group of individuals dedicated to preserving the Backflip website. When Backflip's parent company realigned its business objectives, rather than shut it down or sell to a competitor, they gave us the opportunity to take over the service. We couldn't be more pleased.

Who are we? We are a small team of ex-Backflip employees, who, even though we no longer work there, still love the Backflip service and would like to see it survive (and thrive!).

Because we are no longer a big .com with seizable financial and staffing resources there have been some bumps. We are doing all we can to keep the service smooth but your patience is appreciated when problems do occur. We are all volunteers so we can't always react immediately but we are committed to keeping Backflip running.

Current Issues:

• There are issues with Netscape 6.x and 7.x.

Our #1 priority, however, is keeping the service running and free. This has always been our #1 goal since acquiring the service.

We have received amazing amounts of encouraging emails. We can't answer them all but we thank you! Please continue your support of Backflip.

If you have any comments or questions, please email us at support@backflip.com.