Critical Error: The Big Crash of 2023

It’s Tuesday, May 16th, 6:57AM, and it’s another beautiful day in the boodle home office. The birds are chirping, the sun is out, I’m having our morning breakfast. 6:58AM rolls around quicker than I could take a sip of my morning coffee, and I hear: “ching!”. I expect the usual @channel slack message from someone on the east coast. But instead I look down to see the words “@meredith website is down”.  

Take That boodle-ful Day and Turn it Sideways

Quickly we begin investigating, and sure enough we see this: critical error. 

But, it’s ok. We have been preparing for this day. That is why we had a staging site, backups, and all the things you need to bring a site back to life in minutes. We were able to restore boodle.ai and our staging site while only losing a day’s worth of work. A bummer, but it could be way worse. We quickly rebuilt the lost pages and moved on with our day full of meetings, creating, and telling everyone we could about our exciting new product, boodleGPT.

6PM came around. I had shut down for the day and was preparing dinner. All was right in the world. Then, like a lightning flash, it was 6:07 PM. Now, 6:07 PM may not be of any significance to you, but that is the time that everything really changed for boodleAI. 

Let’s Just Go Ahead and Turn This Day Upside-Down. 

“Ching!” Slack again. Except this time, the message read “@meredith hate to end the day the way I started.” and it included a link to the website. 

I’m pretty sure you can read into that just about as well as I did… 

That’s right. boodle.ai crashed AGAIN. Well, at least we already did this dance and know just how to restore the site back to working order, right? Wrong. 

We tried to restore from backup. Failed.
We tried to restore from staging. Failed. 
We tried to restore from the staging backup. Failed. 

Which led to the most dreaded last option for any Gen X or Gen Z-er. Calling Support. 

That call that we thought would last us a few minutes to restore our site ended up taking four looonnngggg hours of troubleshooting. Could it be a plugin conflict? Could it be a software update? Could it be a hosting issue? Or, maybe, could it be a history of 5 long years of poor website management finally crashing down on us? Either way, with each restoration attempt, the site just broke piece by piece. 

First, the back end of the site broke, but the front end was visible. 
Then, the front end broke, and the back end was visible. 
And eventually, the front end, back end, and everything in between just died.

The call quickly came to a close, and the answer was found… 

That answer: There’s no telling what caused the crash, but we are most certainly dead in the water. 

What Doesn’t Kill You Makes You Stronger

It’s 10:30 PM. And in true boodle fashion, we don’t take no for an answer. We quickly go into recovery mode. We create a simple landing page. No frills. No explanation. Just simply: one half-sentence and a form.

Now midnight, and off to get a couple hours of sleep. 

The Phoenix Rebirth

Wednesday morning came, and waitlist sign ups, well, they had exploded. We were getting sign ups faster than we had in days, and that led us to realize less really is more. 

We quickly spent the next 24 hours rebuilding the website in a way that we believe really evolved the experience. This simple homepage became the entry point for new and interested people to join our waitlist, subscribe to our AI newsletter, and enter into a behind the scenes experience that really is… well, amazing. 

Was the website crashing and losing 5 years of content a bummer? Absolutely. But truly, we are at a better place than we have arguably ever been. We were given a clean start to build what we wanted from the ground up. And we did so with determination and purpose. 

Some Advice From the Survivors

Let us be your warning sign. Don’t overload your website. Have a backup to your backup’s backup. And create a simple, yet compelling, experience to elevate your viewers interest in what you bring to the table. To prepare, or even survive a website crash, here ere are a few tips to keep in your back pocket: 

• If you don’t do anything else, think about this. If your website crashed today, what are the 5 things you know you will need? Whatever those 5 things are, create a separate BREAK IN CASE OF EMERGENCY (BICOM) website that you will always have on hand. That way, if you end up in a pickle like we did, you have an easy way to get back up and running quickly without losing too much sleep. 
• Create a redirect for any 404 pages so that it will automatically direct people to your home page. We lost many pages that were out in the world and are currently being indexed by Google. However, this universal redirect is our saving grace as it’s the only way visitors will reach an operable website from any of those links.
• Backup your site every day and often. While our scenario is a bit of a wild one, the first time our site crashed, our backups saved us. Make sure that you are backing up frequently to keep every element of your site up-to-date. (We would go as far to say you should even back up your BICOM and staging sites as well.)
• Lastly, keep it simple. We have all heard this before, but it truly is one lesson we had to learn the hard way. Simplicity gave us new life for our waitlist, and it will likely do the same for you.

Now, boodle.ai is back up and running. We’d argue – better than ever before. So, go check it out. Experience our creation, and let us know what you think. 

While you’re at it, join our waitlist and AI newsletter which gives the latest updates on all things AI. We’re excited to have you along for the ride.