TalkBass is Down

[Mossman]

I don't know how many people here besides @dabbler and myself who are members, but the forum has been down for three days.

The official report was that there was a fire at their server farm, but there are some conspiracy theories brewing on TDPRI that they got hacked, and the data is being held for ransom, since a commercial forum of that size (and profitability) would surely have a mirror site, or some kind of backup. Three days of down-time is pretty unacceptable, and it seems highly unlikely that they had absolutely no contingency plan in place to deal with a server catastrophe.

[Chocol8]

Did you look over on MOMO to see if a dickhead prior owner sold out to an incompetent asshat? I have heard that can happen to a forum.

[Mossman]

Did you look over on MOMO to see if a dickhead prior owner sold out to an incompetent asshat? I have heard that can happen to a forum.

Heheh... yeah, it's hard not to think of MOMO, but I don't think theyd have anything to offer the owners of TalkBass. That place is a juggernaut. I imagine that forum makes money hand over fist. Teaming up with a start-up like MOMO would be a giant step backwards.

[uwmcscott]

If it's been down for 3 days I would guess that there is not a huge amount of money involved, looks like the site owner is doing a lot on his own based his Facebook Posts. The fact that their datacenter had no offsite mirror or backup is pretty sketchy as well. In any case, hope they get it back up soon.

[nomadh]

I like talkbass. They has some stupendous SX bass lover threads over there. Be terrible if this was a deathblow or this toasted their historic data. We keep thinking this digital data will last forever but it really like much of modern building. Some day there will be only the oldest stone tablets and the oldest pyramids or great wall to see that we were ever here. Old tech is the only lasting tech

[Mossman]

If it's been down for 3 days I would guess that there is not a huge amount of money involved, looks like the site owner is doing a lot on his own based his Facebook Posts. The fact that their datacenter had no offsite mirror or backup is pretty sketchy as well. In any case, hope they get it back up soon.

Hmmm.. I was under the impression that TalkBass was owned by some corporate media conglomerate, but I can't really say why I thought that... I may have been thinking about another forum(s). I'm not saying I believe the ransom theory, I'm just saying that's what some people at TDPRI have opined (mostly IT guys). I haven't been back on there since I first read the thread... I'm gonna go see if any new information has come to light.

[edit]

Nothing much new in that thread... they're all making bass player jokes now.

[Mossman]

I like talkbass. They has some stupendous SX bass lover threads over there. Be terrible if this was a deathblow or this toasted their historic data. We keep thinking this digital data will last forever but it really like much of modern building. Some day there will be only the oldest stone tablets and the oldest pyramids or great wall to see that we were ever here. Old tech is the only lasting tech

That's because old tech was better than what we've got.

Why would a technologically advanced civilization build with stone?

Because they could.

[uwmcscott]

That's because old tech was better than what we've got.

Old or new, it appears the real problem was the strategy of the hosting company in this case - no offsite backups or mirrored instances of any kind. They had all their eggs in one basket so to speak.

[Mossman]

That's because old tech was better than what we've got.

Old or new, it appears the real problem was the strategy of the hosting company in this case - no offsite backups or mirrored instances of any kind. They had all their eggs in one basket so to speak.

Well, that comment was tangential... It wasn't meant as a critique on the current state of server technology.

This is on TalkBass' front page now. The last time I went there, I got a technical message from Cloudflare:


"TalkBass is currently offline due to a catastrophic fire and subsequent fire-suppressant water damage at the datacenter where our servers are hosted.
We have been offline since the evening of Sunday April 4th. Power restoration has been slower than anticipated, as has any info on the possible damage to our servers.

While this is deeply frustrating, please know we are doing everything we can to get TalkBass back online as soon as possible.
Offsite backups were automatically maintained up until the incident, so if deemed the best course we will spin up new servers and start to rebuild from backups.

Thank you for your patience during this time. We've been online for over 20 years, and we will get through this!

Happy bassing to all, I so look forward to seeing you soon.

Paul Determan (Founder and owner)"

[dabbler]

I like talkbass. They has some stupendous SX bass lover threads over there. Be terrible if this was a deathblow or this toasted their historic data. We keep thinking this digital data will last forever but it really like much of modern building. Some day there will be only the oldest stone tablets and the oldest pyramids or great wall to see that we were ever here. Old tech is the only lasting tech

That's because old tech was better than what we've got.

Why would a technologically advanced civilization build with stone?

Because they could.

So, in the digital age, a piece of information REALLY only lasts as long as the last person who remembers it.

[glasshand]

I don't know how many people here besides @dabbler and myself who are members, but the forum has been down for three days.

The official report was that there was a fire at their server farm, but there are some conspiracy theories brewing on TDPRI that they got hacked, and the data is being held for ransom, since a commercial forum of that size (and profitability) would surely have a mirror site, or some kind of backup. Three days of down-time is pretty unacceptable, and it seems highly unlikely that they had absolutely no contingency plan in place to deal with a server catastrophe.

I'm a member too; I'm LostJohnny over there (since glasshand was taken).

As for "they would surely have a mirror site" - well, you would like to think that, but... It's slightly startling to me that TalkBass is apparently "one server in a datacenter somewhere", but if it's basically a hobbyist site or a carelessly maintained commercial site, well, that's what they do. I'm still startled that the site is completely offline, because the backups should have been offsite. Spin up a cloud server, throw those backups on it, update your DNS records, and get on with life. Really, the entire thing should have been in a multi-regional cloud service already, because it is not difficult or expensive to do that these days, but...

I do this for a living, which influences my views on things like this, for better or worse: I'm Senior Manager of DevOps at a large media firm you have heard of, and I have been dealing with this sort of issue every single day for 25 years. My company is now 100% cloud-based, and I'm still a little startled, although maybe I shouldn't be, at how many companies aren't 100% cloud-based.

[uwmcscott]

As for "they would surely have a mirror site" - well, you would like to think that, but... It's slightly startling to me that TalkBass is apparently "one server in a datacenter somewhere", but if it's basically a hobbyist site or a carelessly maintained commercial site, well, that's what they do. I'm still startled that the site is completely offline, because the backups should have been offsite. Spin up a cloud server, throw those backups on it, update your DNS records, and get on with life. Really, the entire thing should have been in a multi-regional cloud service already, because it is not difficult or expensive to do that these days, but...

I do this for a living, which influences my views on things like this, for better or worse: I'm Senior Manager of DevOps at a large media firm you have heard of, and I have been dealing with this sort of issue every single day for 25 years. My company is now 100% cloud-based, and I'm still a little startled, although maybe I shouldn't be, at how many companies aren't 100% cloud-based.

It's also certainly possible ( probable ) that they had no idea the host was this vulnerable. Kind of like car insurance, you really never know how good it is until you have an accident and most people don't.

There are very likely many companies in this boat right now that have no idea whether or not their disaster recovery plan would work, and some don't even have a plan. Not just web hosting companies either. I have transitioned careers twice ( also in IT/IS generally ) and it's pretty amazing how many companies have very little idea of what is going on in their IT sphere. I interviewed for a IT director job at a local transportation company a couple of months back and most of the questions I asked were met with a bunch of head turning and few answers. They had a main "IT guy" who knew everything about everything and left, which is common too.

It's actually surprising stuff works as well as it does

[fatjack]

Really surprised there was no offsite backup. As a linux admin I knew used to preach backup backup backup. ! onsite 1 crosstown and one halfway cross country. That way at least one is safe at all times. Accidentally wiping an active server while doing maintenance as admin cured him of carelessness.

[honyock]

I like talkbass. They has some stupendous SX bass lover threads over there. Be terrible if this was a deathblow or this toasted their historic data. We keep thinking this digital data will last forever but it really like much of modern building. Some day there will be only the oldest stone tablets and the oldest pyramids or great wall to see that we were ever here. Old tech is the only lasting tech

That's because old tech was better than what we've got.

Why would a technologically advanced civilization build with stone?

Because they could.

Because they had slave labor and thought their civilization was destined to last forever...

[Mossman]

Because they had slave labor and thought their civilization was destined to last forever...

Eh... The slave labor theory is specious, especially considering the time-frames in which the pyramids were supposedly built. Also, the Egyptians weren't the only civilization that have produced structures and artifacts of unexplainable manufacture. Nothing we can build with our technology even comes close.

[Mossman]

Here's the latest news:

[dabbler]

Here's the latest news:

Thanks for posting this Mossman. Talkbass and this place are my most-visited sites.

[glasshand]

It's actually surprising stuff works as well as it does

True that, my friend. The more I see, the more I feel that the entire world is secretly held together with chewing gum and bits of string behind the scenes.

[uwmcscott]

It's actually surprising stuff works as well as it does

True that, my friend. The more I see, the more I feel that the entire world is secretly held together with chewing gum and bits of string behind the scenes.

Way back in the day, I worked for an IT Consulting company and out of the blue we got a call from a customer we had been trying get in with for a long time. They had an immediate need and wondered if we could be there same day to help, so of course I jumped on it. I did a little triage to find out exactly what they needed, and it was related to restoring some accidentally deleted important files for a VIP from backup. Their IT director had recently departed and had no idea what to do. I asked what kind of backup software they were using ( remember...this was back in the day...DLT drives and such ) and rounded up one of our engineers who had experience.

We drove over together, and when we got there they took us to the "IT room". They did have the admin passwords available but really had no idea about anything. We asked which one was the backup server and they had no idea. We then asked how they knew what kind of backup software they had and they pulled a box off the shelf of Seagate Backup Exec. So we found a server with a tape drive and they did actually have a stack of dated tapes that someone had been rotating in and out of the server. However, we found that there was not a single instance of any backup software loaded on a single server, so they had basically been feeding tapes into a tape drive for almost a month and not a single file was ever being backed up . This was a local office of a large nationwide company and this backup would( should ) have contained basically everything about their local business.

That's just one story, I'm sure any of you other folks here that work in IT would have many similar ones to share.

[glasshand]

That's just one story, I'm sure any of you other folks here that work in IT would have many similar ones to share.

Not my story, but one I heard: a guy was contracted in to help with an IT issue at a local police office. This is quite a long time ago, so their computer system was basically just one PC. He asked if they had backups, and was pleasantly surprised when they said yes, they had backup software and took full backups every day. He congratulated them on their unusually good practices, and then he asked for the latest full backup, and was a little startled when they pulled out ONE floppy disc. Some investigation revealed that every time when the backup software said during its backup process, "Insert next floppy disc and hit ENTER to continue", the cop in question had just been hitting enter without putting a new disc in...

Which goes to show you why you need to practice your disaster recovery operations for real.

But in another story I've heard, more specifically datacenter-related, a datacenter regularly practiced cutting over from utility power to their generator every month, checking to make sure they had fuel and the generator would run and power the datacenter. One day, they lost power for real, and their automatic systems kicked in and cut smoothly over to generator power just like they were supposed to. Then ten minutes later, their generator coughed and died, and the whole place went dark. It turned out, IIRC, that their fuel filter had never been changed, but as they'd never tried running the generator for longer than a few minutes at a time they never knew the thing wouldn't run longer than that...

[Houblues]

[/quote]

True that, my friend. The more I see, the more I feel that the entire world is secretly held together with chewing gum and bits of string behind the scenes.
[/quote]

And low to mid level workers running around under the corporate radar valiantly trying to prop up deficient systems and processes.

[Mossman]

Thanks for posting this Mossman. Talkbass and this place are my most-visited sites.

You're welcome... I knew that it would be of interest at least to you... But it looks like we have a few other clandestine TalkBass members here too.

[Mossman]

Well, y'all probably know this by now, but TalkBass went back online yesterday with minimal losses...

https://www.talkbass.com/threads/welcom ... t-25145559

[tobijohn]

Eh... The slave labor theory is specious, especially considering the time-frames in which the pyramids were supposedly built. Also, the Egyptians weren't the only civilization that have produced structures and artifacts of unexplainable manufacture. Nothing we can build with our technology even comes close.

Not germane to the OP but it is to your comment. Coincidently, I stumbled across this guy a a month or so ago and was completely intrigued by his theories. You might find it interesting:

[UrenragK]

I worked with a customer (a large financial institution) that had their "Mortgage system" go missing,

I went to the head honcho's desk for him to show me the Mortgage System. He opened up file explorer, navigated to the server, and into a folder, which was empty and said "It should be there"

Turned out it was an Excel spreadsheet that up to 40 staff updated simultaneously, and NFS lost it's mind and put it in the "I don't know what the f*** happened but loads of changes were all made at the same time" (Reconciliation failures) folder!