FreeDesktop.org GitLab Down Due To Drive Failures

[phoronix]

Phoronix: FreeDesktop.org GitLab Down Due To Drive Failures

Centralized development around Mesa, the X.Org Server, and dozens of other open-source projects is at a stand-still this weekend due to FreeDesktop.org GitLab crashing with the entire service down...

https://www.phoronix.com/scan.php?page=news_item&px=FreeDesktop-GitLab-2022-Crash

[tildearrow]

Personally I consider SSDs as volatile storage devices as they fail immediately (HDDs on the other hand usually fail gradually).
Let's hope they have been making periodic backups on HDD...

[lumks]

as volatile storage devices as they fail immediately

Usually SSDs go into a read-only mode for some time before they completely fail. That said, this is only true if they die because of age, not if there is some other problem.

[waxhead]

Not quite on topic, but cfor some reason this reminds me about the good old saying "Unix systems are so reliable they never have to be rebooted". There is one major drawback with something like this and that is once you got everything configured and running nobody knows if it will come up again properly after a reboot So let's hope this is JUST a matter of restoring from (tested) backups.

It also somehow this also reminds me a bit of kernel.org breach back in 2011 where certain services (boot.kernel.org) never came back up afterwards. Let's hope everything gets back up one way or another.

[aufkrawall]

Let's hope they have been making periodic backups on HDD...

I don't think the discussion should be SSDs vs. HDDs, but how it could happen in the first place, i.e. what strategy wasn't pursued that would have prevented this failure.

[Quackdoc]

interesting that 2 drives were enough to take down the services. seems like a misconfiguration if that was all that we needed

[cl333r]

Personally I consider SSDs as volatile storage devices as they fail immediately (HDDs on the other hand usually fail gradually).
Let's hope they have been making periodic backups on HDD...

I prefer HDDs because they make noise so it's easier to find the server in the server room.
Green-Man-Cannabis-photo-4.jpg

[Ericg]

interesting that 2 drives were enough to take down the services. seems like a misconfiguration if that was all that we needed

I'm not familiar with how their Gitlab instance is configured at the OS level, so consider this pure speculation, but if its a single server then its pretty common for servers to be configured with raid1 on the OS drive. If both SSDs were from the same batch then its very possible for them to suffer similar issues / age at the same rate and then die at the same time. This would imply that the freedesktop.org Gitlab isn't isnt running in any form of high availability or clustered mode (a quick check of the docs says that it does support such a configuration), which is a major issue but likely a decision that was made for cost reasons.

Again, pure conjecture but its a story I've seen play out all too often.

[Paradigm Shifter]

Usually SSDs go into a read-only mode for some time before they completely fail. That said, this is only true if they die because of age, not if there is some other problem.

They should, you're right. Sadly, that has not been my experience. The SSDs I've had die on me (both personally and at work) have just gone. There one second and gone the next. Not a specific manufacturer, either; Crucial, Sandisk, OCZ, a couple of never-heard-of-them-before super-cheap 128GB drives which I bought mostly for amusement and curiosity. Not specific to a model; the first SSD I had which died was a 960GB Sandisk Ultra 2 (which was apparently notorious for dying) but I've got three more of them bought at the same time and they're all still working more than 5 years later.

As with all data storage... backup if you care about it.

[zxy_thf]

It's a surprise to me that it is running on bare hard drives instead of Ceph or other cloud storage solution.