Page 1 of 3 123 LastLast
Results 1 to 10 of 41

Thread: KDE Almost Lost All Of Their Git Repositories

Hybrid View

  1. #1
    Join Date
    Jan 2007
    Posts
    15,652

    Default KDE Almost Lost All Of Their Git Repositories

    Phoronix: KDE Almost Lost All Of Their Git Repositories

    There was almost "The Great KDE Disaster Of 2013" when the KDE project almost lost all of their 1,500+ Git repositories...

    http://www.phoronix.com/vr.php?view=MTMzNTc

  2. #2
    Join Date
    Dec 2011
    Posts
    76

    Default Poor funkSTAR

    Quote Originally Posted by phoronix View Post
    Phoronix: KDE Almost Lost All Of Their Git Repositories

    There was almost "The Great KDE Disaster Of 2013" when the KDE project almost lost all of their 1,500+ Git repositories...

    http://www.phoronix.com/vr.php?view=MTMzNTc
    funkSTAR was THIS CLOSE to getting everything he ever wanted!

    Actually, while it would have been a huge PITA, there are enough out-of-repository copies of the source floating around so that I'm sure they could have rebuilt the repos to at least the last-released state of the code base. Still not good, but not lost forever either.

  3. #3

    Default

    Yeah, Linux and Open Source are shaky.

    In fact I had an ext4 corruption on a partition I mount RO daily and remount RW maybe once a week to write a file or two.

  4. #4
    Join Date
    Mar 2012
    Posts
    52

    Default

    Go suck horse cocks with your mother, birdie!

    DO NOT FEED THE TROLL PLEASE

  5. #5

    Default

    This is the silent corruption that ZFS was created to prevent. Unfortunately, KDE's servers were not using it.

  6. #6

    Default

    Quote Originally Posted by ryao View Post
    This is the silent corruption that ZFS was created to prevent. Unfortunately, KDE's servers were not using it.
    The real problem KDE sysadms overlooked is that mirroring is not backing up. ZFS wouldn't have saved them if their HDDs totally failed - ZFS is good at spotting hardware errors - that's true. But ZFS is not a magic bullet.

  7. #7
    Join Date
    Jun 2011
    Posts
    316

    Default

    Quote Originally Posted by ryao View Post
    This is the silent corruption that ZFS was created to prevent. Unfortunately, KDE's servers were not using it.
    Read the KDE blogs.. KDE claims there is too much churn in the data on a git server with 1500 repositories, so you wouldn't have been able to get a workable ZFS snapshot at any point without taking the server or one of the mirrors offline. Since everything was mirroring from the same server, they pretty much had a central point of failure and they also didn't want to take it offline because they wanted 24/7 availability.

    As far as ZFS stopping the corruption altogether, you don't know that. The corruption could have been caused by a software glitch in VM software, git, or the VM guest OS and wouldn't be detected at all by ZFS since the data could have been bit-perfect but still invalid. It might have been data corruption, but perhaps not filesystem level corruption.

    So while ZFS may or may not have helped here, it's still no substitute for backups.




    Quote Originally Posted by ryao View Post
    Unfortunately, proper backups would not have helped as much as one would think without a way to know whether or not the data was bad prior to doing them. It is not clear when the repositories became corrupted, although the blog post suggests that fsck was the trigger.
    Agreed. Although KDE cited their problem being that backing up git servers isn't properly documented.

    There is too much churn in the data which makes it difficult, if not impossible to take a snapshot or backup at any time on their main server and be guranteed it's consistent.

    Typically you would take the server offline for maintenance in these kinds of situations, but KDE believed they had a better solution for backups by using the --mirror.

    Unfortunately it appears git --mirror is a lot like rsync --delete where if things get corrupted at the source, that corruption will be mirrored on sync. KDE claims this isn't properly documented behavior of --mirror, and they're probably right.

    Yes, KDE had tarballs, but they didn't have tarballs for backup purposes.. The Tarballs were individual tarballs of each respository and were meant to make it easier to download repository contents. The tarballs did not contain everything to restore all content that was on the git server. git bundle to suffer from the same problem that it just doesn't copy everything. The only way to copy everything and keep the main server online 24/7, AFAIK, is with --mirror.


    So KDE was right that git --mirror was probably the best way to go if they didn't want to take the server down for maint.. What they failed to do was:
    1. Take a mirror server offline and run a git fsck to make sure that what it mirrored was sane.

    1a. AFAIK, all the KDE's mirrors were always online and KDE claims that since the servers were all online, the data churn on them made it impossible to get a sane snapshot for tarballs, ZFS, or any other backup purposes. Thus, it should have been obvious to KDE, IMO, that they need to take the main server offline or one of the mirrors offline to do proper backups. Since everything is mirroring off of the main server, and git fsck is known to take forever on 1500 repositories (as KDE claims) it makes sense to take one of the mirrors down and use the mirror to do the backups.


    1b. KDE claims the corruption might have started months ago. If they ran git fsck on any of the mirrors after doing a mirror and taking it offline, they would have discovered the problem months ago.

    2. After running git fsck on the mirror, they should then take a snapshot /backup of it. They'd be guaranteed that all their snapshots / backups were sane because the mirror is offline so people aren't pushing to it and it's also git fsck clean.

    3. Keep backups that date back for a solid year, I'm shocked that people still don't do this, but it should be obvious.. Also, make sure those backups are stored at least a couple hundred miles away from your nearest online server because I've known plenty of companies that spend a lot of money doing backups to tape, only to have them wiped out in the same natural disaster.
    Last edited by Sidicas; 03-25-2013 at 10:30 PM.

  8. #8
    Join Date
    Nov 2012
    Posts
    639

    Default

    Quote Originally Posted by birdie View Post
    Yeah, Linux and Open Source are shaky.

    In fact I had an ext4 corruption on a partition I mount RO daily and remount RW maybe once a week to write a file or two.
    This is bunch of some morons bullshit. Furthermore, Linux is the most stable and reliable OS (that matters, I don't care about some casio watch "operating systems"). Winblows just blows up. Linux will have btrfs and Linux can use ZFS as well, while you can only dream about them on Windows.
    Last edited by Pawlerson; 03-25-2013 at 07:10 PM.

  9. #9

    Default

    Quote Originally Posted by Pawlerson View Post
    This is bunch of some morons bullshit. Furthermore, Linux is the most stable and reliable OS (that matters, I don't care about some casio watch "operating systems"). Winblows just blows up.
    Solaris is more reliable. This would not have happened had the master mirror been running a recent installation of Solaris.

  10. #10
    Join Date
    Nov 2012
    Posts
    639

    Default

    Quote Originally Posted by ryao View Post
    Solaris is more reliable. This would not have happened had the master mirror been running a recent installation of Solaris.
    You've got to be kidding me. It's a dead cow. Tell me why nearly nobody is using it? Btw. what are you doing for Gentoo? Last time you were trolling for bsd and now you're trolling for slowlaris. Get the facts till you write another bullshit next time:

    http://unixetc.co.uk/2012/01/22/zfs-...nlinked-files/
    https://forums.oracle.com/forums/thr...art=0&tstart=0 unreliable slowlaris and zfs (somebody should tell KDE devs to not use it).
    Last edited by Pawlerson; 03-25-2013 at 07:27 PM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •