Page 22 of 24 FirstFirst ... 122021222324 LastLast
Results 211 to 220 of 238

Thread: Benchmarking ZFS On FreeBSD vs. EXT4 & Btrfs On Linux

  1. #211
    Join Date
    Nov 2008
    Posts
    418

    Default

    This Linux SGI server with 4096 cores, is not a true SMP server. The biggest SMP servers today have ~64 cpus. Everything bigger than 64 cpus, are basically a cluster with horrible latency, making them unusable for SMP workloads.

    IBM and Oracle and HP are fighting fiercely trying to get the best world record benchmark, for instance in TPC-C. The biggest mature Unix Enterprise servers these guys have, are something like 32-64 cpus. The biggest z10 IBM mainframe have 64 cpus. The biggest z196 IBM Mainframe have 24 cpus.

    Why dont they just insert 128cpus? Or even 256cpus? Or even 16.384 cpus? Answer: no, these Mainframe/Unix servers are SMP servers. The biggest SMP servers have ~64 cpus today. An SMP server, is basically a single fat huge server weighing tons and costing many millions of USD. We talk about scaling vertically, "scale up".

    If we talk about HPC servers, horizontal scaling, "scale out", then we talk about clusters. Anything bigger than 64 cpus, are basically a cluster. A bunch of PCs sitting on a fast switch. Look at the benchmarks for the SGI Altix server; very impressive results indeed. But, all these workloads are cluster benchmarks. Not SMP benchmarks. If you build a cluster, why stop at 4096 cores? You can just change the number "4096" to "16.384" and recompile, and then you can brag about "Linux scales to 16.384 cores, which is far more than the biggest IBM servers with 64 cpus".

    Linux does very well in HPC servers, and scales to many many thousands of cpus. But on SMP servers, Linux scales very bad, maybe ~8 cpus.

    The biggest Linux SMP server, should be an normal 8-socket x86 server. I dont know of any 16-socket x86 server. You can populate that with 8 core Nehalem cpus, getting 64 cores. Thus, the biggest Linux SMP server should be the normal 8 cpu x86 servers that you can purchase today. For instance Oracle X4800 8-socket server, which supports 64 cores, and 1TB RAM. If you want a 16-socket x86 server, then you need special made chip sets, etc. It will be very expensive. And if you want 64 socket cpus, then it will be VERY expensive, costing tens of millions of USD. To scale well on SMP servers, is very difficult to do well. Very difficult. For instance, IBM has tried to do this for many decades, and now scales to 64 cpus.



    Here is a new big Linux server, it scales to up to 8.192 cores, using a single Linux image. The solution is called vSMP.
    http://www.theregister.co.uk/2011/09..._amd_opterons/
    Depending on the cores per chip and the generation you use, you can have from 2,048 to 8,192 cores in a single image.
    ...
    The vSMP hypervisor that glues systems together is not for every workload, but on workloads where there is a lot of message passing between server nodes – financial modeling, supercomputing, data analytics, and similar parallel workloads. Shai Fultheim, the company's founder and chief executive officer, says ScaleMP has over 300 customers now. "We focused on HPC as the low-hanging fruit," Fultheim tells El Reg,
    ...
    vSMP Foundation for Cluster is used to take multiple server images and plunk them on a single server image running one copy of a Linux operating system; you use vSMP and that operating system instead of a cluster manager to run workloads.
    Thus it is a cluster. As someone explained:

    "I tried running a nicely parallel shared memory workload (75% efficiency on 24 cores in a 4 socket opteron box) on a 64 core ScaleMP box with 8 2-socket boards linked by infiniband. Result: horrible. It might look like a shared memory, but access to off-board bits has huge latency."





    Regarding ZFS, it is built for server usage. With many disks. And built to scale well up to many disks. BTRFS lags behind when you start to use many disks, it does not scale as well as ZFS. Benchmarks proves this.
    Last edited by kebabbert; 11-08-2011 at 04:20 AM.

  2. #212

    Default

    Quote Originally Posted by kebabbert View Post
    This Linux SGI server with 4096 cores, is not a true SMP server. The biggest SMP servers today have ~64 cpus. Everything bigger than 64 cpus, are basically a cluster with horrible latency, making them unusable for SMP workloads.

    IBM and Oracle and HP are fighting fiercely trying to get the best world record benchmark, for instance in TPC-C. The biggest mature Unix Enterprise servers these guys have, are something like 32-64 cpus. The biggest z10 IBM mainframe have 64 cpus. The biggest z196 IBM Mainframe have 24 cpus.

    Why dont they just insert 128cpus? Or even 256cpus? Or even 16.384 cpus? Answer: no, these Mainframe/Unix servers are SMP servers. The biggest SMP servers have ~64 cpus today. An SMP server, is basically a single fat huge server weighing tons and costing many millions of USD. We talk about scaling vertically, "scale up".

    If we talk about HPC servers, horizontal scaling, "scale out", then we talk about clusters. Anything bigger than 64 cpus, are basically a cluster. A bunch of PCs sitting on a fast switch. Look at the benchmarks for the SGI Altix server; very impressive results indeed. But, all these workloads are cluster benchmarks. Not SMP benchmarks. If you build a cluster, why stop at 4096 cores? You can just change the number "4096" to "16.384" and recompile, and then you can brag about "Linux scales to 16.384 cores, which is far more than the biggest IBM servers with 64 cpus".

    Linux does very well in HPC servers, and scales to many many thousands of cpus. But on SMP servers, Linux scales very bad, maybe ~8 cpus.

    The biggest Linux SMP server, should be an normal 8-socket x86 server. I dont know of any 16-socket x86 server. You can populate that with 8 core Nehalem cpus, getting 64 cores. Thus, the biggest Linux SMP server should be the normal 8 cpu x86 servers that you can purchase today. For instance Oracle X4800 8-socket server, which supports 64 cores, and 1TB RAM. If you want a 16-socket x86 server, then you need special made chip sets, etc. It will be very expensive. And if you want 64 socket cpus, then it will be VERY expensive, costing tens of millions of USD. To scale well on SMP servers, is very difficult to do well. Very difficult. For instance, IBM has tried to do this for many decades, and now scales to 64 cpus.



    Here is a new big Linux server, it scales to up to 8.192 cores, using a single Linux image. The solution is called vSMP.
    http://www.theregister.co.uk/2011/09..._amd_opterons/

    Thus it is a cluster. As someone explained:

    "I tried running a nicely parallel shared memory workload (75% efficiency on 24 cores in a 4 socket opteron box) on a 64 core ScaleMP box with 8 2-socket boards linked by infiniband. Result: horrible. It might look like a shared memory, but access to off-board bits has huge latency."





    Regarding ZFS, it is built for server usage. With many disks. And built to scale well up to many disks. BTRFS lags behind when you start to use many disks, it does not scale as well as ZFS. Benchmarks proves this.
    It's amazing how someone can post such crap. By saying Linux scales well up to 8 cores you're an idiot. You saw it scales great up to 64 physical CPUs, but you're a lier. Btw. why slowlaris's dead in HPC? About your FUD regarding btrfs - non stable version of btrfs was uses which is obvious.

  3. #213

    Default

    Quote Originally Posted by kebabbert View Post
    This Linux SGI server with 4096 cores, is not a true SMP server. The biggest SMP servers today have ~64 cpus. Everything bigger than 64 cpus, are basically a cluster with horrible latency, making them unusable for SMP workloads.
    So, all of this slowlaris machines with 100+ CPUs aren't real SMP servers? Then what this talk is about, your idiocy or something?

  4. #214

    Default

    It's very hard to imagine slowlaris scaling well while it died in HPC, it's being marginalized in Enterprise, because it's legacy system with binaries slower at least 30% than Linux' ones. Oracle plans to use btrfs as a default file system in their distribution and this means they want to kill slowlaris completely (sooner as one would expect).

  5. #215
    Join Date
    Nov 2008
    Posts
    418

    Default

    Quote Originally Posted by kraftman View Post
    It's amazing how someone can post such crap. By saying Linux scales well up to 8 cores you're an idiot. You saw it scales great up to 64 physical CPUs, but you're a lier. Btw. why slowlaris's dead in HPC? About your FUD regarding btrfs - non stable version of btrfs was uses which is obvious.
    I am not saying that Linux scales well up to 8 cores, read my post again. I am saying that Linux maybe scales well up to 8 CPUS - with 8 cores each, gives 64 cores.

    The reason I say this, is because official SAP benchmarks shows that Solaris is faster than Linux on server with 48 cores. The Linux server used higher clocked CPUs, and used faster DRAM chips - but still the slower Solaris server scored higher. Also, the official benchmarks showed that Linux 48 core SMP server, had a cpu utilization of 87% which is low. Solaris SMP server had 99%. That is the reason that Solaris scored higher - it scaled better. Clearly, Linux had problems scaling on 48 cores, only 87% cpu utilization which is bad. That is the reason that 64 cores may be an upper limit for Linux, hence 8 cpus.



    So, all of this slowlaris machines with 100+ CPUs aren't real SMP servers? Then what this talk is about, your idiocy or something?
    What? How can you say that Solaris scales to 100+ cpus on SMP servers? You denied that fiercely. I explained that Solaris scales beyond 64 cpus, but you denied it:
    http://phoronix.com/forums/showthrea...198#post143198
    "Oh, dumb. You FUD and many people FUD. FUD happens. Idiot, the fact is slowlaris scales only up to 64CPUs and Linux scales up to 4096CPUs. Dot. I don't care if you believe or not. You're an idiot, so you're free to not believe in facts."

    So, are you now agreeing that Solaris scales beyond 64 cpus in SMP servers?



    It's very hard to imagine slowlaris scaling well while it died in HPC, it's being marginalized in Enterprise, because it's legacy system with binaries slower at least 30% than Linux' ones. Oracle plans to use btrfs as a default file system in their distribution and this means they want to kill slowlaris completely (sooner as one would expect).
    I have showed you links where Larry praises Solaris as being the best OS, and where he says that Linux is for lowend and Solaris for highend. And Oracle is increasing development and have more Solaris developers than Sun ever had, I have shown these links too. I find it hard to believe that Oracle is going to kill Solaris? Do you have links or any support? Or is this just wishful thinking from you?

    Regarding HPC work loads, it is easy to do. But you need to heavily modify the kernel, strip out everything. Just a highly specialized kernel, doing only one thing: number crunching. To strip Linux is easy, as it is a naive and structurally simple kernel. On the other hand, Solaris is a mature, highly complex kernel, using lot of macros to generate code and lot of weird stuff. It would be difficult to understand Solaris kernel to modify it. And, Solaris was proprietary, so no one wanted to pay lot of money for Solaris licenses. Also, Oracle is not interested in the HPC market, as Oracle have said it is not profitable. It is too specialized and few customers.

    On the other hand, IBM thinks HPC world records gives good PR, so IBM is willing to bet on HPC. Oracle mostly cares about profit, and walks away from low margin business. I have no doubt that if Solaris ever becomes low margin business, then Oracle will abandon Solaris and open source it and leave it be. Just as Oracle have done recently with OpenOffice which is not profitable, Oracle open sourced it and let OpenOffice go. If Oracle does not earn money, Oracle will just leave the product and try other products.

  6. #216

    Default

    I once said you're making just fool of yourself. You were giving me sap benchmark as a proof slowlaris scales better than Linux, but when I said you're a troll and idiot you didn't want to believe. I tried to explain sap benchmarks are irrelevant, but you didn't believe. I said I've have quite a lot of fun of your stupid argumentation and I've got something funny for you:

    http://centiq.co.uk/the-saps-benchmark.html

    Tell me now how it is to look like an idiot? The reason I say this is "official SAP benchmarks doesn't show that Solaris is faster than Linux". Btw. you don't believe slowlaris which is bloated by design and which has 30% slower binaries could be faster, do you?

  7. #217
    Join Date
    Nov 2008
    Posts
    418

    Default

    Quote Originally Posted by kraftman View Post
    I once said you're making just fool of yourself. You were giving me sap benchmark as a proof slowlaris scales better than Linux, but when I said you're a troll and idiot you didn't want to believe.
    You have many times said I am a Troll and Idiot, yes. Should I believe you on this? Why should I believe you on this? You confessed earlier that you have FUD. Whenever I write something, I speak true. I do not make things up nor lie nor FUD. I can always back up my links. You on the other hand, have several times written things that are not true, for instance when you called me ugly names for not believing you when you said Solaris only scales to 64 cpus, and I showed you links with 106 cpu Solarsi servers and you refused and denied that link. Who is trolling and FUDing?

    You also called Bonwick for FUDer when he said that Linux does not scale well. And Linux does not scale well on SMP servers. Everybody knows this. Bonwick spoke true. Show me a big SMP Linux server, go ahead. There are none on the market! Linux does not scale well on SMP servers. But on clusters, Linux scales very well, everybody knows this (including Bonwick).



    I tried to explain sap benchmarks are irrelevant, but you didn't believe.
    Yes, you explained a lot of things to me. You said things as SAP has a cooperation with Sun, and that is the reason that SAP benchmarks is faster for Solaris, punishing Linux. I posted a link and proved that SAP has cooperation with several Linux companies as well. Why does not SAP punish Solaris? So I dont agree with your explanations.



    I said I've have quite a lot of fun of your stupid argumentation
    I dont think my argumentation is stupid. I ask relevant questions "why should SAP punish Solaris when SAP also has cooperation with Linux companies?" - and to my questions you dont have any good answer. Only more strange explanations.



    and I've got something funny for you:
    http://centiq.co.uk/the-saps-benchmark.html
    Tell me now how it is to look like an idiot? The reason I say this is "official SAP benchmarks doesn't show that Solaris is faster than Linux".
    I have read this entire link you posted. Could you cite the text that is relevant to this discussion? I dont understand your point, can you explain your point?



    Btw. you don't believe slowlaris which is bloated by design and which has 30% slower binaries could be faster, do you?
    Well, benchmarks shows that Solaris scores higher than Linux, so yes, I do believe that Solaris is faster.

    You have many times said that Solaris is bloated and 30% slower. Can you show links that support this claim of yours? Normally, you never show links that support your strange claims, even though I ask. How can Solaris be faster on benchmarks, when the binaries are slower?

    Also, it is strange that FreeBSD runs Linux software faster than Linux itself, dont you think? It shows FreeBSD has better design and less bloated than Linux, dont you think? Even Linus T says that Linux is bloated.

  8. #218

    Default

    Quote Originally Posted by kebabbert View Post
    Well, benchmarks shows that Solaris scores higher than Linux, so yes, I do believe that Solaris is faster.

    You have many times said that Solaris is bloated and 30% slower. Can you show links that support this claim of yours? Normally, you never show links that support your strange claims, even though I ask. How can Solaris be faster on benchmarks, when the binaries are slower?
    Which benchmarks? SAP benchmarks doesn't show slowlaris is faster. When comes to Bonwick then yes, he was FUDing and everyone knows this. The answer to your question is simple - but now it's you who has to answer.

    http://download.sap.com/download.epd...AD2160BC51132A

    http://download.sap.com/download.epd...11DE75E0922A14

    We have 40cores 2,4Ghz each running SuSe Linuxe Enterprise Server and 48cores 2.6Ghz each running slowlaris. Linux which runs on much slower machine kicks slowlaris ass, why it's that Kebbabert? Have a nice day. I figured out some interesting things about you btw. Your a known person and one of the biggest trolls ever.

    Also, it is strange that FreeBSD runs Linux software faster than Linux itself, dont you think? It shows FreeBSD has better design and less bloated than Linux, dont you think? Even Linus T says that Linux is bloated.
    In which Kebbabert's dream? It just shows you''re just a troll. Freebsd is far more bloated than Linux. Unmodified Linux can run even on mobile devices while bsd can't. Linux's success in nearly every area proofs it's much better designed than slowlaris or bsd.

    Edit:

    I would forget:

    http://scalability.org/?p=388

    There are many more things that can be said about slowlaris. You can't even imagine.However, it's sad you trash the forum and I have to keep replying to troll like you. Keep in mind it doesn't give me any satisfaction.
    Last edited by kraftman; 11-13-2011 at 12:07 PM.

  9. #219
    Join Date
    Dec 2010
    Posts
    2

    Default

    Quote Originally Posted by kraftman View Post
    Which benchmarks? SAP benchmarks doesn't show slowlaris is faster. When comes to Bonwick then yes, he was FUDing and everyone knows this. The answer to your question is simple - but now it's you who has to answer.

    http://download.sap.com/download.epd...AD2160BC51132A

    http://download.sap.com/download.epd...11DE75E0922A14

    We have 40cores 2,4Ghz each running SuSe Linuxe Enterprise Server and 48cores 2.6Ghz each running slowlaris. Linux which runs on much slower machine kicks slowlaris ass, why it's that Kebbabert? Have a nice day. I figured out some interesting things about you btw. Your a known person and one of the biggest trolls ever.
    A small detail, 40cores / 80 threads / 30MB L3 vs 48cores / 48 thread / 6MB L3.

    Even Intel Xeon E7- 4850 @ 2.00GHz seems faster than Opteron 8435.
    http://www.cpubenchmark.net/cpu_look...50+%40+2.00GHz

    Quote Originally Posted by kraftman View Post
    In which Kebbabert's dream? It just shows you''re just a troll. Freebsd is far more bloated than Linux. Unmodified Linux can run even on mobile devices while bsd can't. Linux's success in nearly every area proofs it's much better designed than slowlaris or bsd.
    Linux is a kernel, FreeBSD is complete operating system which includes userland utilities, libc, compiler, etc. Just download Linux from kernel.org and see how usable it is compared to FreeBSD.

    Apples and oranges.

  10. #220
    Join Date
    Nov 2008
    Posts
    418

    Default

    Linux which runs on much slower machine kicks slowlaris ass, why it's that Kebbabert? Have a nice day.
    Now you are doing this again.

    I remember when you compared a new Intel core duo 2.4GHz Linux vs an old 800MHz SPARC Solaris and you said it proves that Linux is faster. I rejected that benchmark, but you thought it was a fair and good benchmark. Even if Linux were indeed 30% faster than Solaris, then 800MHz Linux would not be faster than 2.4GHz Solaris. So you are unfair.

    And this SAP benchmark that you show, the Linux server are using cpus that are 3x as fast as the Solaris server cpu. See below:




    I figured out some interesting things about you btw. Your a known person and one of the biggest trolls ever.
    Good for you. You know that I am not making things up, I just quote Linus Torvalds, Andrew Morton and other Linux kernel developers. Do you suggest they are trolls? I have always shown links to what I say. You on the other hand, have confessed you FUD sometimes.




    Quote Originally Posted by kraftman View Post
    Which benchmarks? SAP benchmarks doesn't show slowlaris is faster.
    http://download.sap.com/download.epd...AD2160BC51132A

    http://download.sap.com/download.epd...11DE75E0922A14

    We have 40cores 2,4Ghz each running SuSe Linuxe Enterprise Server and 48cores 2.6Ghz each running slowlaris.
    Now you are doing this again. You are comparing the fastest x86 cpu ever made, to a several years old cpu. You are comparing Intel 32nm Sandybridge Westmere-EX 10-core vs AMD 45nm Opteron 6-core.

    It turns out that the Intel Sandybridge Westmere-EX E7-4870 cpu, is roughly twice as fast as the AMD Opteron 8435 cpu.

    Here we see that the Westmere-EX cpu is 50% faster than an AMD 12-core cpu. See picture
    http://www.anandtech.com/print/4285

    And a 6-core cpu should be half as fast as a 12-core cpu. This means that Intel E7 is 3x as fast as the AMD 6-core cpu.

    Is this fair? You are comparing a Intel cpu that is 3x as fast than an AMD Istanbul cpu. You are comparing Intel 40 cores of the x86 fastest cpu ever made, to 48 cores of the old AMD Opteron.

    Only Linux fans thinks it is fair to compare a 3x faster server, to an old server. Great.




    Also, we see that this Linux Westmere-EX server have 97% cpu utilization which is better than the 87% earlier Linux result. Earlier, the Linux server had 87% cpu utilization, which is bad. But this Westmere-EX server have 97% cpu utilization. Why is that? Answer: The reason the new Linux SAP have better cpu utilization is because Linux uses fewer cpus

    Linux benchmarks:
    4cpus - 97% cpu utilization. This Westmere-EX benchmark
    6cpus - 87% cpu utilization. The earlier SAP benchmark where Linux used same cpus as Solaris (see below for link)

    The more cpus Linux uses, the worse the cpu utilization gets. In other words, Linux scales bad on SMP servers. If Linux was using 8 cpus, I suspect cpu utilzation would drop below 80%. And if using 24 cpus, the cpu utilization would maybe drop below 50%. Linux scales bad on SMP servers. But on clusters, Linux scales very good.

    For the record, Solaris had 99% cpu utilization on 6 cpus, because Solaris has run on 64 cpus and above, for decades.
    http://download.sap.com/download.epd...11DE75E0922A14




    On the other hand, let us compare hardware that is similar. Here we see that Linux uses AMD Opteron 8439 cpus 2.8 GHz
    http://download.sap.com/download.epd...FCA652F4AD1B4C

    which are faster than AMD Opteron 8435, 2.6GHz. And still Solaris is 23% faster.
    http://download.sap.com/download.epd...11DE75E0922A14

    The only way Linux can win, is if Linux is using hardware that is 3x as fast. Desperate, yes?

    There are many benchmarks I know of, which shows 32 cpu Solaris being faster than 4 cpu Linux, but I never post them. Because that is not fair. Such benchmarks proves nothing. Why do you keep posting benchmarks which proves nothing? Do you think that a 3x faster server proves that Linux is faster?




    When comes to Bonwick then yes, he was FUDing and everyone knows this. The answer to your question is simple - but now it's you who has to answer.
    If Bonwick FUDs, then show me a big SMP Linux server. Show me links. Go ahead. I have asked this many times, but you have never showed me a big SMP server. Why? Because there are none! Linux does not scale on SMP servers. This is true. If this is false, then you can show me a big SMP server. I am still waiting, I have asked you this many times. Show me the link. Prove that Bonwick FUDs by showing a big SMP Linux server.




    In which Kebbabert's dream? It just shows you''re just a troll. Freebsd is far more bloated than Linux. Unmodified Linux can run even on mobile devices while bsd can't. Linux's success in nearly every area proofs it's much better designed than slowlaris or bsd.
    http://www.phoronix.com/scan.php?pag...inux_games_bsd
    New tests have revealed that the modern FreeBSD operating system (via PC-BSD 8.2) can actually outperform Linux when it comes to running OpenGL Linux game binaries.

    This is hilarious. Even when running Linux software, FreeBSD is faster. Everybody is faster than Linux. Solaris is faster. FreeBSD is faster. I would not be surprised if even Windows was faster. Studies by Intel shows that Linux has dropped 10% performance. Linux is slowest in the league.
    http://www.theregister.co.uk/2009/09..._bloated_huge/
    "Citing an internal Intel study that tracked kernel releases, Bottomley said Linux performance had dropped about two per centage points at every release, for a cumulative drop of about 12 per cent over the last ten releases. "Is this a problem?" he asked.
    "We're getting bloated and huge. Yes, it's a problem," said Torvalds."




    http://scalability.org/?p=388

    There are many more things that can be said about slowlaris. You can't even imagine.However, it's sad you trash the forum and I have to keep replying to troll like you. Keep in mind it doesn't give me any satisfaction.
    I have no problems with Linux being faster on a single cpu in some benchmarks. It might be true, and I dont deny that. This guy runs a software on his desktop pc, probably it is a single cpu pc. I have no problems if BTRFS is faster than ZFS on a single disk.

    ZFS and Solaris is built for scale, for big SMP servers with 64 cpus and beyond. Linux is not.

    When we compare as few as 16 SSD disks, then ZFS is faster than BTRFS because BTRFS is not built to scale, BTRFS is a desktop filesystem:
    http://www.mail-archive.com/linux-bt.../msg05689.html

    When we compare as few as 48 cores on SAP benchmarks, then Solaris is faster. If we go above 8 cpus, or even 16 cpus, then Solaris crushes easily because Linux is not built for big SMP servers.



    As we have seen, Linux scales bad at 6 cpus in SAP benchmarks with a low 87% cpu utilization. Ext4 creator Ted Tso explains why. The reason is that Linux developers dont have access to big SMP servers, so Linux can not be improved on SMP servers:
    thunk.org/tytso/blog/2010/11/01/i-have-the-money-shot-for-my-lca-presentation/
    ...Ext4 was always designed for the “common case Linux workloads/hardware”, and for a long time, 48 cores/CPU’s and large RAID arrays were in the category of “exotic, expensive hardware”, and indeed, for much of the ext2/3 development time, most of the ext2/3 developers didn’t even have access to such hardware. One of the main reasons why I am working on scalability to 32-64 nodes is because such 32 cores/socket will become available Real Soon Now...
    There dont exist big SMP Linux servers. Only up to 8 cpus, which is bad, compared to 64 cpus and beyond.



    And now you have to answer, Kraftman.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •