>2TB on a fresh raid set

I figured out a pretty big hurdle today in the area over 2TB of data.  For some reason (probably because I first tried using fdisk) … my 2998GB (usable) media raid I could not partition correctly.  I had planned on using xfs, possibly ext3.  At first, I tried fdisk /dev/sdb … but I quickly found that it maxed out ~2TB.  After reading up a little more on google … I found that I needed to use parted … or gparted.  After a couple of days of toil and searching for reasons that libparted kept failing over 2TB … I found that I needed to rewrite my partition table.  This sounds redundant, but I actually needed to REwrite it in a different format.  It seemed to have defaulted to an msdos TYPE of partition table.  I found an article that told me how to resolve the issue.  Essentially, what you have to do is — in gparted you go Device->Create Partition Table->pull down advanced->then select the ‘gpt’ format.  Once I did that, life was grand and I could make the raid the full size of 2.72TB instead of the ‘meager’ 2.0TB that it was being limited to.   Problem solved … one down .. a few more to go

Some server success

After much finagling… I finally got SOME  success.  I am writing this post from the server itself.  So there has to be at least a few positives — right?  I got F9-x86_64 installed and working.  I setup the raid controller from its pre-POST <Ctrl-A/F>? utility.  When I went to install, my two raid partitions showed up perfectly.  The rootfs was ~79GB and the media raid10 was ~2997GB.  I installed F9 to the rootfs drives and away I went.

Here’s the interesting part.  I’ve tried fdisk /dev/sdb (the media raid10) — it accurately shows 2997.8GB when you print out the stats.  I have tried both mkfs.xfs and mkfs.ext3 … after either one, I only get 2.0TB Available … can’t figure it out.  If I use 1000 instead of 1024 … (df -i) .. it reads 2.2TB.  But nothing gets close to the 2997GB.  Does formatting really take up ~1/3 of my drive space!?  Something can’t be right.

At any rate… I did some testing:  Here’s some specs with the current setup:

Max Write Speed: (dd if=/dev/zero of=somefile)  153MB/s

Max Read speed: (dd if=somefile of=/dev/zero)  356MB/s

Not too shabby…. not too shabby indeed.  I’ll keep you posted about the space dilemna.  If that’s the case .. I may need to switch to raid 5EE or 6.

More Server Woes

So, tonight I got some more parts … and ran into more problems.  I found some M2A-VM motherboards at work today and bought them for ~$45/each.  I got two.  My thought was, I could use one in the server and have a backup in case anything ever happened.

I got the miniSAS to SATA cables in today as well.  I put together the M2A-VM in the server case and found out … the only PCI-E slot was located where there was no slot in the backplane.  My raid controller is a PCI-E x8 slot.   So, that plan failed.  I ended up deciding to pull my motherboard out of my main desktop machine.  It’s an Intel Q6600 with 4GB of DDR2-800 on a P5K~ something motherboard (it has built-in wifi).  It USED to have Vista … Eventually, I will re-install Windows XP.

As I was putting together the screws in the hard drive carriers, I removed the tubs (removed 2 screws) and went to install the hard drives.  I put the two screws in from the tubs, but the other screws that came with the case had some sort of “tite-lock” or something similar.  So, they would get between 1 and 1.5 turns before they locked up –nowhere near flush (like they were supposed to be).  I resorted to using only 2 screws.

So I installed all 8 of the 750GB hard drives in a raid 10 for the media storage array.  I then added two more hard drives for the root files system.  I had an 80GB SATA hard drive and I “borrowed” Chris’ 500GB SATA hard drive in order to make the root file system a raid 1.

I then went to install the OS (Fedora 9, x86_64), but the two copies of the install DVDs were both scratched .. or at least neither worked.  Sooooo …. I’m downloading a new copy of it.  Meanwhile, the long awaited server sits put together (minus the extender for the front-panel connector), but … no OS.

System Allocations

Foreword:  I traded Marc my original iPhone (1st Gen) for a Dell Laptop — Latitude D505.  Dual-core 1.6GHz with 512MB RAM and a ~30GB HDD.  Not too bad.  I re-installed Windows XP Pro on it today and finally got all of the updates done … what a pain.  I forgot how tedious Windows installs were.

Laptop:

• Manufacturer:  Dell
• Case: Laptop
• CPU: Dual-core 1.6GHz
• RAM: 512MB RAM DDR-333 (Ordered 2GB)
• HDD: 30GB
• OS: Windows XP Pro (w/ SP3) … soon to Dual-boot with Fedora 9 (i386/32-bit)

New Server:

• Manufacturer: Custom
• Case: Supermicro 3U Rackmount
• CPU:  Intel Quad-core Q6600 — Quad 2.4GHz 64-bit CPU
• Mobo: P5K~Wifi
• RAM: 4GB DDR2-800
• HDD: 2xSATA ~100GB for base install + 8×750GB SATA drives in raid10
• OS: Fedora 9 (i386 / 32-bit)

Main Desktop:

• Manufacturer: Custom
• Case: Lian Li Case
• HDD: 2x 147GB SATA WD Raptor @ 10k RPM in Hardware Raid0 = ~330GB Logical drive C:
• CPU: AM2 Dual-Core 2.3GHz
• Mobo: M2A-VM
• RAM: 4GB DDR2-800
• OS: Windows Vista Ultimate
• Video: nVidia 7800GTX

Windows Box (on TV):

• Manufacturer: Custom
• Case: Beige
• CPU:
• Mobo: A8N32-SLI
• RAM:
• HDD:
• OS: Windows XP Pro

Dell 8400:

• Manufacturer: Dell
• Case: Dell
• CPU: Intel 2.4GHz 32-bit CPU
• Mobo:  Dell?
• RAM: 2GB?
• HDD: Internal SATA drive + 1TB Drive in hot-swap carrier
• OS: Fedora 9 – i386 / 32-bit

Beige box (To Come):

• Manufacturer: Custom
• Case: Beige
• CPU: Dual-core 2.3GHz AM2 cpu
• Mobo: M2~
• RAM: 2GB DDR2-800
• HDD:
• OS: Windows XP Pro

Old Black Case:

• Manufacturer: Custom
• Case: Black, windowed case
• CPU: Athlon XP Barton 2800+
• Mobo: ECS?
• RAM: 2GB DDR?
• HDD:
• OS: Windows XP Pro
• Bad PSU

Old Server Case:

• Manufacturer: Custom
• Case: Thermaltake Xaser II
• CPU: Athlon FX-55 (Dual core)
• Mobo: A8N32-SLI
• RAM: 4GB DDR-400
• HDD: 250GB PATA IDE
• OS:
• Somewhat empty at the moment

Data Server (incomplete)

So, yesterday I started working on a data server.  I had one that we WERE using, but for various reasons I got frustrated and decided to “go all out” and buy some really expensive parts and piece together a more reasonable long-term data storage solution.

Original box:  A8N-SLI Premium with an FX-55 and 4GB DDR-800 on Fedora 9 — also 650W PSU and a Thermaltake Xaser (I think it is a model ‘II’) case.  I had the / fs on a software raid 1 between two PATA IDE drives (a 200GB and a 250GB … 200GB logical space).  Then I had 5 SATA 750GB drives running software raid 5 (for a total of 3TB logical space).  I also had a PATA 750GB that I was using for  my /var/www/ mount point.

New Case

Story:  We had a lightning storm take out the power at the apartment … multiple times.  I got home from work one evening and the server was off.  Not good.  I booted it up .. and I noticed that my /dev/md2 (3TB raid5) mount did not mount correctly.  I verified it existed correctly in /etc/fstab .. that looked good.  I then proceeded to try to “recreate” its existence.  Aparently, I did it incorrectly … long story short — I hosed my 3TB raid partition.  So, I rebuilt it from “scratch” … with no data.  I luckily had a backup of ~1/2 of the data (another amusing story).  Moral of the story was … Per my previous notes, you notice that I make my original raid5 partition with 3 drives … then I grow it by adding 2 disks (1 at a time).  I then show how to grow the xfs partition as well … all without unmounting.  Its pretty slick really.  However, one key thing that was lacking (originally) was the fact that we didn’t rewrite /etc/mdadm.conf.  So, when the server rebooted, it was trying to initialize a 3-disk raid5 which no longer existed.  I have since gone back and added the necessary changes to my guide, but it costs me a bit.  I chalked it up to a learning experience.  Moving on … I resynced all of my backup data to the newly formatted 3TB drive.

We have since moved to a house.  The new house has some pretty serious power issues, which we readily found and tried to mitigate by using UPS’s.  Strangely enough .. the same ones we had been using at the apartment.  This was strike two … I assumed that the UPS was good and the system just shutdown improperly.  We had fixed the software problem with the raid.  Now … we had another power problem.  The UPS worked (the first time).  And then it didn’t.  I woke up last Friday morning to a system that was powered off AND would not power back up.  I assumed power supply, which I later confirmed.

So, with all of these problems … plus the angst of having to move all of my gear .. I decided to just “go all out” and buy an uber RAID controller and a sweet rack-mount case that could hold tons of sata drives.  I did a bit of homework.  I had been eyeing up an adaptec controller for a while.  Here’s what I ended up getting:

Adaptec Raid Controller

Adaptec 51645 – Dual 1.2GHz cpu with 512MB DDR2 (yeah .. its JUST a PCI-E card).  It occupies a PCI-E x8 slot and controls 16 internal and 4 external SAS or SATA ports via 4-to-1 cables that connect to mini-SAS ports. ~$825

SuperMicro Rackmount case with 15 SATA drives – This bad boy has room for 15 SATA drives and a triple-redundant, hot-swap 700W power supply. ~$750

So, I ordered both and awaited their arival.  They were both supposed to arrive on Thursday.  They did, but I ended up being sick that day … and I had them shipped to work, so I didn’t get to them until Friday.  when I took a look I found a couple of things REALLY lacking.  First, the Adaptec Raid controller came with NO cables … zip, zero, nada.  I considered this to be REALLY lame considering the amount I spent for the thing.  Somewhere along the line, I thought I read that it came with 2.  Apparently I was mistaken.  I took a gander around the web and came up with some $25 cables from PROVANTAGE.  They should do the trick.  I got 4 of them.

Second MAJOR thing was the front-panel cable on the server case is about 3-4″ too short.  Entirely non-sensical.  It was designed to work with THEIR branded motherboards.  However, they sell the case separately without regard for a STANDARD installation location for a front-panel header.  How stupid.  At any rate, I found this page at Performance-PCS.com which has extensions for the 10-pin front-panel connector.  By the look of it, its about 3-4″ of extension.  I couldn’t be sure, so I ordered 2 of them.  They were only $12.95+S/H each.

I was eager to get started.  I thought … you know I could still install the hardware (Mobo/memory/CPU/expansion cards) and then install the OS and get all the drivers working right … even without all of those cables.  First, I tried some modifications to my existing front-panel cable to make it longer.  Lets just say, I didn’t have the appropriate tools and I just ended up smelling up the entire house of solder.

So I started putting together the parts.  I purchased some DDR2 and I had an M2-based board laying around and I have a dual-core AM2 processor laying around … its like 2.3GHz.  I was going to throw it together for this build.  That did NOT end up happening.  Couple of reasons, I didn’t remember that the board only had 2 memory slots.  2GB is probably enough, but the board isn’t designed for all of the function that I have in mind.  So, I scratched that idea (plus I didn’t have a CPU fan/heatsink).

Bad CPU fan too tall.

Then I decided to verify that the “old” server JUST had a bad power supply.  It was only the power supply … so I ended up deciding to just use it in the new case for now.  After I secured it,  I verified that it powere up properly. Then I went to connect the CPU fan when I noticed my next problem.  It was too tall to fit in the case.

Talk about devastation and disgust.  Sick days … down servers and poor part design have all added up to one fiasco of a project.  The good news is, I have ordered 2 more CPU heatsink/fans and 3 more Power supplies and some thermal grease.  So, when this is all said and done, we should have a few “extra” computers floating around for our LAN party days.

My goal for this machine:   End up in my rack, hosting 3TB of logical space across a raid10 on 8 physical 750GB drives and running Fedora 9 hosting a couple of samba shares as well as some other random services that are key to our “base of operations”.

Patch panel and 48-port Gigabit (fully) switch

New Rack with 1of10 image processing servers