Read/Write Speeds on Adaptec 51645

Raid0 (two 10k RPM Raptors)
Read: 389MB/s (99.9% CPU usage)
Write: 107MB/s (99.8% CPU usage)
Raid10 (8x 750GB SATA 16MB Cache 7200RPM)
Read: 343MB/s (99.9% CPU usage)
Write: 111MB/s (71% CPU usage)
Single 500GB 32MB Cache 7200 RPM
Read: 408MB/s (99.9% CPU usage)
Write: 69.8MB/s (75% CPU usage)

>2TB on a fresh raid set

I figured out a pretty big hurdle today in the area over 2TB of data.  For some reason (probably because I first tried using fdisk) … my 2998GB (usable) media raid I could not partition correctly.  I had planned on using xfs, possibly ext3.  At first, I tried fdisk /dev/sdb … but I quickly found that it maxed out ~2TB.  After reading up a little more on google … I found that I needed to use parted … or gparted.  After a couple of days of toil and searching for reasons that libparted kept failing over 2TB … I found that I needed to rewrite my partition table.  This sounds redundant, but I actually needed to REwrite it in a different format.  It seemed to have defaulted to an msdos TYPE of partition table.  I found an article that told me how to resolve the issue.  Essentially, what you have to do is — in gparted you go Device->Create Partition Table->pull down advanced->then select the ‘gpt’ format.  Once I did that, life was grand and I could make the raid the full size of 2.72TB instead of the ‘meager’ 2.0TB that it was being limited to.   Problem solved … one down .. a few more to go

Some server success

After much finagling… I finally got SOME  success.  I am writing this post from the server itself.  So there has to be at least a few positives — right?  I got F9-x86_64 installed and working.  I setup the raid controller from its pre-POST <Ctrl-A/F>? utility.  When I went to install, my two raid partitions showed up perfectly.  The rootfs was ~79GB and the media raid10 was ~2997GB.  I installed F9 to the rootfs drives and away I went.

Here’s the interesting part.  I’ve tried fdisk /dev/sdb (the media raid10) — it accurately shows 2997.8GB when you print out the stats.  I have tried both mkfs.xfs and mkfs.ext3 … after either one, I only get 2.0TB Available … can’t figure it out.  If I use 1000 instead of 1024 … (df -i) .. it reads 2.2TB.  But nothing gets close to the 2997GB.  Does formatting really take up ~1/3 of my drive space!?  Something can’t be right.

At any rate… I did some testing:  Here’s some specs with the current setup:

Max Write Speed: (dd if=/dev/zero of=somefile)  153MB/s

Max Read speed: (dd if=somefile of=/dev/zero)  356MB/s

Not too shabby…. not too shabby indeed.  I’ll keep you posted about the space dilemna.  If that’s the case .. I may need to switch to raid 5EE or 6.

More Server Woes

So, tonight I got some more parts … and ran into more problems.  I found some M2A-VM motherboards at work today and bought them for ~$45/each.  I got two.  My thought was, I could use one in the server and have a backup in case anything ever happened.

I got the miniSAS to SATA cables in today as well.  I put together the M2A-VM in the server case and found out … the only PCI-E slot was located where there was no slot in the backplane.  My raid controller is a PCI-E x8 slot.   So, that plan failed.  I ended up deciding to pull my motherboard out of my main desktop machine.  It’s an Intel Q6600 with 4GB of DDR2-800 on a P5K~ something motherboard (it has built-in wifi).  It USED to have Vista … Eventually, I will re-install Windows XP.

As I was putting together the screws in the hard drive carriers, I removed the tubs (removed 2 screws) and went to install the hard drives.  I put the two screws in from the tubs, but the other screws that came with the case had some sort of “tite-lock” or something similar.  So, they would get between 1 and 1.5 turns before they locked up –nowhere near flush (like they were supposed to be).  I resorted to using only 2 screws.

So I installed all 8 of the 750GB hard drives in a raid 10 for the media storage array.  I then added two more hard drives for the root files system.  I had an 80GB SATA hard drive and I “borrowed” Chris’ 500GB SATA hard drive in order to make the root file system a raid 1.

I then went to install the OS (Fedora 9, x86_64), but the two copies of the install DVDs were both scratched .. or at least neither worked.  Sooooo …. I’m downloading a new copy of it.  Meanwhile, the long awaited server sits put together (minus the extender for the front-panel connector), but … no OS.

System Allocations

Foreword:  I traded Marc my original iPhone (1st Gen) for a Dell Laptop — Latitude D505.  Dual-core 1.6GHz with 512MB RAM and a ~30GB HDD.  Not too bad.  I re-installed Windows XP Pro on it today and finally got all of the updates done … what a pain.  I forgot how tedious Windows installs were.

Laptop:

• Manufacturer:  Dell
• Case: Laptop
• CPU: Dual-core 1.6GHz
• RAM: 512MB RAM DDR-333 (Ordered 2GB)
• HDD: 30GB
• OS: Windows XP Pro (w/ SP3) … soon to Dual-boot with Fedora 9 (i386/32-bit)

New Server:

• Manufacturer: Custom
• Case: Supermicro 3U Rackmount
• CPU:  Intel Quad-core Q6600 — Quad 2.4GHz 64-bit CPU
• Mobo: P5K~Wifi
• RAM: 4GB DDR2-800
• HDD: 2xSATA ~100GB for base install + 8×750GB SATA drives in raid10
• OS: Fedora 9 (i386 / 32-bit)

Main Desktop:

• Manufacturer: Custom
• Case: Lian Li Case
• HDD: 2x 147GB SATA WD Raptor @ 10k RPM in Hardware Raid0 = ~330GB Logical drive C:
• CPU: AM2 Dual-Core 2.3GHz
• Mobo: M2A-VM
• RAM: 4GB DDR2-800
• OS: Windows Vista Ultimate
• Video: nVidia 7800GTX

Windows Box (on TV):

• Manufacturer: Custom
• Case: Beige
• CPU:
• Mobo: A8N32-SLI
• RAM:
• HDD:
• OS: Windows XP Pro

Dell 8400:

• Manufacturer: Dell
• Case: Dell
• CPU: Intel 2.4GHz 32-bit CPU
• Mobo:  Dell?
• RAM: 2GB?
• HDD: Internal SATA drive + 1TB Drive in hot-swap carrier
• OS: Fedora 9 – i386 / 32-bit

Beige box (To Come):

• Manufacturer: Custom
• Case: Beige
• CPU: Dual-core 2.3GHz AM2 cpu
• Mobo: M2~
• RAM: 2GB DDR2-800
• HDD:
• OS: Windows XP Pro

Old Black Case:

• Manufacturer: Custom
• Case: Black, windowed case
• CPU: Athlon XP Barton 2800+
• Mobo: ECS?
• RAM: 2GB DDR?
• HDD:
• OS: Windows XP Pro
• Bad PSU

Old Server Case:

• Manufacturer: Custom
• Case: Thermaltake Xaser II
• CPU: Athlon FX-55 (Dual core)
• Mobo: A8N32-SLI
• RAM: 4GB DDR-400
• HDD: 250GB PATA IDE
• OS:
• Somewhat empty at the moment

Data Server (incomplete)

So, yesterday I started working on a data server.  I had one that we WERE using, but for various reasons I got frustrated and decided to “go all out” and buy some really expensive parts and piece together a more reasonable long-term data storage solution.

Original box:  A8N-SLI Premium with an FX-55 and 4GB DDR-800 on Fedora 9 — also 650W PSU and a Thermaltake Xaser (I think it is a model ‘II’) case.  I had the / fs on a software raid 1 between two PATA IDE drives (a 200GB and a 250GB … 200GB logical space).  Then I had 5 SATA 750GB drives running software raid 5 (for a total of 3TB logical space).  I also had a PATA 750GB that I was using for  my /var/www/ mount point.

New Case

Story:  We had a lightning storm take out the power at the apartment … multiple times.  I got home from work one evening and the server was off.  Not good.  I booted it up .. and I noticed that my /dev/md2 (3TB raid5) mount did not mount correctly.  I verified it existed correctly in /etc/fstab .. that looked good.  I then proceeded to try to “recreate” its existence.  Aparently, I did it incorrectly … long story short — I hosed my 3TB raid partition.  So, I rebuilt it from “scratch” … with no data.  I luckily had a backup of ~1/2 of the data (another amusing story).  Moral of the story was … Per my previous notes, you notice that I make my original raid5 partition with 3 drives … then I grow it by adding 2 disks (1 at a time).  I then show how to grow the xfs partition as well … all without unmounting.  Its pretty slick really.  However, one key thing that was lacking (originally) was the fact that we didn’t rewrite /etc/mdadm.conf.  So, when the server rebooted, it was trying to initialize a 3-disk raid5 which no longer existed.  I have since gone back and added the necessary changes to my guide, but it costs me a bit.  I chalked it up to a learning experience.  Moving on … I resynced all of my backup data to the newly formatted 3TB drive.

We have since moved to a house.  The new house has some pretty serious power issues, which we readily found and tried to mitigate by using UPS’s.  Strangely enough .. the same ones we had been using at the apartment.  This was strike two … I assumed that the UPS was good and the system just shutdown improperly.  We had fixed the software problem with the raid.  Now … we had another power problem.  The UPS worked (the first time).  And then it didn’t.  I woke up last Friday morning to a system that was powered off AND would not power back up.  I assumed power supply, which I later confirmed.

So, with all of these problems … plus the angst of having to move all of my gear .. I decided to just “go all out” and buy an uber RAID controller and a sweet rack-mount case that could hold tons of sata drives.  I did a bit of homework.  I had been eyeing up an adaptec controller for a while.  Here’s what I ended up getting:

Adaptec Raid Controller

Adaptec 51645 – Dual 1.2GHz cpu with 512MB DDR2 (yeah .. its JUST a PCI-E card).  It occupies a PCI-E x8 slot and controls 16 internal and 4 external SAS or SATA ports via 4-to-1 cables that connect to mini-SAS ports. ~$825

SuperMicro Rackmount case with 15 SATA drives – This bad boy has room for 15 SATA drives and a triple-redundant, hot-swap 700W power supply. ~$750

So, I ordered both and awaited their arival.  They were both supposed to arrive on Thursday.  They did, but I ended up being sick that day … and I had them shipped to work, so I didn’t get to them until Friday.  when I took a look I found a couple of things REALLY lacking.  First, the Adaptec Raid controller came with NO cables … zip, zero, nada.  I considered this to be REALLY lame considering the amount I spent for the thing.  Somewhere along the line, I thought I read that it came with 2.  Apparently I was mistaken.  I took a gander around the web and came up with some $25 cables from PROVANTAGE.  They should do the trick.  I got 4 of them.

Second MAJOR thing was the front-panel cable on the server case is about 3-4″ too short.  Entirely non-sensical.  It was designed to work with THEIR branded motherboards.  However, they sell the case separately without regard for a STANDARD installation location for a front-panel header.  How stupid.  At any rate, I found this page at Performance-PCS.com which has extensions for the 10-pin front-panel connector.  By the look of it, its about 3-4″ of extension.  I couldn’t be sure, so I ordered 2 of them.  They were only $12.95+S/H each.

I was eager to get started.  I thought … you know I could still install the hardware (Mobo/memory/CPU/expansion cards) and then install the OS and get all the drivers working right … even without all of those cables.  First, I tried some modifications to my existing front-panel cable to make it longer.  Lets just say, I didn’t have the appropriate tools and I just ended up smelling up the entire house of solder.

So I started putting together the parts.  I purchased some DDR2 and I had an M2-based board laying around and I have a dual-core AM2 processor laying around … its like 2.3GHz.  I was going to throw it together for this build.  That did NOT end up happening.  Couple of reasons, I didn’t remember that the board only had 2 memory slots.  2GB is probably enough, but the board isn’t designed for all of the function that I have in mind.  So, I scratched that idea (plus I didn’t have a CPU fan/heatsink).

Bad CPU fan too tall.

Then I decided to verify that the “old” server JUST had a bad power supply.  It was only the power supply … so I ended up deciding to just use it in the new case for now.  After I secured it,  I verified that it powere up properly. Then I went to connect the CPU fan when I noticed my next problem.  It was too tall to fit in the case.

Talk about devastation and disgust.  Sick days … down servers and poor part design have all added up to one fiasco of a project.  The good news is, I have ordered 2 more CPU heatsink/fans and 3 more Power supplies and some thermal grease.  So, when this is all said and done, we should have a few “extra” computers floating around for our LAN party days.

My goal for this machine:   End up in my rack, hosting 3TB of logical space across a raid10 on 8 physical 750GB drives and running Fedora 9 hosting a couple of samba shares as well as some other random services that are key to our “base of operations”.

Patch panel and 48-port Gigabit (fully) switch

New Rack with 1of10 image processing servers

Fedora 9 + VMWare server 1.0.5/6a

Step 1: Download VMWare Server files from here

If you prefer cmd line:

wget http://download3.vmware.com/software/vmserver/VMware-server-1.0.6-91891.tar.gz

Step 2: Install required packages

yum install kernel-devel-`uname -r`

If that does not work, you have to make sure that you update all of your packages (yum update) and then try it again. If it still does not work, downgrade your kernel version. Take note of whatever the most recent kernel-devel uses for the kernel #, then just do a yum install kernel-that# If you have to install a different kernel, you will have to update your grub/lilo bootloader to use your newly installed kernel. Then you will have to reboot and boot into that kernel. THEN you can do the above yum install.

yum install make kernel-devel-`uname -r` kernel-headers-`uname -r` xinetd gcc gcc-c++ perl-devel perl-ExtUtils-Embed

Extract the VMWare tarball:

tar xvfz VMware-server-*.tar.gz
cd vmware-server-distrib
./vmware-install.pl

I have had this work and I have had it fail. If it fails, you need to use the any-any patch. Some use the 115 or the 117. I have gotten both to work. I use a mix of F8 and F9 on both i386 and x64 platforms. Here is a link to the 115 and here is a link to the 117. After extracting these patches, run the runme.pl scripts to apply them. After they apply, they should automatically start the installer script. Sometimes even after the any-any update, the vmware-cmd does not compile correctly. This is OK. Its not perfect, but we can function without it.

You can get Serial Codes from here

A quick note about VM’s – every “instance” of a network card gets a new ID. The setup file for each VM appliance saves the MAC address of the network card. If you ever move the VM, or copy it, or anything that puts the device in a way that it is not the original file, it will assign it a new MAC address. However, the config file will still try to use the original MAC address. You have to comment out the MAC from the .vmx file to get the network card to re-acquire a new MAC. The safest thing to do is actually to remove all NIC’s and just re-add the ones you want.

Additionally, if your OS has knowledge of the NIC’s physical address, it will not work until it is fixed. I.E. – for Fedora 9, you have to either comment out – or update the /etc/sysconfig/network-scripts/ifcfg-eth0 portion of the file that contains the physical address (or MAC). If you don’t, then the OS will try to use the old MAC – only it will fail and the interface won’t come up and you’ll be left clueless as to why your network is fubar.

F9 Must-have programs

This is a list of programs that I must have – anytime I install F9. I’m sure this list will grow with time. I wanted to get this list officially started.

yum install:

• munin
• lm_sensors
• lm_sensors-sensord
• ntfs-3g
• httpd (Apache)
• mysql-server
• phpMyAdmin
• hddtemp
• sysstat
• procps

F9 Command list

This is just a bunch of notes that I had in a txt file that I wanted to document. Some cool commands to be familiar with.

iostat          Monitor Hard Disk I/O
vmstat          Monitor CPU Utilization
sensors         Monitor CPU / System Temps
hddtemp         Monitor HDD Temps
fuser <file>    Shows all processes accessing <file>
du -ah <folder> Shows size of all files in a folder -human readable

cat /proc/mdstat        Check Raid Status
hdparm /dev/sda         Check Drive parameters on /dev/sda

###  Raid 5 Commands ###
yum install xfs xfsprogs xfstools kmod-xfs

## Create Initial Raid
fdisk /dev/sdd
        (delete all previous partitions)
        n               New Partition
                p       Primary
                1       Partition #1
                1       Starting Block
                <enter> Max size of drive
        t               Type of partition
                fd      (or df - Linux raid autodetect)
        w               Write-out changes and exit
## Repeat for all drives in raid

## Create Raid 5
mdadm --create --verbose /dev/md2 --level=5 --auto=yes --raid-devices=3 /dev/sd[jid]1
## Wait for it to finish
watch cat /proc/mdstat
##Format the new device
mkfs.xfs -f /dev/md2
##Mount new device
mount /dev/md2 /media/raid5

Software Raid 5 – Fedora 9 style

Use fdisk to create a single primary partition on sd[cde] of type fd (linux raid)

If you create raid volume initially with all but one disk (you have to do minimum of 3 initially) – then you can see how to add a disk to the raid after the fact. At any rate – here’s how to do the initial setup. Afterwards, I’ll show you how to add a disk to the raid.

mdadm --create --verbose /dev/md2 --level=5 --auto=yes --raid-devices=3 /dev/sd[cde]1

Wait for the initial build to complete (150 minutes with 750GB drives with SATA-300, ~80MB/s)

watch cat /proc/mdstat

Install xfs File system (you can dynamically expand it without unmounting )

yum install xfsprogs xfstools kmod-xfs

Create xfs partition in the raid

mkfs.xfs -f /dev/md2

Mount it … copy some files … do with it what you will. You now have a raid 5 partition!

Expanding by 1 disk:

Create a single partition (fdisk) as before – type fd (linux raid)
Add it to the existing array as a spare

mdadm --add /dev/md2 /dev/sdf1

Grow the array from 3 to 4 disks

mdadm --grow /dev/md2 --raid-devices=4

Allow the array to reshape. (As a reference … I went from 4x 750GB drives to 5x 750GB – it took me somewhere between 7 and 8 days to reshape … I was only reshaping @ 1MB/s due to a bad SATA controller.)

When it finishes, grow the xfs to fill the newly expanded array.

xfs_grow /dev/md2

*** Note: Any time you resize your raid — be sure to re-create /etc/mdadm.conf… I have good horror stories of what happens when you don’t

cat /etc/mdadm.conf | grep -v md2 > ~/tempfile; mdadm --detail --scan | grep md2 >> ~/tempfile; mv /etc/mdadm.conf /etc/mdadm.conf.old; mv ~/tempfile /etc/mdadm.conf