Automating FC SAN failover using pexpect

The code is on github and is fairly straight forward. One of these days I'll figure out how to do this using SMI-S or other XML magic pixie dust. The Cisco MDS is a little obnoxious in that it requires you to entire config mode to toggle vsans. The switch is already configured for pubkey authentication. Here it is in action,

ppetraki@mark21:~/Sandbox/cisco-automation$ ./cisco_pexpect.py 
logging in...
logged in
entered config mode
enable both vsans, wait 30 secs
disable vsan 10, wait 60 secs
enable vsan 10
exit config mode
exit fc switch
terminated True

In a separate window I have 'watch -d multipath -ll' on the client and confirm during the 60 sec sleep that the path group did indeed go out of service and then return.

$ git clone git://github.com/ppetraki/cisco-fc-failover.git

and enjoy!

Determine Linux "sd" names for debugging

The way the Linux SD driver works is it reserves 16 minor 'slots' per block device, the first minor represents the whole disk, and the remaining 15 are reserved for partitions. Knowing this offset one can write a little python code to create an index table that aids debugging, mapping minor numbers back to block device names. This of course presumes that names are being recycled.

# python -c "print zip(range(0,443,16), map(chr, range(97, 123)))" [(0, 'a'), (16, 'b'), (32, 'c'), (48, 'd'), (64, 'e'), (80, 'f'), (96, 'g'), (112, 'h'), (128, 'i'), (144, 'j'), (160, 'k'), (176, 'l'), (192, 'm'), (208, 'n'), (224, 'o'), (240, 'p'), (256, 'q'), (272, 'r'), (288, 's'), (304, 't'), (320, 'u'), (336, 'v'), (352, 'w'), (368, 'x'), (384, 'y'), (400, 'z')]

So sdb would be 8,16 and sdm would be 8,192, sdm1 is 8,193. Now the next time you see something awful like this:

Jul 10 15:08:25 | checker failed path 8:144 in map mpath2 Jul 10 15:08:25 | libdevmapper: ioctl/libdm-iface.c(1740): dm message mpath2 NF fail_path 8:144 [16384]

You can say with confidence that this is device sdj.

Introducing the Disk Puller

A cheap and effective way to hotplug disks automatically that I pioneered at Stratus while working on internal storage.

The basic idea is instead of letting the backplane or computer power  the drive, we do instead with our external power supply. According  to the storage controller's firmware, phy down == hotplug removed, which can be achieved by removing power to the target. This is 100% authentic and equal to walking up to a system and pulling/reinserting the disk by hand.

Parts list:

• Acquire a PDU e.g. those things that let you power servers on/off remotely. You really just need the ability to twiddle the power source remotely via SNMP or a web interface. 
• An external molex power supply like this. 
• http://www.amazon.com/BENCH-COMPUTER-POWER-SUPPLY-ADAPTER/dp/B0013OJOE0
• Backplane to breakout phy and power separately for any SAS/SATA disk 
• http://www.cs-electronics.com/sas-adapters.htm
• ADP-4200 
•  A SAS/SATA cable which has the composite male edge like the drive does and breaks out the phys and power. 
• http://www.cs-electronics.com/sas-cables.htm
• SAS-7P82-P/1m 

Great way to beat on MD, LVM, and the twisty maze that is the removal  path of the SCSI mid-layer.

It's pretty inexpensive, with a nice managed PDU and all the parts to automate 1 disk amounts to $80 per port. You can use the PDU for  other neat things too like say inserting and removing an external
USB hub, and it's connected peripherals, by controlling it's power source.

The equivalent functionality as part of an actual SAS/SATA target  emulator could cost you between 15-20K. The vendor whom I have the most experience with this product, who will remain nameless out of respect; required an exceptional amount of upkeep and would break for seemingly no reason at all. Instead of purchasing additional target emulators for fault injection, we instead installed disk pullers
everywhere for a fraction of the cost and with virtually no maintenance.