On Sat, 24 Dec 2005 15:42:14 +0100, Axel Thimm <[log in to unmask]> wrote: >On Thu, Dec 22, 2005 at 09:22:10PM -0600, Jeffry R. Abramson wrote: >> Speaking of SAN's, has anyone gotten the multipath feature in SL42 to work? >> We have an IBM blade equipped with a dual-port Qlogic controller connected >> through two independent fabrics to a DS400/Adaptec array with a dual-port >> controller. fdisk -l shows two paths to every logical array, ie. each disk >> shows up twice. > >You should be seeing more than that, you have at least four paths >(2xHBA ports x 2xcontroller ports). The disk array has a A/B pair of controllers in an active/passive configuration. We have zoned out the passive controller for now so only the A controller is used. Each disk controller has two FC ports configured as follows: ----[ Initiator 'blade6_FC0' ]------------------------------------------ WWN: 21000011259350e8 Name: blade6_FC0 ID: 0 Port access: Port A0 Port A1 Port B0 Port B1 --------------------------------------------------------------------------- YES NO YES NO LUN Mappings: LUN logical [Capacity] --------------------------------------------------------------------------- 0 Device-1 [683.48GB] 1 Device-2 [683.48GB] ----[ Initiator 'blade6_FC1' ]------------------------------------------ WWN: 21000011259350e9 Name: blade6_FC1 ID: 2 Port access: Port A0 Port A1 Port B0 Port B1 --------------------------------------------------------------------------- NO YES NO YES LUN Mappings: LUN logical [Capacity] --------------------------------------------------------------------------- 0 Device-1 [683.48GB] 1 Device-2 [683.48GB] The two san fabrics are completely independent so with this configuration there should be two paths to each logical device. fdisk -l yields the following: Disk /dev/sda: 73.4 GB, 73407488000 bytes 255 heads, 63 sectors/track, 8924 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 33 265041 83 Linux /dev/sda2 34 1077 8385930 83 Linux /dev/sda3 1078 8924 63031027+ 8e Linux LVM Disk /dev/sdb: 73.4 GB, 73407488000 bytes 255 heads, 63 sectors/track, 8924 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 1 8924 71681998+ 83 Linux Disk /dev/sdc: 733.8 GB, 733877370880 bytes 255 heads, 63 sectors/track, 89222 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdc1 1 89222 716675683+ 8e Linux LVM Disk /dev/sdd: 733.8 GB, 733877370880 bytes 255 heads, 63 sectors/track, 89222 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdd1 1 44611 358337826 8e Linux LVM /dev/sdd2 44612 89222 358337857+ 8e Linux LVM Disk /dev/sde: 733.8 GB, 733877370880 bytes 255 heads, 63 sectors/track, 89222 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sde1 1 89222 716675683+ 8e Linux LVM Disk /dev/sdf: 733.8 GB, 733877370880 bytes 255 heads, 63 sectors/track, 89222 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdf1 1 44611 358337826 8e Linux LVM /dev/sdf2 44612 89222 358337857+ 8e Linux LVM sda and sdb are the internal SCSI drives. sdc through sdf are the two volumes on the disk array. I put a single partition on one of the volumes and two on the other so it is easy to see which devices are actually the same. In this case, sdc and sde refer to the same volume as does sdd and sdf. /proc/scsi/scsi is as follows: Attached devices: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: IBM-ESXS Model: ST973401LC FN Rev: B41A Type: Direct-Access ANSI SCSI revision: 04 Host: scsi0 Channel: 00 Id: 01 Lun: 00 Vendor: IBM-ESXS Model: ST973401LC FN Rev: B41A Type: Direct-Access ANSI SCSI revision: 04 Host: scsi1 Channel: 00 Id: 00 Lun: 00 Vendor: IBM Model: DS400 S320 Rev: 7.00 Type: Direct-Access ANSI SCSI revision: 04 Host: scsi1 Channel: 00 Id: 00 Lun: 01 Vendor: IBM Model: DS400 S320 Rev: 7.00 Type: Direct-Access ANSI SCSI revision: 04 Host: scsi2 Channel: 00 Id: 00 Lun: 00 Vendor: IBM Model: DS400 S320 Rev: 7.00 Type: Direct-Access ANSI SCSI revision: 04 Host: scsi2 Channel: 00 Id: 00 Lun: 01 Vendor: IBM Model: DS400 S320 Rev: 7.00 Type: Direct-Access ANSI SCSI revision: 04 Host: scsi3 Channel: 00 Id: 00 Lun: 00 Vendor: TEAC Model: FD-05PUB Rev: 3200 Type: Direct-Access ANSI SCSI revision: 02 Host: scsi4 Channel: 00 Id: 00 Lun: 00 Vendor: HL-DT-ST Model: DVD-ROM GDR8082N Rev: 0L03 Type: CD-ROM ANSI SCSI revision: 02 cat /proc/scsi/qla2xxx/1 returns: QLogic PCI to Fibre Channel Host Adapter for IBM FCEC: Firmware version 3.03.15 IPX, Driver version 8.01.00b5-rh2 ISP: ISP2312, Serial# N04656 Request Queue = 0x11fd80000, Response Queue = 0x11fd10000 Request Queue count = 4096, Response Queue count = 512 Total number of active commands = 0 Total number of interrupts = 581 Device queue depth = 0x20 Number of free request entries = 3843 Number of mailbox timeouts = 0 Number of ISP aborts = 0 Number of loop resyncs = 0 Number of retries for empty slots = 0 Number of reqs in pending_q= 0, retry_q= 0, done_q= 0, scsi_retry_q= 0 Host adapter:loop state = <READY>, flags = 0x1a03 Dpc flags = 0x0 MBX flags = 0x0 Link down Timeout = 030 Port down retry = 030 Login retry count = 030 Commands retried with dropped frame(s) = 0 Product ID = 4953 5020 2020 0002 SCSI Device Information: scsi-qla0-adapter-node=20000011259350e8; scsi-qla0-adapter-port=21000011259350e8; scsi-qla0-target-0=21000000d12671dd; FC Port Information: scsi-qla0-port-0=20000000d12671dd:21000000d12671dd:010f00:81; SCSI LUN Information: (Id:Lun) * - indicates lun is not registered with the OS. ( 0: 0): Total reqs 72, Pending reqs 0, flags 0x0, 0:0:81 00 ( 0: 1): Total reqs 107, Pending reqs 0, flags 0x0, 0:0:81 00 cat /proc/scsi/qla2xxx/2 returns: QLogic PCI to Fibre Channel Host Adapter for IBM FCEC: Firmware version 3.03.15 IPX, Driver version 8.01.00b5-rh2 ISP: ISP2312, Serial# N04912 Request Queue = 0xdfc00000, Response Queue = 0x11fce0000 Request Queue count = 4096, Response Queue count = 512 Total number of active commands = 0 Total number of interrupts = 580 Device queue depth = 0x20 Number of free request entries = 3830 Number of mailbox timeouts = 0 Number of ISP aborts = 0 Number of loop resyncs = 0 Number of retries for empty slots = 0 Number of reqs in pending_q= 0, retry_q= 0, done_q= 0, scsi_retry_q= 0 Host adapter:loop state = <READY>, flags = 0x1a03 Dpc flags = 0x0 MBX flags = 0x0 Link down Timeout = 030 Port down retry = 030 Login retry count = 030 Commands retried with dropped frame(s) = 0 Product ID = 4953 5020 2020 0002 SCSI Device Information: scsi-qla1-adapter-node=20000011259350e9; scsi-qla1-adapter-port=21000011259350e9; scsi-qla1-target-0=21010000d12671dd; FC Port Information: scsi-qla1-port-0=20000000d12671dd:21010000d12671dd:010f00:81; SCSI LUN Information: (Id:Lun) * - indicates lun is not registered with the OS. ( 0: 0): Total reqs 72, Pending reqs 0, flags 0x0, 1:0:81 00 ( 0: 1): Total reqs 107, Pending reqs 0, flags 0x0, 1:0:81 00 > >> The instructions make it look easy (comment everything out >> from /etc/multipath.conf, start up multipathd and away you go). Running >> multipath -v2 -l shows nothing. multipath -v3 shows that it sees all of the >> devices but I don't know how to decipher where it is going wrong (I'd attach >> the output but I'm typing this from home). Ideas? -- Jeff > >Use multipath -F and try again. > Ran multipath -F followed by multipath -v2 and got the same results. multipath -v3 reveals the following: load path identifiers cache ux_socket_connect error ===== no paths ===== dm-0 blacklisted dm-1 blacklisted dm-2 blacklisted dm-3 blacklisted dm-4 blacklisted dm-5 blacklisted md0 blacklisted ram0 blacklisted ram10 blacklisted ram11 blacklisted ram12 blacklisted ram13 blacklisted ram14 blacklisted ram15 blacklisted ram1 blacklisted ram2 blacklisted ram3 blacklisted ram4 blacklisted ram5 blacklisted ram6 blacklisted ram7 blacklisted ram8 blacklisted ram9 blacklisted path sda not found in pathvec ===== path info sda (mask 0x1f) ===== bus = 1 dev_t = 8:0 size = 143374000 vendor = IBM-ESXS product = ST973401LC FN rev = B41A h:b:t:l = 0:0:0:0 tgt_node_name = serial = 3LB06LLL00007603NPNN path checker = readsector0 (internal default) state = 2 getprio = (null) (internal default) prio = 1 getuid = /sbin/scsi_id -g -u -s /block/%n (internal default) uid = SIBM-ESXSST973401LC____F3LB06LLL00007603NPNN (callout) path sdb not found in pathvec ===== path info sdb (mask 0x1f) ===== bus = 1 dev_t = 8:16 size = 143374000 vendor = IBM-ESXS product = ST973401LC FN rev = B41A h:b:t:l = 0:0:1:0 tgt_node_name = serial = 3LB06KLK0000760803PM path checker = readsector0 (internal default) state = 2 getprio = (null) (internal default) prio = 1 getuid = /sbin/scsi_id -g -u -s /block/%n (internal default) uid = SIBM-ESXSST973401LC____F3LB06KLK0000760803PM (callout) path sdc not found in pathvec ===== path info sdc (mask 0x1f) ===== bus = 1 dev_t = 8:32 size = 1433354240 vendor = IBM product = DS400 S320 rev = 7.00 h:b:t:l = 1:0:0:0 tgt_node_name = 0x20000000d12671dd serial = 00000000002671DD423829CD path checker = readsector0 (internal default) state = 2 getprio = (null) (internal default) prio = 1 getuid = /sbin/scsi_id -g -u -s /block/%n (internal default) uid = 20000d12671dd29cd (callout) path sdd not found in pathvec ===== path info sdd (mask 0x1f) ===== bus = 1 dev_t = 8:48 size = 1433354240 vendor = IBM product = DS400 S320 rev = 7.00 h:b:t:l = 1:0:0:1 tgt_node_name = 0x20000000d12671dd serial = 00000000002671DD423829CE path checker = readsector0 (internal default) state = 2 getprio = (null) (internal default) prio = 1 getuid = /sbin/scsi_id -g -u -s /block/%n (internal default) uid = 20000d12671dd29ce (callout) path sde not found in pathvec ===== path info sde (mask 0x1f) ===== bus = 1 dev_t = 8:64 size = 1433354240 vendor = IBM product = DS400 S320 rev = 7.00 h:b:t:l = 2:0:0:0 tgt_node_name = 0x20000000d12671dd serial = 00000000002671DD423829CD path checker = readsector0 (internal default) state = 2 getprio = (null) (internal default) prio = 1 getuid = /sbin/scsi_id -g -u -s /block/%n (internal default) uid = 20000d12671dd29cd (callout) path sdf not found in pathvec ===== path info sdf (mask 0x1f) ===== bus = 1 dev_t = 8:80 size = 1433354240 vendor = IBM product = DS400 S320 rev = 7.00 h:b:t:l = 2:0:0:1 tgt_node_name = 0x20000000d12671dd serial = 00000000002671DD423829CE path checker = readsector0 (internal default) state = 2 getprio = (null) (internal default) prio = 1 getuid = /sbin/scsi_id -g -u -s /block/%n (internal default) uid = 20000d12671dd29ce (callout) path sdg not found in pathvec ===== path info sdg (mask 0x1f) ===== bus = 1 dev_t = 8:96 sr0 blacklisted What's the significance of the "ux_socket_connect error" ? >Other than that check with scsi_id that the paths are to the same >device, e.g. > ># for x in sdb sdc sdd sde; do /sbin/scsi_id -g -u -s /block/$x; done >3600508b4000106f30006300000240000 >3600508b4000106f30006300000240000 >3600508b4000106f30006300000240000 >3600508b4000106f30006300000240000 > for x in sda sdb sdc sdd sde sdf ; do /sbin/scsi_id -g -u -s /block/$x; done SIBM-ESXSST973401LC____F3LB06LLL00007603NPNN SIBM-ESXSST973401LC____F3LB06KLK0000760803PM 20000d12671dd29cd 20000d12671dd29ce 20000d12671dd29cd 20000d12671dd29ce [root@blade6 tmp]# multipath -l [root@blade6 tmp]# ls -l /dev/dm* brw------- 1 root root 253, 0 Dec 23 08:30 /dev/dm-0 brw------- 1 root root 253, 1 Dec 23 08:30 /dev/dm-1 brw------- 1 root root 253, 2 Dec 23 08:30 /dev/dm-2 brw------- 1 root root 253, 3 Dec 23 08:30 /dev/dm-3 brw------- 1 root root 253, 4 Dec 23 08:30 /dev/dm-4 brw------- 1 root root 253, 5 Dec 23 08:30 /dev/dm-5 [root@blade6 tmp]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup_OS-root 1008M 227M 731M 24% / /dev/sda1 251M 17M 221M 8% /boot none 3.9G 0 3.9G 0% /dev/shm /dev/mapper/VolGroup_OS-home 1.2G 33M 1.1G 3% /home /dev/mapper/VolGroup_OS-opt 4.0G 70M 3.7G 2% /opt /dev/mapper/VolGroup_OS-tmp 4.0G 34M 3.8G 1% /tmp /dev/mapper/VolGroup_OS-usr 12G 6.9G 4.5G 61% /usr /dev/mapper/VolGroup_OS-var 2.0G 189M 1.7G 10% /var [root@blade6 tmp]# ls -l /dev/mapper total 0 crw------- 1 root root 10, 63 Dec 23 08:30 control brw-rw---- 1 root disk 253, 5 Dec 23 08:30 VolGroup_OS-home brw-rw---- 1 root disk 253, 1 Dec 23 08:30 VolGroup_OS-opt brw-rw---- 1 root disk 253, 0 Dec 23 08:30 VolGroup_OS-root brw-rw---- 1 root disk 253, 2 Dec 23 08:30 VolGroup_OS-tmp brw-rw---- 1 root disk 253, 3 Dec 23 08:30 VolGroup_OS-usr brw-rw---- 1 root disk 253, 4 Dec 23 08:30 VolGroup_OS-var [root@blade6 tmp]# dmsetup ls VolGroup_OS-opt (253, 1) VolGroup_OS-home (253, 5) VolGroup_OS-tmp (253, 2) VolGroup_OS-root (253, 0) VolGroup_OS-usr (253, 3) VolGroup_OS-var (253, 4) >That's what multipath does, too, to construct the multipath maps. >-- >Axel.Thimm at ATrpms.net >