On Sat, 24 Dec 2005 15:42:14 +0100, Axel Thimm <[log in to unmask]> wrote:
>On Thu, Dec 22, 2005 at 09:22:10PM -0600, Jeffry R. Abramson wrote:
>> Speaking of SAN's, has anyone gotten the multipath feature in SL42 to work?
>> We have an IBM blade equipped with a dual-port Qlogic controller connected
>> through two independent fabrics to a DS400/Adaptec array with a dual-port
>> controller. fdisk -l shows two paths to every logical array, ie. each disk
>> shows up twice.
>
>You should be seeing more than that, you have at least four paths
>(2xHBA ports x 2xcontroller ports).
The disk array has a A/B pair of controllers in an active/passive
configuration. We have zoned out the passive controller for now so only the
A controller is used. Each disk controller has two FC ports configured as
follows:
----[ Initiator 'blade6_FC0' ]------------------------------------------
WWN: 21000011259350e8
Name: blade6_FC0
ID: 0
Port access:
Port A0 Port A1 Port B0 Port B1
---------------------------------------------------------------------------
YES NO YES NO
LUN Mappings:
LUN logical [Capacity]
---------------------------------------------------------------------------
0 Device-1 [683.48GB]
1 Device-2 [683.48GB]
----[ Initiator 'blade6_FC1' ]------------------------------------------
WWN: 21000011259350e9
Name: blade6_FC1
ID: 2
Port access:
Port A0 Port A1 Port B0 Port B1
---------------------------------------------------------------------------
NO YES NO YES
LUN Mappings:
LUN logical [Capacity]
---------------------------------------------------------------------------
0 Device-1 [683.48GB]
1 Device-2 [683.48GB]
The two san fabrics are completely independent so with this configuration
there should be two paths to each logical device. fdisk -l yields the
following:
Disk /dev/sda: 73.4 GB, 73407488000 bytes
255 heads, 63 sectors/track, 8924 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 33 265041 83 Linux
/dev/sda2 34 1077 8385930 83 Linux
/dev/sda3 1078 8924 63031027+ 8e Linux LVM
Disk /dev/sdb: 73.4 GB, 73407488000 bytes
255 heads, 63 sectors/track, 8924 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 8924 71681998+ 83 Linux
Disk /dev/sdc: 733.8 GB, 733877370880 bytes
255 heads, 63 sectors/track, 89222 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdc1 1 89222 716675683+ 8e Linux LVM
Disk /dev/sdd: 733.8 GB, 733877370880 bytes
255 heads, 63 sectors/track, 89222 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdd1 1 44611 358337826 8e Linux LVM
/dev/sdd2 44612 89222 358337857+ 8e Linux LVM
Disk /dev/sde: 733.8 GB, 733877370880 bytes
255 heads, 63 sectors/track, 89222 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sde1 1 89222 716675683+ 8e Linux LVM
Disk /dev/sdf: 733.8 GB, 733877370880 bytes
255 heads, 63 sectors/track, 89222 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdf1 1 44611 358337826 8e Linux LVM
/dev/sdf2 44612 89222 358337857+ 8e Linux LVM
sda and sdb are the internal SCSI drives. sdc through sdf are the two
volumes on the disk array. I put a single partition on one of the volumes
and two on the other so it is easy to see which devices are actually the
same. In this case, sdc and sde refer to the same volume as does sdd and
sdf. /proc/scsi/scsi is as follows:
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: IBM-ESXS Model: ST973401LC FN Rev: B41A
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: IBM-ESXS Model: ST973401LC FN Rev: B41A
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: IBM Model: DS400 S320 Rev: 7.00
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi1 Channel: 00 Id: 00 Lun: 01
Vendor: IBM Model: DS400 S320 Rev: 7.00
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi2 Channel: 00 Id: 00 Lun: 00
Vendor: IBM Model: DS400 S320 Rev: 7.00
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi2 Channel: 00 Id: 00 Lun: 01
Vendor: IBM Model: DS400 S320 Rev: 7.00
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: TEAC Model: FD-05PUB Rev: 3200
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi4 Channel: 00 Id: 00 Lun: 00
Vendor: HL-DT-ST Model: DVD-ROM GDR8082N Rev: 0L03
Type: CD-ROM ANSI SCSI revision: 02
cat /proc/scsi/qla2xxx/1 returns:
QLogic PCI to Fibre Channel Host Adapter for IBM FCEC:
Firmware version 3.03.15 IPX, Driver version 8.01.00b5-rh2
ISP: ISP2312, Serial# N04656
Request Queue = 0x11fd80000, Response Queue = 0x11fd10000
Request Queue count = 4096, Response Queue count = 512
Total number of active commands = 0
Total number of interrupts = 581
Device queue depth = 0x20
Number of free request entries = 3843
Number of mailbox timeouts = 0
Number of ISP aborts = 0
Number of loop resyncs = 0
Number of retries for empty slots = 0
Number of reqs in pending_q= 0, retry_q= 0, done_q= 0, scsi_retry_q= 0
Host adapter:loop state = <READY>, flags = 0x1a03
Dpc flags = 0x0
MBX flags = 0x0
Link down Timeout = 030
Port down retry = 030
Login retry count = 030
Commands retried with dropped frame(s) = 0
Product ID = 4953 5020 2020 0002
SCSI Device Information:
scsi-qla0-adapter-node=20000011259350e8;
scsi-qla0-adapter-port=21000011259350e8;
scsi-qla0-target-0=21000000d12671dd;
FC Port Information:
scsi-qla0-port-0=20000000d12671dd:21000000d12671dd:010f00:81;
SCSI LUN Information:
(Id:Lun) * - indicates lun is not registered with the OS.
( 0: 0): Total reqs 72, Pending reqs 0, flags 0x0, 0:0:81 00
( 0: 1): Total reqs 107, Pending reqs 0, flags 0x0, 0:0:81 00
cat /proc/scsi/qla2xxx/2 returns:
QLogic PCI to Fibre Channel Host Adapter for IBM FCEC:
Firmware version 3.03.15 IPX, Driver version 8.01.00b5-rh2
ISP: ISP2312, Serial# N04912
Request Queue = 0xdfc00000, Response Queue = 0x11fce0000
Request Queue count = 4096, Response Queue count = 512
Total number of active commands = 0
Total number of interrupts = 580
Device queue depth = 0x20
Number of free request entries = 3830
Number of mailbox timeouts = 0
Number of ISP aborts = 0
Number of loop resyncs = 0
Number of retries for empty slots = 0
Number of reqs in pending_q= 0, retry_q= 0, done_q= 0, scsi_retry_q= 0
Host adapter:loop state = <READY>, flags = 0x1a03
Dpc flags = 0x0
MBX flags = 0x0
Link down Timeout = 030
Port down retry = 030
Login retry count = 030
Commands retried with dropped frame(s) = 0
Product ID = 4953 5020 2020 0002
SCSI Device Information:
scsi-qla1-adapter-node=20000011259350e9;
scsi-qla1-adapter-port=21000011259350e9;
scsi-qla1-target-0=21010000d12671dd;
FC Port Information:
scsi-qla1-port-0=20000000d12671dd:21010000d12671dd:010f00:81;
SCSI LUN Information:
(Id:Lun) * - indicates lun is not registered with the OS.
( 0: 0): Total reqs 72, Pending reqs 0, flags 0x0, 1:0:81 00
( 0: 1): Total reqs 107, Pending reqs 0, flags 0x0, 1:0:81 00
>
>> The instructions make it look easy (comment everything out
>> from /etc/multipath.conf, start up multipathd and away you go). Running
>> multipath -v2 -l shows nothing. multipath -v3 shows that it sees all of the
>> devices but I don't know how to decipher where it is going wrong (I'd attach
>> the output but I'm typing this from home). Ideas? -- Jeff
>
>Use multipath -F and try again.
>
Ran multipath -F followed by multipath -v2 and got the same results.
multipath -v3 reveals the following:
load path identifiers cache
ux_socket_connect error
===== no paths =====
dm-0 blacklisted
dm-1 blacklisted
dm-2 blacklisted
dm-3 blacklisted
dm-4 blacklisted
dm-5 blacklisted
md0 blacklisted
ram0 blacklisted
ram10 blacklisted
ram11 blacklisted
ram12 blacklisted
ram13 blacklisted
ram14 blacklisted
ram15 blacklisted
ram1 blacklisted
ram2 blacklisted
ram3 blacklisted
ram4 blacklisted
ram5 blacklisted
ram6 blacklisted
ram7 blacklisted
ram8 blacklisted
ram9 blacklisted
path sda not found in pathvec
===== path info sda (mask 0x1f) =====
bus = 1
dev_t = 8:0
size = 143374000
vendor = IBM-ESXS
product = ST973401LC FN
rev = B41A
h:b:t:l = 0:0:0:0
tgt_node_name =
serial = 3LB06LLL00007603NPNN
path checker = readsector0 (internal default)
state = 2
getprio = (null) (internal default)
prio = 1
getuid = /sbin/scsi_id -g -u -s /block/%n (internal default)
uid = SIBM-ESXSST973401LC____F3LB06LLL00007603NPNN (callout)
path sdb not found in pathvec
===== path info sdb (mask 0x1f) =====
bus = 1
dev_t = 8:16
size = 143374000
vendor = IBM-ESXS
product = ST973401LC FN
rev = B41A
h:b:t:l = 0:0:1:0
tgt_node_name =
serial = 3LB06KLK0000760803PM
path checker = readsector0 (internal default)
state = 2
getprio = (null) (internal default)
prio = 1
getuid = /sbin/scsi_id -g -u -s /block/%n (internal default)
uid = SIBM-ESXSST973401LC____F3LB06KLK0000760803PM (callout)
path sdc not found in pathvec
===== path info sdc (mask 0x1f) =====
bus = 1
dev_t = 8:32
size = 1433354240
vendor = IBM
product = DS400 S320
rev = 7.00
h:b:t:l = 1:0:0:0
tgt_node_name = 0x20000000d12671dd
serial = 00000000002671DD423829CD
path checker = readsector0 (internal default)
state = 2
getprio = (null) (internal default)
prio = 1
getuid = /sbin/scsi_id -g -u -s /block/%n (internal default)
uid = 20000d12671dd29cd (callout)
path sdd not found in pathvec
===== path info sdd (mask 0x1f) =====
bus = 1
dev_t = 8:48
size = 1433354240
vendor = IBM
product = DS400 S320
rev = 7.00
h:b:t:l = 1:0:0:1
tgt_node_name = 0x20000000d12671dd
serial = 00000000002671DD423829CE
path checker = readsector0 (internal default)
state = 2
getprio = (null) (internal default)
prio = 1
getuid = /sbin/scsi_id -g -u -s /block/%n (internal default)
uid = 20000d12671dd29ce (callout)
path sde not found in pathvec
===== path info sde (mask 0x1f) =====
bus = 1
dev_t = 8:64
size = 1433354240
vendor = IBM
product = DS400 S320
rev = 7.00
h:b:t:l = 2:0:0:0
tgt_node_name = 0x20000000d12671dd
serial = 00000000002671DD423829CD
path checker = readsector0 (internal default)
state = 2
getprio = (null) (internal default)
prio = 1
getuid = /sbin/scsi_id -g -u -s /block/%n (internal default)
uid = 20000d12671dd29cd (callout)
path sdf not found in pathvec
===== path info sdf (mask 0x1f) =====
bus = 1
dev_t = 8:80
size = 1433354240
vendor = IBM
product = DS400 S320
rev = 7.00
h:b:t:l = 2:0:0:1
tgt_node_name = 0x20000000d12671dd
serial = 00000000002671DD423829CE
path checker = readsector0 (internal default)
state = 2
getprio = (null) (internal default)
prio = 1
getuid = /sbin/scsi_id -g -u -s /block/%n (internal default)
uid = 20000d12671dd29ce (callout)
path sdg not found in pathvec
===== path info sdg (mask 0x1f) =====
bus = 1
dev_t = 8:96
sr0 blacklisted
What's the significance of the "ux_socket_connect error" ?
>Other than that check with scsi_id that the paths are to the same
>device, e.g.
>
># for x in sdb sdc sdd sde; do /sbin/scsi_id -g -u -s /block/$x; done
>3600508b4000106f30006300000240000
>3600508b4000106f30006300000240000
>3600508b4000106f30006300000240000
>3600508b4000106f30006300000240000
>
for x in sda sdb sdc sdd sde sdf ; do /sbin/scsi_id -g -u -s /block/$x; done
SIBM-ESXSST973401LC____F3LB06LLL00007603NPNN
SIBM-ESXSST973401LC____F3LB06KLK0000760803PM
20000d12671dd29cd
20000d12671dd29ce
20000d12671dd29cd
20000d12671dd29ce
[root@blade6 tmp]# multipath -l
[root@blade6 tmp]# ls -l /dev/dm*
brw------- 1 root root 253, 0 Dec 23 08:30 /dev/dm-0
brw------- 1 root root 253, 1 Dec 23 08:30 /dev/dm-1
brw------- 1 root root 253, 2 Dec 23 08:30 /dev/dm-2
brw------- 1 root root 253, 3 Dec 23 08:30 /dev/dm-3
brw------- 1 root root 253, 4 Dec 23 08:30 /dev/dm-4
brw------- 1 root root 253, 5 Dec 23 08:30 /dev/dm-5
[root@blade6 tmp]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup_OS-root
1008M 227M 731M 24% /
/dev/sda1 251M 17M 221M 8% /boot
none 3.9G 0 3.9G 0% /dev/shm
/dev/mapper/VolGroup_OS-home
1.2G 33M 1.1G 3% /home
/dev/mapper/VolGroup_OS-opt
4.0G 70M 3.7G 2% /opt
/dev/mapper/VolGroup_OS-tmp
4.0G 34M 3.8G 1% /tmp
/dev/mapper/VolGroup_OS-usr
12G 6.9G 4.5G 61% /usr
/dev/mapper/VolGroup_OS-var
2.0G 189M 1.7G 10% /var
[root@blade6 tmp]# ls -l /dev/mapper
total 0
crw------- 1 root root 10, 63 Dec 23 08:30 control
brw-rw---- 1 root disk 253, 5 Dec 23 08:30 VolGroup_OS-home
brw-rw---- 1 root disk 253, 1 Dec 23 08:30 VolGroup_OS-opt
brw-rw---- 1 root disk 253, 0 Dec 23 08:30 VolGroup_OS-root
brw-rw---- 1 root disk 253, 2 Dec 23 08:30 VolGroup_OS-tmp
brw-rw---- 1 root disk 253, 3 Dec 23 08:30 VolGroup_OS-usr
brw-rw---- 1 root disk 253, 4 Dec 23 08:30 VolGroup_OS-var
[root@blade6 tmp]# dmsetup ls
VolGroup_OS-opt (253, 1)
VolGroup_OS-home (253, 5)
VolGroup_OS-tmp (253, 2)
VolGroup_OS-root (253, 0)
VolGroup_OS-usr (253, 3)
VolGroup_OS-var (253, 4)
>That's what multipath does, too, to construct the multipath maps.
>--
>Axel.Thimm at ATrpms.net
>
|
