test

Successful Failover Test Results – Multipathing on Intel Modular Multi-Flex Server with Citrix Xenserver 6.0.2

Initial setup

 

Both Storage Controller Modules attached and Intel Modular Server running on Citrix Xenserver 6.0.2 with patched dh_scsi_alua driver.

See my other blog post on how to set up stable multipathing on IMS with Citrix Xenserver 6.0.2

Initial multipath state:

[root@xenserver1 ~]# multipath -ll
22202000155fdb615 dm-0 Intel,Multi-Flex
[size=100G][features=1 queue_if_no_path][hwhandler=1 alua][rw]
\_ round-robin 0 [prio=50][active]
\_ 0:0:0:0 sda 8:0 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 0:0:1:0 sdb 8:16 [active][ready]

Note that I waited around 5 minutes after each SCM pull out / push back in order to let the system stabilize.

I did not reboot the host between these events.

Pull out SCM2 for the first time

 

Messages:

Apr 7 17:26:35 xenserver1 kernel: sd 0:0:1:0: [sdb] Synchronizing SCSI cache
Apr 7 17:26:35 xenserver1 kernel: sd 0:0:1:0: [sdb] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Apr 7 17:26:35 xenserver1 kernel: mptsas: ioc0: delete expander: num_phys 25, sas_addr (0x5001e671d12253ff)
Apr 7 17:26:35 xenserver1 kernel: mptbase: ioc0: LogInfo(0x31140000): Originator={PL}, Code={IO Executed}, SubCode(0x0000)
Apr 7 17:26:35 xenserver1 kernel: mptscsih: ioc0: ERROR – Received a mf that was already freed
Apr 7 17:26:35 xenserver1 kernel: mptscsih: ioc0: ERROR – req_idx=beaf req_idx_MR=43 mf=ec984980 mr=ec9825d0 sc=(null)
Apr 7 17:26:35 xenserver1 multipathd: sdb: remove path (operator)
Apr 7 17:26:35 xenserver1 multipathd: 22202000155fdb615: load table [0 209715200 multipath 1 queue_if_no_path 1 alua 1 1 round-robin 0 1 1 8:0 100]
Apr 7 17:26:35 xenserver1 multipathd: Path event for 22202000155fdb615, calling mpathcount
Apr 7 17:26:42 xenserver1 kernel: sd 0:0:0:0: alua: port group 00 state A supports touSnA

Multipath state:

[root@xenserver1 ~]# multipath -ll
22202000155fdb615 dm-0 Intel,Multi-Flex
[size=100G][features=1 queue_if_no_path][hwhandler=1 alua][rw]
\_ round-robin 0 [prio=50][active]
\_ 0:0:0:0 sda 8:0 [active][ready]

 

Push back SCM2 for the first time

 

Messages:

Apr 7 17:34:26 xenserver1 kernel: mptsas: ioc0: add expander: num_phys 25, sas_addr (0x5001e671d12253ff)
Apr 7 17:34:27 xenserver1 kernel: mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 1, phy 11, sas_addr 0x500015500002050a
Apr 7 17:34:27 xenserver1 kernel: target0:0:2: mptsas: ioc0: add device: fw_channel 0, fw_id 1, phy 11, sas_addr 0x500015500002050a
Apr 7 17:34:27 xenserver1 kernel: scsi 0:0:2:0: Direct-Access Intel Multi-Flex 0308 PQ: 0 ANSI: 5
Apr 7 17:34:27 xenserver1 kernel: scsi 0:0:2:0: mptscsih: ioc0: qdepth=64, tagged=1, simple=1, ordered=0, scsi_level=6, cmd_que=1
Apr 7 17:34:27 xenserver1 kernel: scsi 0:0:2:0: alua: supports explicit TPGS
Apr 7 17:34:27 xenserver1 kernel: scsi 0:0:2:0: alua: port group 01 rel port 06
Apr 7 17:34:27 xenserver1 kernel: scsi 0:0:2:0: alua: port group 01 state S supports touSnA
Apr 7 17:34:27 xenserver1 kernel: sd 0:0:2:0: Attached scsi generic sg1 type 0
Apr 7 17:34:27 xenserver1 kernel: sd 0:0:2:0: [sdb] 209715200 512-byte logical blocks: (107 GB/100 GiB)
Apr 7 17:34:27 xenserver1 kernel: sd 0:0:2:0: [sdb] Write Protect is off
Apr 7 17:34:27 xenserver1 kernel: sd 0:0:2:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
Apr 7 17:34:27 xenserver1 kernel: sdb:
Apr 7 17:34:27 xenserver1 kernel: ldm_validate_partition_table(): Disk read failed.
Apr 7 17:34:27 xenserver1 kernel: unable to read partition table
Apr 7 17:34:27 xenserver1 kernel: sd 0:0:2:0: [sdb] Attached SCSI disk
Apr 7 17:34:29 xenserver1 multipathd: sdb: add path (operator)
Apr 7 17:34:29 xenserver1 multipathd: 22202000155fdb615: load table [0 209715200 multipath 1 queue_if_no_path 1 alua 2 1 round-robin 0 1 1 8:0 100 round-robin 0 1 1
Apr 7 17:34:29 xenserver1 multipathd: Path event for 22202000155fdb615, calling mpathcount
Apr 7 17:34:43 xenserver1 kernel: sd 0:0:0:0: alua: port group 00 state A supports touSnA

Multipath state:

22202000155fdb615 dm-0 Intel,Multi-Flex
[size=100G][features=1 queue_if_no_path][hwhandler=1 alua][rw]
\_ round-robin 0 [prio=50][active]
\_ 0:0:0:0 sda 8:0 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 0:0:2:0 sdb 8:16 [active][ready]

 

Pull out SCM2 for the second time

 

Messages:

Apr 7 17:39:13 xenserver1 kernel: sd 0:0:2:0: [sdb] Synchronizing SCSI cache
Apr 7 17:39:13 xenserver1 kernel: sd 0:0:2:0: [sdb] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Apr 7 17:39:13 xenserver1 kernel: mptsas: ioc0: delete expander: num_phys 25, sas_addr (0x5001e671d12253ff)
Apr 7 17:39:13 xenserver1 multipathd: sdb: remove path (operator)
Apr 7 17:39:13 xenserver1 multipathd: 22202000155fdb615: load table [0 209715200 multipath 1 queue_if_no_path 1 alua 1 1 round-robin 0 1 1 8:0 100]
Apr 7 17:39:13 xenserver1 multipathd: Path event for 22202000155fdb615, calling mpathcount
Apr 7 17:39:14 xenserver1 kernel: mptbase: ioc0: LogInfo(0x31140000): Originator={PL}, Code={IO Executed}, SubCode(0x0000)
Apr 7 17:39:14 xenserver1 kernel: mptscsih: ioc0: ERROR – Received a mf that was already freed
Apr 7 17:39:14 xenserver1 kernel: mptscsih: ioc0: ERROR – req_idx=beaf req_idx_MR=d2 mf=ec989100 mr=ec9822b0 sc=(null)
Apr 7 17:39:16 xenserver1 kernel: sd 0:0:0:0: alua: port group 00 state A supports touSnA

Multipath state:

22202000155fdb615 dm-0 Intel,Multi-Flex
[size=100G][features=1 queue_if_no_path][hwhandler=1 alua][rw]
\_ round-robin 0 [prio=50][active]
\_ 0:0:0:0 sda 8:0 [active][ready]

Push back SCM2 for the second time

 

Messages:

Apr 7 17:47:36 xenserver1 kernel: mptsas: ioc0: add expander: num_phys 25, sas_addr (0x5001e671d12253ff)
Apr 7 17:47:37 xenserver1 kernel: mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 1, phy 11, sas_addr 0x500015500002050a
Apr 7 17:47:37 xenserver1 kernel: target0:0:3: mptsas: ioc0: add device: fw_channel 0, fw_id 1, phy 11, sas_addr 0x500015500002050a
Apr 7 17:47:37 xenserver1 kernel: scsi 0:0:3:0: Direct-Access Intel Multi-Flex 0308 PQ: 0 ANSI: 5
Apr 7 17:47:37 xenserver1 kernel: scsi 0:0:3:0: mptscsih: ioc0: qdepth=64, tagged=1, simple=1, ordered=0, scsi_level=6, cmd_que=1
Apr 7 17:47:37 xenserver1 kernel: scsi 0:0:3:0: alua: supports explicit TPGS
Apr 7 17:47:37 xenserver1 kernel: scsi 0:0:3:0: alua: port group 01 rel port 06
Apr 7 17:47:37 xenserver1 kernel: scsi 0:0:3:0: alua: port group 01 state S supports touSnA
Apr 7 17:47:37 xenserver1 kernel: sd 0:0:3:0: Attached scsi generic sg1 type 0
Apr 7 17:47:37 xenserver1 kernel: sd 0:0:3:0: [sdb] 209715200 512-byte logical blocks: (107 GB/100 GiB)
Apr 7 17:47:37 xenserver1 kernel: sd 0:0:3:0: [sdb] Write Protect is off
Apr 7 17:47:37 xenserver1 kernel: sd 0:0:3:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
Apr 7 17:47:37 xenserver1 kernel: sdb:
Apr 7 17:47:37 xenserver1 kernel: ldm_validate_partition_table(): Disk read failed.
Apr 7 17:47:37 xenserver1 kernel: unable to read partition table
Apr 7 17:47:37 xenserver1 kernel: sd 0:0:3:0: [sdb] Attached SCSI disk
Apr 7 17:47:39 xenserver1 multipathd: sdb: add path (operator)
Apr 7 17:47:39 xenserver1 multipathd: 22202000155fdb615: load table [0 209715200 multipath 1 queue_if_no_path 1 alua 2 1 round-robin 0 1 1 8:0 100 round-robin 0 1 1
Apr 7 17:47:39 xenserver1 multipathd: Path event for 22202000155fdb615, calling mpathcount
Apr 7 17:47:53 xenserver1 kernel: sd 0:0:0:0: alua: port group 00 state A supports touSnA

Multipath state:

22202000155fdb615 dm-0 Intel,Multi-Flex
[size=100G][features=1 queue_if_no_path][hwhandler=1 alua][rw]
\_ round-robin 0 [prio=50][active]
\_ 0:0:0:0 sda 8:0 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 0:0:3:0 sdb 8:16 [active][ready]

 

Pull out SCM1 for the first time

 

Messages:

Apr 7 17:52:54 xenserver1 multipathd: sda3: remove path (operator)
Apr 7 17:52:54 xenserver1 multipathd: sda3: spurious uevent, path not in pathvec
Apr 7 17:52:54 xenserver1 multipathd: sda2: remove path (operator)
Apr 7 17:52:54 xenserver1 multipathd: sda2: spurious uevent, path not in pathvec
Apr 7 17:52:54 xenserver1 multipathd: sda1: remove path (operator)
Apr 7 17:52:54 xenserver1 multipathd: sda1: spurious uevent, path not in pathvec
Apr 7 17:52:54 xenserver1 kernel: sd 0:0:0:0: [sda] Synchronizing SCSI cache
Apr 7 17:52:54 xenserver1 kernel: sd 0:0:0:0: [sda] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Apr 7 17:52:54 xenserver1 kernel: mptsas: ioc0: delete expander: num_phys 25, sas_addr (0x5001e671d12252ff)
Apr 7 17:52:55 xenserver1 kernel: mptbase: ioc0: LogInfo(0x31140000): Originator={PL}, Code={IO Executed}, SubCode(0x0000)
Apr 7 17:52:55 xenserver1 kernel: mptscsih: ioc0: ERROR – Received a mf that was already freed
Apr 7 17:52:55 xenserver1 kernel: mptscsih: ioc0: ERROR – req_idx=beaf req_idx_MR=6d mf=ec985e80 mr=ec9817c0 sc=(null)
Apr 7 17:52:55 xenserver1 multipathd: sda: remove path (operator)
Apr 7 17:52:55 xenserver1 multipathd: 22202000155fdb615: load table [0 209715200 multipath 1 queue_if_no_path 1 alua 1 1 round-robin 0 1 1 8:16 100]
Apr 7 17:52:55 xenserver1 multipathd: Path event for 22202000155fdb615, calling mpathcount
Apr 7 17:52:56 xenserver1 kernel: sd 0:0:3:0: alua: port group 01 state S supports touSnA
Apr 7 17:52:56 xenserver1 kernel: sd 0:0:3:0: alua: port group 01 switched to state A

Multipath state:

22202000155fdb615 dm-0 Intel,Multi-Flex
[size=100G][features=1 queue_if_no_path][hwhandler=1 alua][rw]
\_ round-robin 0 [prio=1][active]
\_ 0:0:3:0 sdb 8:16 [active][ready]

Push back SCM1 for the first time

 

Messages:
Apr 7 18:03:40 xenserver1 kernel: mptsas: ioc0: add expander: num_phys 25, sas_addr (0x5001e671d12252ff)
Apr 7 18:03:41 xenserver1 kernel: mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 0, phy 11, sas_addr 0x500015500002040a
Apr 7 18:03:41 xenserver1 kernel: target0:0:4: mptsas: ioc0: add device: fw_channel 0, fw_id 0, phy 11, sas_addr 0x500015500002040a
Apr 7 18:03:41 xenserver1 kernel: scsi 0:0:4:0: Direct-Access Intel Multi-Flex 0308 PQ: 0 ANSI: 5
Apr 7 18:03:41 xenserver1 kernel: scsi 0:0:4:0: mptscsih: ioc0: qdepth=64, tagged=1, simple=1, ordered=0, scsi_level=6, cmd_que=1
Apr 7 18:03:41 xenserver1 kernel: scsi 0:0:4:0: alua: supports explicit TPGS
Apr 7 18:03:41 xenserver1 kernel: scsi 0:0:4:0: alua: port group 00 rel port 03
Apr 7 18:03:41 xenserver1 kernel: scsi 0:0:4:0: alua: port group 00 state S supports touSnA
Apr 7 18:03:41 xenserver1 kernel: sd 0:0:4:0: Attached scsi generic sg0 type 0
Apr 7 18:03:41 xenserver1 kernel: sd 0:0:4:0: [sda] 209715200 512-byte logical blocks: (107 GB/100 GiB)
Apr 7 18:03:41 xenserver1 kernel: sd 0:0:4:0: [sda] Write Protect is off
Apr 7 18:03:41 xenserver1 kernel: sd 0:0:4:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
Apr 7 18:03:41 xenserver1 kernel: sda:
Apr 7 18:03:41 xenserver1 kernel: ldm_validate_partition_table(): Disk read failed.
Apr 7 18:03:41 xenserver1 kernel: unable to read partition table
Apr 7 18:03:41 xenserver1 kernel: sd 0:0:4:0: [sda] Attached SCSI disk
Apr 7 18:03:43 xenserver1 multipathd: sda: add path (operator)
Apr 7 18:03:43 xenserver1 multipathd: 22202000155fdb615: load table [0 209715200 multipath 1 queue_if_no_path 1 alua 2 1 round-robin 0 1 1 8:0 100 round-robin 0 1 1
Apr 7 18:03:43 xenserver1 multipathd: Path event for 22202000155fdb615, calling mpathcount
Apr 7 18:03:57 xenserver1 kernel: sd 0:0:4:0: alua: port group 00 state S supports touSnA
Apr 7 18:03:57 xenserver1 kernel: sd 0:0:4:0: alua: port group 00 switched to state A

Multipath state:

22202000155fdb615 dm-0 Intel,Multi-Flex
[size=100G][features=1 queue_if_no_path][hwhandler=1 alua][rw]
\_ round-robin 0 [prio=50][active]
\_ 0:0:4:0 sda 8:0 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 0:0:3:0 sdb 8:16 [active][ready]

Pull out SCM1 for the second time

 

Messages:

Apr 7 18:09:31 xenserver1 kernel: sd 0:0:4:0: [sda] Synchronizing SCSI cache
Apr 7 18:09:31 xenserver1 kernel: sd 0:0:4:0: [sda] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Apr 7 18:09:31 xenserver1 kernel: mptsas: ioc0: delete expander: num_phys 25, sas_addr (0x5001e671d12252ff)
Apr 7 18:09:31 xenserver1 multipathd: sda: remove path (operator)
Apr 7 18:09:31 xenserver1 multipathd: 22202000155fdb615: load table [0 209715200 multipath 1 queue_if_no_path 1 alua 1 1 round-robin 0 1 1 8:16 100]
Apr 7 18:09:31 xenserver1 multipathd: Path event for 22202000155fdb615, calling mpathcount
Apr 7 18:09:37 xenserver1 kernel: sd 0:0:3:0: alua: port group 01 state A supports touSnA

Multipath state:

22202000155fdb615 dm-0 Intel,Multi-Flex
[size=100G][features=1 queue_if_no_path][hwhandler=1 alua][rw]
\_ round-robin 0 [prio=1][active]
\_ 0:0:3:0 sdb 8:16 [active][ready]

 

Push back SCM1 for the second time

 

Messages:

Apr 7 18:17:55 xenserver1 kernel: mptsas: ioc0: add expander: num_phys 25, sas_addr (0x5001e671d12252ff)
Apr 7 18:17:57 xenserver1 kernel: mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 0, phy 11, sas_addr 0x500015500002040a
Apr 7 18:17:57 xenserver1 kernel: target0:0:5: mptsas: ioc0: add device: fw_channel 0, fw_id 0, phy 11, sas_addr 0x500015500002040a
Apr 7 18:17:57 xenserver1 kernel: scsi 0:0:5:0: Direct-Access Intel Multi-Flex 0308 PQ: 0 ANSI: 5
Apr 7 18:17:57 xenserver1 kernel: scsi 0:0:5:0: mptscsih: ioc0: qdepth=64, tagged=1, simple=1, ordered=0, scsi_level=6, cmd_que=1
Apr 7 18:17:57 xenserver1 kernel: scsi 0:0:5:0: alua: supports explicit TPGS
Apr 7 18:17:57 xenserver1 kernel: scsi 0:0:5:0: alua: port group 00 rel port 03
Apr 7 18:17:57 xenserver1 kernel: scsi 0:0:5:0: alua: port group 00 state S supports touSnA
Apr 7 18:17:57 xenserver1 kernel: sd 0:0:5:0: Attached scsi generic sg0 type 0
Apr 7 18:17:57 xenserver1 kernel: sd 0:0:5:0: [sda] 209715200 512-byte logical blocks: (107 GB/100 GiB)
Apr 7 18:17:57 xenserver1 kernel: sd 0:0:5:0: [sda] Write Protect is off
Apr 7 18:17:57 xenserver1 kernel: sd 0:0:5:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
Apr 7 18:17:57 xenserver1 kernel: sda:
Apr 7 18:17:57 xenserver1 kernel: ldm_validate_partition_table(): Disk read failed.
Apr 7 18:17:57 xenserver1 kernel: unable to read partition table
Apr 7 18:17:57 xenserver1 kernel: sd 0:0:5:0: [sda] Attached SCSI disk
Apr 7 18:17:58 xenserver1 multipathd: sda: add path (operator)
Apr 7 18:17:58 xenserver1 multipathd: 22202000155fdb615: load table [0 209715200 multipath 1 queue_if_no_path 1 alua 2 1 round-robin 0 1 1 8:0 100 round-robin 0 1 1
Apr 7 18:17:58 xenserver1 multipathd: Path event for 22202000155fdb615, calling mpathcount
Apr 7 18:18:06 xenserver1 kernel: sd 0:0:5:0: alua: port group 00 state S supports touSnA
Apr 7 18:18:06 xenserver1 kernel: sd 0:0:5:0: alua: port group 00 switched to state A

Multipath state:

22202000155fdb615 dm-0 Intel,Multi-Flex
[size=100G][features=1 queue_if_no_path][hwhandler=1 alua][rw]
\_ round-robin 0 [prio=50][active]
\_ 0:0:5:0 sda 8:0 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 0:0:3:0 sdb 8:16 [active][ready]