fcsp re-authentication failures when Port VSAN not allowed on Port-Channel

MDS1# show run int po1version 3.3(5)interface port-channel 1fspf cost 100 vsan 20switchport speed 1000switchport mode Eno shutdownchannel mode activeswitchport trunk allowed vsan add 5switchport trunk allowed vsan add 20switchport trunk allowed vsan add 30MDS1# show run int fc1/9version 3.3(5)interface fc1/9switchport speed 1000switchport mode Echannel-group 1 forcefcsp auto-active 1no shutdown And here is the output of the other side of the link:

MDS2# show run int po1version 3.3(5)interface port-channel 1fspf cost 100 vsan 20switchport speed 1000switchport mode Eno shutdownchannel mode activeswitchport trunk allowed vsan add 5switchport trunk allowed vsan add 20switchport trunk allowed vsan add 30MDS2# show run int fc1/9version 3.3(5)interface fc1/9switchport speed 1000switchport mode Echannel-group 1 forcefcsp auto-passiveno shutdown So you can see the MDS1 side is set to "auto-active" with a re-authentication time of 1 minute, and the other side is auto-passive. When both sides are set to auto-active, I do not see this issue. You will also see that on the Port Channel Trunk I am allowing VSAN's 5, 20 and 30. The port VSAN I have set for this trunk (not shown) is VSAN 2. What happens is that the initial authentication works just fine:

MDS1# show fcsp interf fc1/9-11fc1/9:fcsp authentication mode:SEC_MODE_AUTO_ACTIVEreauthentication timeout (in minutes):1Status:Successfully authenticatedAuthenticated using local password databasefc1/10:fcsp authentication mode:SEC_MODE_AUTO_ACTIVEreauthentication timeout (in minutes):1Status:Successfully authenticatedAuthenticated using local password databasefc1/11:fcsp authentication mode:SEC_MODE_AUTO_ACTIVEreauthentication timeout (in minutes):1Status:Successfully authenticatedAuthenticated using local password database After one minute however, the re-authentication fails:

2010 Aug 6 20:27:03 MDS1 %FCSP-MGR-2-FCSP_AUTHENT_FAILURE: FC-SP Authentication failure on Port fc1/10 (FC-SP Failure Reason: FCSP_AUTHENT_FAILURE )2010 Aug 6 20:27:03 MDS1 %FCSP-MGR-2-FCSP_AUTHENT_FAILURE: FC-SP Authentication failure on Port fc1/11 (FC-SP Failure Reason: FCSP_AUTHENT_FAILURE )2010 Aug 6 20:27:03 MDS1 %FCSP-MGR-2-FCSP_AUTHENT_FAILURE: FC-SP Authentication failure on Port fc1/9 (FC-SP Failure Reason: FCSP_AUTHENT_FAILURE ) Simply bouncing the physical interfaces you wish to authenticate will allow successful authentication since it treats it as an initial authentication, but then it will fail again in one minute:

MDS1# show fcsp interf fc1/9-11fc1/9:fcsp authentication mode:SEC_MODE_AUTO_ACTIVEreauthentication timeout (in minutes):1Status:FC-SP authentication failedfc1/10:fcsp authentication mode:SEC_MODE_AUTO_ACTIVEreauthentication timeout (in minutes):1Status:FC-SP authentication failedfc1/11:fcsp authentication mode:SEC_MODE_AUTO_ACTIVEreauthentication timeout (in minutes):1Status:FC-SP authentication failed Here you can see with fcanalyzer, that the re-auths never make it to the other side:

MDS1(config)# fcanalyzer local brief display-filter fcsp.opcodeWarning: Couldn't obtain netmask info (eth2: no IPv4 address assigned).Capturing on eth211.900199 ff.ff.fd -> ff.ff.fd 0x6ab 0xffff SW_ILS AUTH_Negotiate11.992773 ff.ff.fd -> ff.ff.fd 0x698 0xffff SW_ILS DHCHAP_Challenge12.185244 ff.ff.fd -> ff.ff.fd 0x6b4 0xffff SW_ILS DHCHAP_Reply12.193308 ff.ff.fd -> ff.ff.fd 0x6b5 0xffff SW_ILS AUTH_Negotiate12.204058 ff.ff.fd -> ff.ff.fd 0x6b6 0xffff SW_ILS AUTH_Negotiate12.374826 ff.ff.fd -> ff.ff.fd 0x6a2 0xffff SW_ILS DHCHAP_Challenge12.381704 ff.ff.fd -> ff.ff.fd 0x6a3 0xffff SW_ILS DHCHAP_Success12.474281 ff.ff.fd -> ff.ff.fd 0x6a4 0xffff SW_ILS DHCHAP_Challenge12.556602 ff.ff.fd -> ff.ff.fd 0x6ba 0xffff SW_ILS DHCHAP_Reply12.571106 ff.ff.fd -> ff.ff.fd 0x6bb 0xffff SW_ILS DHCHAP_Success12.668658 ff.ff.fd -> ff.ff.fd 0x6a7 0xffff SW_ILS DHCHAP_Success12.754110 ff.ff.fd -> ff.ff.fd 0x6bd 0xffff SW_ILS DHCHAP_Reply12.820886 ff.ff.fd -> ff.ff.fd 0x6c0 0xffff SW_ILS DHCHAP_Success12.958055 ff.ff.fd -> ff.ff.fd 0x6ac 0xffff SW_ILS DHCHAP_Success13.015277 ff.ff.fd -> ff.ff.fd 0x6c3 0xffff SW_ILS DHCHAP_Success72.763784 ff.ff.fd -> ff.ff.fd 0x6c7 0xffff SW_ILS AUTH_Negotiate73.003558 ff.ff.fd -> ff.ff.fd 0x6c8 0xffff SW_ILS AUTH_Negotiate73.063469 ff.ff.fd -> ff.ff.fd 0x6c9 0xffff SW_ILS AUTH_Negotiate76.763897 ff.ff.fd -> ff.ff.fd 0x6ca 0xffff SW_ILS AUTH_Negotiate77.004037 ff.ff.fd -> ff.ff.fd 0x6cb 0xffff SW_ILS AUTH_Negotiate77.064184 ff.ff.fd -> ff.ff.fd 0x6cc 0xffff SW_ILS AUTH_Negotiate80.763286 ff.ff.fd -> ff.ff.fd 0x6cd 0xffff SW_ILS AUTH_Negotiate81.003504 ff.ff.fd -> ff.ff.fd 0x6ce 0xffff SW_ILS AUTH_Negotiate81.063660 ff.ff.fd -> ff.ff.fd 0x6cf 0xffff SW_ILS AUTH_Negotiate84.762893 ff.ff.fd -> ff.ff.fd 0x6d0 0xffff SW_ILS AUTH_Negotiate85.002943 ff.ff.fd -> ff.ff.fd 0x6d1 0xffff SW_ILS AUTH_Negotiate85.062965 ff.ff.fd -> ff.ff.fd 0x6d2 0xffff SW_ILS AUTH_Negotiate88.762322 ff.ff.fd -> ff.ff.fd 0x6d3 0xffff SW_ILS AUTH_Negotiate89.002531 ff.ff.fd -> ff.ff.fd 0x6d4 0xffff SW_ILS AUTH_Negotiate89.063700 ff.ff.fd -> ff.ff.fd 0x6d5 0xffff SW_ILS AUTH_Negotiate92.762232 ff.ff.fd -> ff.ff.fd 0x6d6 0xffff SW_ILS AUTH_Negotiate93.002584 ff.ff.fd -> ff.ff.fd 0x6d7 0xffff SW_ILS AUTH_Negotiate93.062299 ff.ff.fd -> ff.ff.fd 0x6d8 0xffff SW_ILS AUTH_Negotiate96.761330 ff.ff.fd -> ff.ff.fd 0x6d9 0xffff SW_ILS AUTH_Negotiate97.001363 ff.ff.fd -> ff.ff.fd 0x6da 0xffff SW_ILS AUTH_Negotiate97.061330 ff.ff.fd -> ff.ff.fd 0x6db 0xffff SW_ILS AUTH_Negotiate100.760808 ff.ff.fd -> ff.ff.fd 0x6dc 0xffff SW_ILS AUTH_Negotiate101.000880 ff.ff.fd -> ff.ff.fd 0x6dd 0xffff SW_ILS AUTH_Negotiate101.060861 ff.ff.fd -> ff.ff.fd 0x6de 0xffff SW_ILS AUTH_Negotiate104.760385 ff.ff.fd -> ff.ff.fd 0x6df 0xffff SW_ILS AUTH_Negotiate105.000525 ff.ff.fd -> ff.ff.fd 0x6e0 0xffff SW_ILS AUTH_Negotiate105.060406 ff.ff.fd -> ff.ff.fd 0x6e1 0xffff SW_ILS AUTH_Negotiate108.759895 ff.ff.fd -> ff.ff.fd 0x6e2 0xffff SW_ILS AUTH_Negotiate109.000001 ff.ff.fd -> ff.ff.fd 0x6e3 0xffff SW_ILS AUTH_Negotiate109.059927 ff.ff.fd -> ff.ff.fd 0x6e4 0xffff SW_ILS AUTH_Negotiate2010 Aug 6 20:52:29 MDS1 %FCSP-MGR-2-FCSP_AUTHENT_FAILURE: FC-SP Authentication failure on Port fc1/10 (FC-SP Failure Reason: FCSP_AUTHENT_FAILURE )2010 Aug 6 20:52:29 MDS1 %FCSP-MGR-2-FCSP_AUTHENT_FAILURE: FC-SP Authentication failure on Port fc1/9 (FC-SP Failure Reason: FCSP_AUTHENT_FAILURE )2010 Aug 6 20:52:29 MDS1 %FCSP-MGR-2-FCSP_AUTHENT_FAILURE: FC-SP Authentication failure on Port fc1/11 (FC-SP Failure Reason: FCSP_AUTHENT_FAILURE ) and here is MDS2:

MDS2(config)# fcanalyzer local brief display-filter fcsp.opcodeWarning: Couldn't obtain netmask info (eth2: no IPv4 address assigned).Capturing on eth29.466279 ff.ff.fd -> ff.ff.fd 0x6ab 0xffff SW_ILS AUTH_Negotiate9.558533 ff.ff.fd -> ff.ff.fd 0x698 0xffff SW_ILS DHCHAP_Challenge9.751347 ff.ff.fd -> ff.ff.fd 0x6b4 0xffff SW_ILS DHCHAP_Reply9.759394 ff.ff.fd -> ff.ff.fd 0x6b5 0xffff SW_ILS AUTH_Negotiate9.770069 ff.ff.fd -> ff.ff.fd 0x6b6 0xffff SW_ILS AUTH_Negotiate9.940639 ff.ff.fd -> ff.ff.fd 0x6a2 0xffff SW_ILS DHCHAP_Challenge9.947458 ff.ff.fd -> ff.ff.fd 0x6a3 0xffff SW_ILS DHCHAP_Success10.040026 ff.ff.fd -> ff.ff.fd 0x6a4 0xffff SW_ILS DHCHAP_Challenge10.122701 ff.ff.fd -> ff.ff.fd 0x6ba 0xffff SW_ILS DHCHAP_Reply10.137177 ff.ff.fd -> ff.ff.fd 0x6bb 0xffff SW_ILS DHCHAP_Success10.234418 ff.ff.fd -> ff.ff.fd 0x6a7 0xffff SW_ILS DHCHAP_Success10.320151 ff.ff.fd -> ff.ff.fd 0x6bd 0xffff SW_ILS DHCHAP_Reply10.387025 ff.ff.fd -> ff.ff.fd 0x6c0 0xffff SW_ILS DHCHAP_Success10.523860 ff.ff.fd -> ff.ff.fd 0x6ac 0xffff SW_ILS DHCHAP_Success10.581328 ff.ff.fd -> ff.ff.fd 0x6c3 0xffff SW_ILS DHCHAP_Success You can see from comparing the two fcanalyzer outputs that on MDS1, the last bit of data to go between the switches successfully was at timestamp 13.015277. I was also able to do a full packet dump using my PAA-2 and provide the pcap to TAC to be analyzed. As I get more information I will post it in this thread.