FCNS inconsistency issues update

So, I got my fcns's all consistant with you on the phone the other day, and now I have let it sit overnight. I wake up today and VSAN 40, which is the VSAN we left the fcroute in, has inconsistant FCNS database, here is the relevant information. Attached to this email is a VSAN topology diagram for VSAN 40. MDS1 MDS1# show run | inc fcroutefcroute 0x10000 0xff0000 interface fc1/14 domain 1 metric 10 vsan 40MDS1# show fcdomain domain-list v 40Number of domains: 4Domain ID WWN--------- -----------------------0xef(239) 20:28:00:0d:ec:0e:b2:41 0x01(1) 20:28:00:0d:ec:19:c9:010xee(238) 20:28:00:0d:ec:10:05:41 0x50(80) 50:00:53:07:ff:f0:00:50 MDS1# show fcdomain domain-list v 40Number of domains: 4Domain ID WWN--------- -----------------------0xef(239) 20:28:00:0d:ec:0e:b2:41 0x01(1) 20:28:00:0d:ec:19:c9:010xee(238) 20:28:00:0d:ec:10:05:41 0x50(80) 50:00:53:07:ff:f0:00:50 MDS1# show fcns internal info v 40Info for vsan 40 =================Interop mode: 0R_A_TOV: 10000; D_S_TOV: 5000Local Domain: 0xef(239) Remote Domains: 0x50(80) 0xee(238) Info for 0x50(80)updating_db = 0refreshing_db = 0num_ports = 0Info for 0xee(238)updating_db = 0refreshing_db = 0num_ports = 0Indexed objects details:port_id index::size:128 incr_factor:128 slots_free:126portwwn index::size:128 incr_factor:128 slots_free:126nodewwn index::size:128 incr_factor:128 slots_free:126ip addr index::size:64 incr_factor:64 slots_free:64total entries in database = 54 MDS2 MDS2# show run | inc fcroutefcroute 0xef0000 0xff0000 interface fc1/15 domain 237 metric 10 remote vsan 40MDS2# show fcdomain domain-list v 40Number of domains: 4Domain ID WWN--------- -----------------------0xef(239) 20:28:00:0d:ec:0e:b2:41 0x01(1) 20:28:00:0d:ec:19:c9:01 0xee(238) 20:28:00:0d:ec:10:05:41 0x50(80) 50:00:53:07:ff:f0:00:50 MDS2# show fcns internal info v 40Info for vsan 40 =================Interop mode: 0R_A_TOV: 10000; D_S_TOV: 5000Local Domain: 0x1(1) Remote Domains: 0x50(80) 0xee(238) Info for 0x50(80)updating_db = 0refreshing_db = 0num_ports = 0Info for 0xee(238)updating_db = 0refreshing_db = 0num_ports = 0Indexed objects details:port_id index::size:128 incr_factor:128 slots_free:127portwwn index::size:128 incr_factor:128 slots_free:127nodewwn index::size:128 incr_factor:128 slots_free:127ip addr index::size:64 incr_factor:64 slots_free:64total entries in database = 52 MDS3 MDS3# show fcdomain domain-list vsan 40Number of domains: 4Domain ID WWN--------- -----------------------0xef(239) 20:28:00:0d:ec:0e:b2:41 0x01(1) 20:28:00:0d:ec:19:c9:010xee(238) 20:28:00:0d:ec:10:05:41 0x50(80) 50:00:53:07:ff:f0:00:50 MDS3# show fcns internal info v 40Info for vsan 40 =================Interop mode: 0R_A_TOV: 10000; D_S_TOV: 5000Local Domain: 0xee(238) Remote Domains: 0x1(1) 0x50(80) 0xef(239) Info for 0x1(1)updating_db = 0refreshing_db = 0num_ports = 0Info for 0x50(80)updating_db = 0refreshing_db = 0num_ports = 0Info for 0xef(239)updating_db = 0refreshing_db = 0num_ports = 0Indexed objects details:port_id index::size:128 incr_factor:128 slots_free:126portwwn index::size:128 incr_factor:128 slots_free:126nodewwn index::size:128 incr_factor:128 slots_free:126ip addr index::size:64 incr_factor:64 slots_free:64total entries in database = 46Some more helpful information:MDS1# show fcroute unicast vsan 40D:direct R:remote P:permanent V:volatile A:active N:non-active # NextProtocol VSAN FC ID/Mask RCtl/Mask Flags Hops Cost-------- ---- -------- -------- ---- ---- ----- ---- ----static 40 0x010000 0xff0000 0x00 0x00 D P A 1 10fspf 40 0x010000 0xff0000 0x00 0x00 D P N 1 500fspf 40 0x500000 0xff0000 0x00 0x00 D P A 1 1fspf 40 0xee0000 0xff0000 0x00 0x00 D P A 1 500local 40 0xef0000 0xffffff 0x00 0x00 D P A 1 1MDS1# show fcroute unicast 0x010000 0xff0000 v 40D:direct R:remote P:permanent V:volatile A:active N:non-active # NextProtocol VSAN FC ID/Mask RCtl/Mask Flags Hops Cost-------- ---- -------- -------- ---- ---- ----- ---- ----static 40 0x010000 0xff0000 0x00 0x00 D P A 1 10 fc1/14 Domain 0x01(1)fspf 40 0x010000 0xff0000 0x00 0x00 D P N 1 500 fc1/7 Domain 0x01(1)MDS2# show fcroute unicast vsan 40D:direct R:remote P:permanent V:volatile A:active N:non-active # NextProtocol VSAN FC ID/Mask RCtl/Mask Flags Hops Cost-------- ---- -------- -------- ---- ---- ----- ---- ----fspf 40 0x500000 0xff0000 0x00 0x00 D P A 1 1fspf 40 0xee0000 0xff0000 0x00 0x00 D P A 1 500static 40 0xef0000 0xff0000 0x00 0x00 R P A 1 10fspf 40 0xef0000 0xff0000 0x00 0x00 D P N 1 500MDS2# show fcroute unicast 0xef0000 0xff0000 v 40D:direct R:remote P:permanent V:volatile A:active N:non-active # NextProtocol VSAN FC ID/Mask RCtl/Mask Flags Hops Cost-------- ---- -------- -------- ---- ---- ----- ---- ----static 40 0xef0000 0xff0000 0x00 0x00 R P A 1 10 fc1/15 Domain 0xed(237)fspf 40 0xef0000 0xff0000 0x00 0x00 D P N 1 500 fc1/7 Domain 0xef(239)MDS3# show fcroute unicast v 40D:direct R:remote P:permanent V:volatile A:active N:non-active # NextProtocol VSAN FC ID/Mask RCtl/Mask Flags Hops Cost-------- ---- -------- -------- ---- ---- ----- ---- ----fspf 40 0x010000 0xff0000 0x00 0x00 D P A 1 500fspf 40 0x500000 0xff0000 0x00 0x00 D P A 1 1fspf 40 0xef0000 0xff0000 0x00 0x00 D P A 1 500I believe I do see an issue with this VSAN and the static routing. On MDS1, I am routing domain 10 over fc1/14, which is the link to MDS3. Yet domain 10 is MDS2. Because the "remote" keyword is not specified in the fcroute, this would likely lead to a potential issue.MDS2, tries to use fcroute to reach domain 239, which is on MDS1. The fcroute points over fc1/15 and has the "remote" keyword, which is correct since fc1/15 is to MDS3 which is an intermediate hop.Even more telling, is that MDS3's fcdomain and fcns databases are consistant. The inconsistancies exist in MDS1 and MDS2 which is where the fcroutes are being pulled.I was going to try to add the "remote" keyword as a test to the fcroute on MDS1, as this clearly looks like a mistake on my part. I believe that should be a valid configuration and should not cause issues. I can see where the current configuration is not valid because I am using an fcroute on MDS1 and not using remote keyword yet I am routing through an intermediate switch.I have not analyzed VSAN 20's fcroutes in this way yet (as you know for now they are removed for testing). However, I may go back and check them as deeply to see if there may be a configuration issue also going on there as well.I always understood it that FCNS traffic used 0xFFFCxx where xx=the domain the switch is trying to reach, this is a special format I believe unique to FCNS, other services I thought use a more standard fabric broadcast of 0xFFFFFx where x= the well known service.In any case, I hope I am on the right track and close to solving this mystery. I just wanted to give you an update and see what you thought. So, so far, it looks like this may be the issue. It doesn't explain the similar issue I see with VSAN 20, but I will need to look at that more extensively to see if something similar is going on. Point is, be very careful of fcroutes. The "remote" keyword can be a bit confusing to understand. Also it would appear that FCNS traffic uses the fcroutes. Cisco uses a format for FCNS manager traffic in order to work with VSAN's. The FCNS manager of say domain EF has an FCID of 0xFFFCEF. If you had a static route for 0xEF0000 0xFF0000 it would be followed for traffic to reach this special addressing format. At least that's what my observation is. So although the domain is in the third byte, it still knows to follow regular FSPF and static routing in the same way a normal FCID would.