We just spent about a week trying to put a bunch of systems into production that had been ordered with 25Gbit fiber interfaces. We had planned to collect those on two of our Arista 7050CX3, using 100GBit QSFP28 in 4 * 25GBit mode and MPO breakout cables to 4 * LC for the 25Gbit SFP28 end. So we cable everything up, configure our LACP channels on both ends, and ... nothing. All of the links stay down.
They do show a signal on the transciever though (at least on the switch side where we can look at optics information). An show interfaces et10/1-4 status says "notconnect" for all four subinterfaces. An show interfaces et10/1-4 phy displays an "errDisabled" on the phy layer. We are stumped.
Over the course of the next few days, we try several changes, to no avail. Directly connecting two Arista switches works though, as does a direct connection between two end hosts. We even swap everything down to 40G on the Arista side and 10G SFP+ in the end hosts, which turns out perfectly fine (so at least our cabling is correct).
At this point, support for the appliances we're trying to connect gives us credentials for shell access. It's a non-root user on what turns out as a normal Linux system, but at least I can see that it comes with QLogic Corp. FastLinQ QL45000 Series 25GbE controllers (for a short moment we had suspected we had the wrong controllers), and I can get some information by using ethtool. One of those is that ethtool reports the host interfaces as "25GBASE-KR", which tells me nothing. Someone on IRC mentions that "-KR" denotes an "electrical backplane" connection. Armed with those two small bits of information, I hit the search engines, and find this useful table in a document on the Marvell web site:
It's accompanied by the following text:
There's two different, incompatible, error correction mechanisms on the bitstream layer of 25Gbit interfaces!? I didn't know that.
Since the default on Arista switches seems to be Reed-Solomon, and I don't have any way to configure a detail like that on the end host, we change the configuration on the Arista side:
interface et10/1-4
error-correction encoding fire-code
That's all. We do the same for three other interface groups, and all links work just excpected (except for one that apparently has a bad transciever in the end host). I call off the screen-sharing session with Arista support planned for five minutes later.
They do show a signal on the transciever though (at least on the switch side where we can look at optics information). An show interfaces et10/1-4 status says "notconnect" for all four subinterfaces. An show interfaces et10/1-4 phy displays an "errDisabled" on the phy layer. We are stumped.
Over the course of the next few days, we try several changes, to no avail. Directly connecting two Arista switches works though, as does a direct connection between two end hosts. We even swap everything down to 40G on the Arista side and 10G SFP+ in the end hosts, which turns out perfectly fine (so at least our cabling is correct).
At this point, support for the appliances we're trying to connect gives us credentials for shell access. It's a non-root user on what turns out as a normal Linux system, but at least I can see that it comes with QLogic Corp. FastLinQ QL45000 Series 25GbE controllers (for a short moment we had suspected we had the wrong controllers), and I can get some information by using ethtool. One of those is that ethtool reports the host interfaces as "25GBASE-KR", which tells me nothing. Someone on IRC mentions that "-KR" denotes an "electrical backplane" connection. Armed with those two small bits of information, I hit the search engines, and find this useful table in a document on the Marvell web site:
It's accompanied by the following text:
The –S short reach interfaces aim to support high-quality cables without
ForwardError Correction (FEC) to minimize latency. Full reach interfaces
aim to support the lowest possible cable or backplane cost and the longest
possible reach, which do require the use of FEC. FEC options include
BASE-R FEC (also referred to as Fire Code) and RS-FEC (also referred to
as Reed-Solomon).
ForwardError Correction (FEC) to minimize latency. Full reach interfaces
aim to support the lowest possible cable or backplane cost and the longest
possible reach, which do require the use of FEC. FEC options include
BASE-R FEC (also referred to as Fire Code) and RS-FEC (also referred to
as Reed-Solomon).
There's two different, incompatible, error correction mechanisms on the bitstream layer of 25Gbit interfaces!? I didn't know that.
Since the default on Arista switches seems to be Reed-Solomon, and I don't have any way to configure a detail like that on the end host, we change the configuration on the Arista side:
interface et10/1-4
error-correction encoding fire-code
That's all. We do the same for three other interface groups, and all links work just excpected (except for one that apparently has a bad transciever in the end host). I call off the screen-sharing session with Arista support planned for five minutes later.
;){kind=link}