[j-nsp] Recommended firmware for QFX5100-48T

Post by Paul S.
Hi folks,
Are everyone running the JTAC recommended 14.1X53-D35.3 or have you
found better stability at some newer revision?
My problem is that the "tri state" 10g ports (copper) don't seem to
want to run at anything less than 10g. It links up when connected to a
1g device, but still claims that the port is operating in 10g mode.
The biggest issue I have is that if I assign a /30 to the p2p
interfaces (between the qfx and any copper 1g device), my p2p latency
is somewhere from 10 to 40ms.
I asked around to see if there's any way to force the ports into 1g
mode, but the "speed" knob is missing. I deliberately turned on
auto-negotiation, does not seem to help.
802.3ad LAGs created using any of these links also claim to have
speeds of 20g/40g when there's in reality only 4g of capacity.
Can someone hit me with a clue stick? Thanks!

I presume that you're specifing the port config as ge-0/0/foo rather
than xe-0/0/foo

joel@ show interfaces ge-0/0/46
Physical interface: ge-0/0/46, Enabled, Physical link is Up
Interface index: 701, SNMP ifIndex: 616
Description:
Link-level type: Ethernet, MTU: 1514, Speed: 1000mbps, BPDU Error:
None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering:
Disabled, Flow control: Disabled, Auto-negotiation: Enabled,
Remote fault: Online, Media type: Copper
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x4000
Link flags : None
CoS queues : 12 supported, 12 maximum usable queues
Current address:
Last flapped : 2016-08-26 03:31:56 UTC (6w3d 19:46 ago)
Input rate : 0 bps (0 pps)
Output rate : 0 bps (0 pps)
Active alarms : None
Active defects : None
Interface transmit statistics: Disabled

Logical interface ge-0/0/46.0 (Index 563) (SNMP ifIndex 617)
Flags: SNMP-Traps 0x4004000 Encapsulation: ENET2
Input packets : 146287
Output packets: 292513
Protocol inet, MTU: 1500
Flags: Sendbcast-pkt-to-re
Addresses, Flags: Is-Preferred Is-Primary
Destination:

joel@> show chassis hardware
Hardware inventory:
Item Version Part number Serial number Description
Chassis QFX5100-48S-6Q
Pseudo CB 0
Routing Engine 0 BUILTIN BUILTIN QFX Routing Engine
FPC 0 REV 05 QFX5100-48S-6Q

Xcvr 46 NON-JNPR SFP-T

joel@> show version
fpc0:
--------------------------------------------------------------------------
Hostname:
Model: qfx5100-48s-6q
JUNOS Base OS Software Suite [13.2X51-D38]

Post by Paul S.
_______________________________________________
https://puck.nether.net/mailman/listinfo/juniper-nsp

Paul S.

2016-10-10 18:05:24 UTC

Hi Joel,

Thanks for replying.

What are the steps to configure the ports as "ge." Do I just get rid of
xe from the config and replace it with ge for that port, that's all?

I presume that you're specifing the port config as ge-0/0/foo rather
than xe-0/0/foo
Physical interface: ge-0/0/46, Enabled, Physical link is Up
Interface index: 701, SNMP ifIndex: 616
Disabled, Flow control: Disabled, Auto-negotiation: Enabled,
Remote fault: Online, Media type: Copper
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x4000
Link flags : None
CoS queues : 12 supported, 12 maximum usable queues
Last flapped : 2016-08-26 03:31:56 UTC (6w3d 19:46 ago)
Input rate : 0 bps (0 pps)
Output rate : 0 bps (0 pps)
Active alarms : None
Active defects : None
Interface transmit statistics: Disabled
Logical interface ge-0/0/46.0 (Index 563) (SNMP ifIndex 617)
Flags: SNMP-Traps 0x4004000 Encapsulation: ENET2
Input packets : 146287
Output packets: 292513
Protocol inet, MTU: 1500
Flags: Sendbcast-pkt-to-re
Addresses, Flags: Is-Preferred Is-Primary
Item Version Part number Serial number Description
Chassis QFX5100-48S-6Q
Pseudo CB 0
Routing Engine 0 BUILTIN BUILTIN QFX Routing Engine
FPC 0 REV 05 QFX5100-48S-6Q
Xcvr 46 NON-JNPR SFP-T
--------------------------------------------------------------------------
Model: qfx5100-48s-6q
JUNOS Base OS Software Suite [13.2X51-D38]

Post by Paul S.
_______________________________________________
https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Dale Shaw

2016-10-10 20:34:35 UTC

Hi all,

Hi Paul,
Correct, just use 'replace pattern xe-0/0/0 with ge-0/0/0' etc and you
should be fine.

I don't think that's how it works on qfx5100-48**T**

It's more like copper based EX, where the interfaces are configured as
ge-x/x/x even if they're plugged into something capable of only 10 or
100BASE-T operation.

I seem to recall that there is a trick to making it work that involves
autoneg but I'm currently using a tiny screen and an imaginary keyboard so
digging up the details isn't possible right now.

I'm certain that JTAC could help but I'll try to find something when I'm
properly online.

Cheers,
Dale

Post by Paul S.
Hi Joel,
Thanks for replying.
What are the steps to configure the ports as "ge." Do I just get rid of

Post by Paul S.
from the config and replace it with ge for that port, that's all?

QFX5100-48S-6Q

Post by joel jaeggli
Xcvr 46 NON-JNPR SFP-T
------------------------------------------------------------
--------------
Model: qfx5100-48s-6q
JUNOS Base OS Software Suite [13.2X51-D38]

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Brent Jones

2016-10-10 21:32:56 UTC

You can set "ether-options autonegotiate" on XE interfaces, and it will do
just as you'd expect.
Firmware on QFX-5100-48T is bad in a lot of ways, but we've gotten them to
work mostly as youd want.
We are on 14.1X53-D35.3

xe-0/0/0 {
ether-options {
auto-negotiation;
}
unit 0 {
family ethernet-switching;
}
}

Post by Dale Shaw
Hi all,

Hi Paul,
Correct, just use 'replace pattern xe-0/0/0 with ge-0/0/0' etc and you
should be fine.

I don't think that's how it works on qfx5100-48**T**
It's more like copper based EX, where the interfaces are configured as
ge-x/x/x even if they're plugged into something capable of only 10 or
100BASE-T operation.
I seem to recall that there is a trick to making it work that involves
autoneg but I'm currently using a tiny screen and an imaginary keyboard so
digging up the details isn't possible right now.
I'm certain that JTAC could help but I'll try to find something when I'm
properly online.
Cheers,
Dale

Post by Paul S.
Hi Joel,
Thanks for replying.
What are the steps to configure the ports as "ge." Do I just get rid of

Post by Paul S.
from the config and replace it with ge for that port, that's all?

Post by Paul S.
1g device, but still claims that the port is operating in 10g mode.
The biggest issue I have is that if I assign a /30 to the p2p
interfaces (between the qfx and any copper 1g device), my p2p latency
is somewhere from 10 to 40ms.
I asked around to see if there's any way to force the ports into 1g
mode, but the "speed" knob is missing. I deliberately turned on
auto-negotiation, does not seem to help.
802.3ad LAGs created using any of these links also claim to have
speeds of 20g/40g when there's in reality only 4g of capacity.
Can someone hit me with a clue stick? Thanks!

QFX5100-48S-6Q

Post by joel jaeggli
Xcvr 46 NON-JNPR SFP-T
------------------------------------------------------------
--------------
Model: qfx5100-48s-6q
JUNOS Base OS Software Suite [13.2X51-D38]

_______________________________________________
https://puck.nether.net/mailman/listinfo/juniper-nsp

--
Brent Jones
***@brentrjones.com
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Paul S.

2016-10-11 05:16:14 UTC

Hi Dale,

Yeah, seems to be the case. I tried configuring one as GE, the show
command simply said it couldn't find that interface afterwards.

Auto negotiation being turned on doesn't seem to matter, it linked up
even without it.

On sh int xe-0/0/0 media, you can however find these:

Link mode: Full-duplex, Flow control: None, Remote fault: OK,
Link partner Speed: 1000 Mbps

So I think that part's OK. monitor interface and ae<x> interfaces still
claim to run at 10g, but I think that's cosmetic?

I just need to figure out where the latency is coming from now. Does the
QFX do any ICMP deprio by default now? This is a VC of two 5100-48Ts.

Has anyone tried any of the newer firmwares? Juniper says 16.1 is latest
now.

Post by Dale Shaw
Hi all,

Hi Paul,
Correct, just use 'replace pattern xe-0/0/0 with ge-0/0/0' etc and you
should be fine.

I don't think that's how it works on qfx5100-48**T**
It's more like copper based EX, where the interfaces are configured as
ge-x/x/x even if they're plugged into something capable of only 10 or
100BASE-T operation.
I seem to recall that there is a trick to making it work that involves
autoneg but I'm currently using a tiny screen and an imaginary
keyboard so digging up the details isn't possible right now.
I'm certain that JTAC could help but I'll try to find something when
I'm properly online.
Cheers,
Dale

Post by Paul S.
Hi Joel,
Thanks for replying.
What are the steps to configure the ports as "ge." Do I just get

rid of xe

Post by Paul S.
from the config and replace it with ge for that port, that's all?

connected to a

latency

Post by Paul S.
is somewhere from 10 to 40ms.
I asked around to see if there's any way to force the ports into 1g
mode, but the "speed" knob is missing. I deliberately turned on
auto-negotiation, does not seem to help.
802.3ad LAGs created using any of these links also claim to have
speeds of 20g/40g when there's in reality only 4g of capacity.
Can someone hit me with a clue stick? Thanks!

I presume that you're specifing the port config as ge-0/0/foo rather
than xe-0/0/foo
Physical interface: ge-0/0/46, Enabled, Physical link is Up
Interface index: 701, SNMP ifIndex: 616
Disabled, Flow control: Disabled, Auto-negotiation: Enabled,
Remote fault: Online, Media type: Copper
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x4000
Link flags : None
CoS queues : 12 supported, 12 maximum usable queues
Last flapped : 2016-08-26 03:31:56 UTC (6w3d 19:46 ago)
Input rate : 0 bps (0 pps)
Output rate : 0 bps (0 pps)
Active alarms : None
Active defects : None
Interface transmit statistics: Disabled
Logical interface ge-0/0/46.0 (Index 563) (SNMP ifIndex 617)
Flags: SNMP-Traps 0x4004000 Encapsulation: ENET2
Input packets : 146287
Output packets: 292513
Protocol inet, MTU: 1500
Flags: Sendbcast-pkt-to-re
Addresses, Flags: Is-Preferred Is-Primary
Item Version Part number Serial number Description
Chassis QFX5100-48S-6Q
Pseudo CB 0
Routing Engine 0 BUILTIN BUILTIN QFX Routing
Engine
FPC 0 REV 05 QFX5100-48S-6Q
Xcvr 46 NON-JNPR SFP-T
------------------------------------------------------------
--------------
Model: qfx5100-48s-6q
JUNOS Base OS Software Suite [13.2X51-D38]

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Nik Geyer

2016-10-12 18:39:07 UTC

You should be able to fix the ae cosmetic issue with "set interfaces aeX aggregated-ether-options link-speed 1g".

The latency just seems to be a QFX5100 thing and doesn't actually impact anything operationally, i.e. transitive latency through the switch (which is what matters after all right?) is fine and in the microsecond range as one would expect. I have run various iterations of 14, 15 and 16 and all exhibit the same behaviour.

Tried Junos 16.1R1 on some of our lab switches (24Q and 48S) and it was an instant train wreck, as expected for the first release of a major revision change. We run *lots* of these things some weird and wonderful configurations and the JTAC recommended version, 14.1X53-D35, is what I would recommend unless you have a reason not to use it, e.g. feature requirement.

-----Original Message-----
From: juniper-nsp [mailto:juniper-nsp-***@puck.nether.net] On Behalf Of Paul S.
Sent: Tuesday, 11 October 2016 1:16 AM
To: Dale Shaw <dale.shaw+j-***@gmail.com>; Graham Brown <juniper-***@grahambrown.info>
Cc: juniper-***@puck.nether.net
Subject: Re: [j-nsp] Recommended firmware for QFX5100-48T

Hi Dale,

Yeah, seems to be the case. I tried configuring one as GE, the show command simply said it couldn't find that interface afterwards.

Auto negotiation being turned on doesn't seem to matter, it linked up even without it.

On sh int xe-0/0/0 media, you can however find these:

Link mode: Full-duplex, Flow control: None, Remote fault: OK, Link partner Speed: 1000 Mbps

So I think that part's OK. monitor interface and ae<x> interfaces still claim to run at 10g, but I think that's cosmetic?

I just need to figure out where the latency is coming from now. Does the QFX do any ICMP deprio by default now? This is a VC of two 5100-48Ts.

Has anyone tried any of the newer firmwares? Juniper says 16.1 is latest now.

Post by Dale Shaw
Hi all,

Hi Paul,
Correct, just use 'replace pattern xe-0/0/0 with ge-0/0/0' etc and
you should be fine.

I don't think that's how it works on qfx5100-48**T**
It's more like copper based EX, where the interfaces are configured as
ge-x/x/x even if they're plugged into something capable of only 10 or
100BASE-T operation.
I seem to recall that there is a trick to making it work that involves
autoneg but I'm currently using a tiny screen and an imaginary
keyboard so digging up the details isn't possible right now.
I'm certain that JTAC could help but I'll try to find something when
I'm properly online.
Cheers,
Dale

Post by Paul S.
Hi Joel,
Thanks for replying.
What are the steps to configure the ports as "ge." Do I just get

rid of xe

Post by Paul S.
from the config and replace it with ge for that port, that's all?

connected to a

latency