[j-nsp] QFX5110 : Q-in-Q in VXLAN

[Olivier FRUQUET]

Hello,

We've got an IP fabric composed with 2 MX204 routers as cores, 2 QFX5110
switches as Provider Edges, and 2 EX2300 as Customer Edges.
The MX and the QFX communicate together with BGP underlay and overlay
groups, the overlay group use evpn signaling.
Everything to run fine except when we want to make Q-in-Q vlans transiting
through the VXLAN tunnel (only one VNI : 1001).
The Q-in-Q encapsulation comes from the EX2300 switches to the QFX switches
(the S-VLAN Q-in-Q tag is also 1001), but on the other end of the tunnel we
don't have the Q-in-Q frames coming.
We tested some scenarios : for example when we stop the Q-in-Q and just
putting VLAN 1001 tagged from the CE switches, the ping is OK through the
VXLAN tunnel.
We used the pattern 3 from your documentation on the Juniper's website :
https://www.juniper.net/documentation/en_US/junos/topics/topic-map/evpn-vxlan-flexible-vlan-tag.html

Do you have some ideas ?
Thank you !
Olivier
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Brian Rak]

Are you trying to push multiple .1q tags onto the VXLAN traffic?
(meaning you're trying to add a C-VLAN *and* a S-VLAN)?

If so, JTAC has told me that QFX series devices (apparently the entire
line...) do not support adding multiple .q1 tags

On 9/10/2018 2:32 PM, Olivier FRUQUET wrote:
> Hello,
>
> We've got an IP fabric composed with 2 MX204 routers as cores, 2 QFX5110
> switches as Provider Edges, and 2 EX2300 as Customer Edges.
> The MX and the QFX communicate together with BGP underlay and overlay
> groups, the overlay group use evpn signaling.
> Everything to run fine except when we want to make Q-in-Q vlans transiting
> through the VXLAN tunnel (only one VNI : 1001).
> The Q-in-Q encapsulation comes from the EX2300 switches to the QFX switches
> (the S-VLAN Q-in-Q tag is also 1001), but on the other end of the tunnel we
> don't have the Q-in-Q frames coming.
> We tested some scenarios : for example when we stop the Q-in-Q and just
> putting VLAN 1001 tagged from the CE switches, the ping is OK through the
> VXLAN tunnel.
> We used the pattern 3 from your documentation on the Juniper's website :
> https://www.juniper.net/documentation/en_US/junos/topics/topic-map/evpn-vxlan-flexible-vlan-tag.html
>
> Do you have some ideas ?
> Thank you !
> Olivier
> _______________________________________________
> juniper-nsp mailing list juniper-***@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Raphael Maunier]

There is some limitation on the qfx due to the Broadcom chip. On the 10k, you don’t have all the limitations to the cheap chip ^^

On the 5100 you have a lot of limitation on the 5110, you have less, but not sure it support the double tag encapsulated frames (Broadcom)

I have a bunch of evpn infrastructure deployed, but I never tried to use q-in-q, so not sure if 5k will support this.
I have 10k’s in the lab I may be able to test it

Raphael


Envoyé de mon iPhone

> Le 10 sept. 2018 à 20:48, Brian Rak <***@gameservers.com> a écrit :
>
> Are you trying to push multiple .1q tags onto the VXLAN traffic? (meaning you're trying to add a C-VLAN *and* a S-VLAN)?
>
> If so, JTAC has told me that QFX series devices (apparently the entire line...) do not support adding multiple .q1 tags
>
>> On 9/10/2018 2:32 PM, Olivier FRUQUET wrote:
>> Hello,
>>
>> We've got an IP fabric composed with 2 MX204 routers as cores, 2 QFX5110
>> switches as Provider Edges, and 2 EX2300 as Customer Edges.
>> The MX and the QFX communicate together with BGP underlay and overlay
>> groups, the overlay group use evpn signaling.
>> Everything to run fine except when we want to make Q-in-Q vlans transiting
>> through the VXLAN tunnel (only one VNI : 1001).
>> The Q-in-Q encapsulation comes from the EX2300 switches to the QFX switches
>> (the S-VLAN Q-in-Q tag is also 1001), but on the other end of the tunnel we
>> don't have the Q-in-Q frames coming.
>> We tested some scenarios : for example when we stop the Q-in-Q and just
>> putting VLAN 1001 tagged from the CE switches, the ping is OK through the
>> VXLAN tunnel.
>> We used the pattern 3 from your documentation on the Juniper's website :
>> https://www.juniper.net/documentation/en_US/junos/topics/topic-map/evpn-vxlan-flexible-vlan-tag.html
>>
>> Do you have some ideas ?
>> Thank you !
>> Olivier
>> _______________________________________________
>> juniper-nsp mailing list juniper-***@puck.nether.net
>> https://puck.nether.net/mailman/listinfo/juniper-nsp
> _______________________________________________
> juniper-nsp mailing list juniper-***@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https:

[Raphael Mazelier]

On 10/09/2018 21:22, Raphael Maunier wrote:
>
> There is some limitation on the qfx due to the Broadcom chip. On the 10k, you don’t have all the limitations to the cheap chip ^^
>
> On the 5100 you have a lot of limitation on the 5110, you have less, but not sure it support the double tag encapsulated frames (Broadcom)
>
> I have a bunch of evpn infrastructure deployed, but I never tried to use q-in-q, so not sure if 5k will support this.
> I have 10k’s in the lab I may be able to test it
>

Got the same problem type of problem on EX5440 and QFX5100.
There are (very) slow limit on label stacking due to the chip.

Quoting the doc (Read
https://www.juniper.net/documentation/en_US/junos/topics/reference/general/mpls-limitations-qfx-series.html)
:

- Push of a maximum of three labels is supported in the MPLS edge switch
if label swap is not done.

- Push of a maximum of two labels is supported in the MPLS edge switch
if label swap is done.


--
Raphael Mazelier
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.ne

[Olivier FRUQUET]

Hello,
Thank you for your answers, very bad news for me :(
There's no solution to make a transparent pseudowire with support of VLAN
tags with my hardware ?

Olivier

Le lun. 10 sept. 2018 à 21:32, Raphael Mazelier <***@futomaki.net> a
écrit :

> On 10/09/2018 21:22, Raphael Maunier wrote:
> >
> > There is some limitation on the qfx due to the Broadcom chip. On the
> 10k, you don’t have all the limitations to the cheap chip ^^
> >
> > On the 5100 you have a lot of limitation on the 5110, you have less, but
> not sure it support the double tag encapsulated frames (Broadcom)
> >
> > I have a bunch of evpn infrastructure deployed, but I never tried to use
> q-in-q, so not sure if 5k will support this.
> > I have 10k’s in the lab I may be able to test it
> >
>
> Got the same problem type of problem on EX5440 and QFX5100.
> There are (very) slow limit on label stacking due to the chip.
>
> Quoting the doc (Read
>
> https://www.juniper.net/documentation/en_US/junos/topics/reference/general/mpls-limitations-qfx-series.html)
>
> :
>
> - Push of a maximum of three labels is supported in the MPLS edge switch
> if label swap is not done.
>
> - Push of a maximum of two labels is supported in the MPLS edge switch
> if label swap is done.
>
>
> --
> Raphael Mazelier
> _______________________________________________
> juniper-nsp mailing list juniper-***@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.

[Raphael Maunier]

I was about to suggest this .

If you only need to do A to B, I will suggest doing pseudowire instead of Evpn transport.
But, you cannot end the pseudowire on a sub-if ( on MX you can ), on QFX, Interface are dedicated to one protocol

Raphael

On 10/09/2018 21:37, "juniper-nsp on behalf of Olivier FRUQUET" <juniper-nsp-***@puck.nether.net on behalf of ***@gmail.com> wrote:

Hello,
Thank you for your answers, very bad news for me :(
There's no solution to make a transparent pseudowire with support of VLAN
tags with my hardware ?

Olivier

Le lun. 10 sept. 2018 à 21:32, Raphael Mazelier <***@futomaki.net> a
écrit :

> On 10/09/2018 21:22, Raphael Maunier wrote:
> >
> > There is some limitation on the qfx due to the Broadcom chip. On the
> 10k, you don’t have all the limitations to the cheap chip ^^
> >
> > On the 5100 you have a lot of limitation on the 5110, you have less, but
> not sure it support the double tag encapsulated frames (Broadcom)
> >
> > I have a bunch of evpn infrastructure deployed, but I never tried to use
> q-in-q, so not sure if 5k will support this.
> > I have 10k’s in the lab I may be able to test it
> >
>
> Got the same problem type of problem on EX5440 and QFX5100.
> There are (very) slow limit on label stacking due to the chip.
>
> Quoting the doc (Read
>
> https://www.juniper.net/documentation/en_US/junos/topics/reference/general/mpls-limitations-qfx-series.html)
>
> :
>
> - Push of a maximum of three labels is supported in the MPLS edge switch
> if label swap is not done.
>
> - Push of a maximum of two labels is supported in the MPLS edge switch
> if label swap is done.
>
>
> --
> Raphael Mazelier
> _______________________________________________
> juniper-nsp mailing list juniper-***@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp


_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/ju

[Raphael Mazelier]

On 10/09/2018 21:40, Raphael Maunier wrote:
> I was about to suggest this .
>
> If you only need to do A to B, I will suggest doing pseudowire instead of Evpn transport.
> But, you cannot end the pseudowire on a sub-if ( on MX you can ), on QFX, Interface are dedicated to one protocol
>

Yep simple CCC should work (of memory it worked for me on a simple setup
with only one P in the middle).

[Olivier FRUQUET]

Okay!
Do you have some configurations examples of simple CCC with .1q support
please ?
Does this permit to transit layer2 trafic like STP or LLDP ? And what about
multicast ?
I'm dedicating one physical interface on each QFX to connect the CE devices.

Le lun. 10 sept. 2018 à 21:44, Raphael Mazelier <***@futomaki.net> a
écrit :

> On 10/09/2018 21:40, Raphael Maunier wrote:
> > I was about to suggest this .
> >
> > If you only need to do A to B, I will suggest doing pseudowire instead
> of Evpn transport.
> > But, you cannot end the pseudowire on a sub-if ( on MX you can ), on
> QFX, Interface are dedicated to one protocol
> >
>
> Yep simple CCC should work (of memory it worked for me on a simple setup
> with only one P in the middle).
>
> --
> Raphael Mazelier
>
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/li

[Pavel Lunin]

Hey,

The Q-in-Q encapsulation comes from the EX2300 switches to the QFX switches
> (the S-VLAN Q-in-Q tag is also 1001), but on the other end of the tunnel we
> don't have the Q-in-Q frames coming.
>

I am curious if the packets don't leave the ingress VTEP at all or the
tail-end VTEP can't treat them.

"ping -f -s 1000" and "monitor interface traffic" can help to figure out
where the packets are dropped. And if the VXLAN packets leave the
core-facing interface of the ingress VTEP, I'd suggest to put a sniffer in
the middle and take a look at the packets closely.

Not that I am sure that it will work but... Juniper explicitly says that
it's supported on all Trident2/2+/3-based switches in the very very fresh
JUNOS (make sure that you don't miss this point).

I suspect that it's not a honest Q-in-Q-in-VXLAN but some cheating, e. g.
pop the VLAN tags and push them back on the other side, while VXLAN tunnel
carries untagged / single-tagged frames. (Yes this requires per-vlan
tunnels and might not work for your need).

This is how VLAN-based Martini pseudowires work on those switches (even on
EX4550, if memory serves). It's officially supported in EX-to-EX/QFX-to-QFX
mode, where it works out-of-the-box, while in case where you have an
MX-like router on the other end, you need to manually push/pop VLAN tags
and disable control-word on the MX side (discussed many times in this list).

[Alain Hebert]

    Hi,

    On QFX5100 the L2TP/Q-in-Q services has limitations.  I have to dig
through my pile of tickets for details...  but I remember something
about PVST+ packets not being forwarded at all.

    So we just switched everything to MPLS/l2circuits/VLAN CCC (for
now) instead of battling with the QFX5100 platforms.  We'll deploy
EVPN/VXLAN to our customers once we find a solution but we're not in a rush.

    PS: I heard a rumor that there is a fix in 18.x for the QFX5100...

    To note that the QFX5110 platform do not seem to be suffering from
the same issues...  I suggest to get a demo to confirm the functionality.

-----
Alain Hebert ***@pubnix.net
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: [phone] http://www.pubnix.net Fax: [phone]

On 09/10/18 17:20, Pavel Lunin wrote:
> Hey,
>
> The Q-in-Q encapsulation comes from the EX2300 switches to the QFX switches
>> (the S-VLAN Q-in-Q tag is also 1001), but on the other end of the tunnel we
>> don't have the Q-in-Q frames coming.
>>
> I am curious if the packets don't leave the ingress VTEP at all or the
> tail-end VTEP can't treat them.
>
> "ping -f -s 1000" and "monitor interface traffic" can help to figure out
> where the packets are dropped. And if the VXLAN packets leave the
> core-facing interface of the ingress VTEP, I'd suggest to put a sniffer in
> the middle and take a look at the packets closely.
>
> Not that I am sure that it will work but... Juniper explicitly says that
> it's supported on all Trident2/2+/3-based switches in the very very fresh
> JUNOS (make sure that you don't miss this point).
>
> I suspect that it's not a honest Q-in-Q-in-VXLAN but some cheating, e. g.
> pop the VLAN tags and push them back on the other side, while VXLAN tunnel
> carries untagged / single-tagged frames. (Yes this requires per-vlan
> tunnels and might not work for your need).
>
> This is how VLAN-based Martini pseudowires work on those switches (even on
> EX4550, if memory serves). It's officially supported in EX-to-EX/QFX-to-QFX
> mode, where it works out-of-the-box, while in case where you have an
> MX-like router on the other end, you need to manually push/pop VLAN tags
> and disable control-word on the MX side (discussed many times in this list).
>
>
> --
> Pavel
> _______________________________________________
> juniper-nsp mailing list juniper-***@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listi

[Alexandre Guimaraes]

Well...

From my experience with QFX5100, Q-inQ, does not work at all, only one(in one) vlan have the right to pass(VXLAN), L2TP Protocols don't have the rights to pass, funny thing....

For years I discuss with local Juniper representative or Juniper team itself... but... no one cent about a solution.... once again, when I need a QinQ(full services), I use Cisco Catalysts

Don't expend time and energy trying to find a solution for this neglect....

I heard the same thing about the version 18.x, but honestly... leave it there... I still prefer keep my environment work and running with 14.1X53 with Mpls stuffs and an EX facing the customer..... flow like water....


Att.,
Alexandre Guimarães
+55.19. 3517-7724
+55.19. 99822.4866

***@ascenty.com
www.ascenty.com <http://www.ascenty.com/>



Em 11/09/2018 09:46, "juniper-nsp em nome de Alain Hebert" <juniper-nsp-***@puck.nether.net em nome de ***@pubnix.net> escreveu:

Hi,

On QFX5100 the L2TP/Q-in-Q services has limitations. I have to dig
through my pile of tickets for details... but I remember something
about PVST+ packets not being forwarded at all.

So we just switched everything to MPLS/l2circuits/VLAN CCC (for
now) instead of battling with the QFX5100 platforms. We'll deploy
EVPN/VXLAN to our customers once we find a solution but we're not in a rush.

PS: I heard a rumor that there is a fix in 18.x for the QFX5100...

To note that the QFX5110 platform do not seem to be suffering from
the same issues... I suggest to get a demo to confirm the functionality.

-----
Alain Hebert ***@pubnix.net
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: [phone] http://www.pubnix.net Fax: [phone]

On 09/10/18 17:20, Pavel Lunin wrote:
> Hey,
>
> The Q-in-Q encapsulation comes from the EX2300 switches to the QFX switches
>> (the S-VLAN Q-in-Q tag is also 1001), but on the other end of the tunnel we
>> don't have the Q-in-Q frames coming.
>>
> I am curious if the packets don't leave the ingress VTEP at all or the
> tail-end VTEP can't treat them.
>
> "ping -f -s 1000" and "monitor interface traffic" can help to figure out
> where the packets are dropped. And if the VXLAN packets leave the
> core-facing interface of the ingress VTEP, I'd suggest to put a sniffer in
> the middle and take a look at the packets closely.
>
> Not that I am sure that it will work but... Juniper explicitly says that
> it's supported on all Trident2/2+/3-based switches in the very very fresh
> JUNOS (make sure that you don't miss this point).
>
> I suspect that it's not a honest Q-in-Q-in-VXLAN but some cheating, e. g.
> pop the VLAN tags and push them back on the other side, while VXLAN tunnel
> carries untagged / single-tagged frames. (Yes this requires per-vlan
> tunnels and might not work for your need).
>
> This is how VLAN-based Martini pseudowires work on those switches (even on
> EX4550, if memory serves). It's officially supported in EX-to-EX/QFX-to-QFX
> mode, where it works out-of-the-box, while in case where you have an
> MX-like router on the other end, you need to manually push/pop VLAN tags
> and disable control-word on the MX side (discussed many times in this list).
>
>
> --
> Pavel
> _______________________________________________
> juniper-nsp mailing list juniper-***@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp


_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.n

[Olivier FRUQUET]

Hello,

Finally, these QFX went in production... with a L2 circuit configured to
bypass this VXLAN issue.
I'm totally giving up VXLAN for a while.
Thank you all for your replies.
Olivier

Le mar. 11 sept. 2018 à 15:03, Alexandre Guimaraes <
***@ascenty.com> a écrit :

> Well...
>
> From my experience with QFX5100, Q-inQ, does not work at all, only
> one(in one) vlan have the right to pass(VXLAN), L2TP Protocols don't have
> the rights to pass, funny thing....
>
> For years I discuss with local Juniper representative or Juniper
> team itself... but... no one cent about a solution.... once again, when I
> need a QinQ(full services), I use Cisco Catalysts
>
> Don't expend time and energy trying to find a solution for this
> neglect....
>
> I heard the same thing about the version 18.x, but honestly...
> leave it there... I still prefer keep my environment work and running with
> 14.1X53 with Mpls stuffs and an EX facing the customer..... flow like
> water....
>
>
> Att.,
> Alexandre Guimarães
> +55.19. 3517-7724
> +55.19. 99822.4866
>
> ***@ascenty.com
> www.ascenty.com <http://www.ascenty.com/>
>
>
>
> Em 11/09/2018 09:46, "juniper-nsp em nome de Alain Hebert" <
> juniper-nsp-***@puck.nether.net em nome de ***@pubnix.net>
> escreveu:
>
> Hi,
>
> On QFX5100 the L2TP/Q-in-Q services has limitations. I have to
> dig
> through my pile of tickets for details... but I remember something
> about PVST+ packets not being forwarded at all.
>
> So we just switched everything to MPLS/l2circuits/VLAN CCC (for
> now) instead of battling with the QFX5100 platforms. We'll deploy
> EVPN/VXLAN to our customers once we find a solution but we're not in a
> rush.
>
> PS: I heard a rumor that there is a fix in 18.x for the QFX5100...
>
> To note that the QFX5110 platform do not seem to be suffering
> from
> the same issues... I suggest to get a demo to confirm the
> functionality.
>
> -----
> Alain Hebert ***@pubnix.net
> PubNIX Inc.
> 50 boul. St-Charles
> P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
> Tel: [phone] http://www.pubnix.net Fax: [phone]
>
> On 09/10/18 17:20, Pavel Lunin wrote:
> > Hey,
> >
> > The Q-in-Q encapsulation comes from the EX2300 switches to the QFX
> switches
> >> (the S-VLAN Q-in-Q tag is also 1001), but on the other end of the
> tunnel we
> >> don't have the Q-in-Q frames coming.
> >>
> > I am curious if the packets don't leave the ingress VTEP at all or
> the
> > tail-end VTEP can't treat them.
> >
> > "ping -f -s 1000" and "monitor interface traffic" can help to figure
> out
> > where the packets are dropped. And if the VXLAN packets leave the
> > core-facing interface of the ingress VTEP, I'd suggest to put a
> sniffer in
> > the middle and take a look at the packets closely.
> >
> > Not that I am sure that it will work but... Juniper explicitly says
> that
> > it's supported on all Trident2/2+/3-based switches in the very very
> fresh
> > JUNOS (make sure that you don't miss this point).
> >
> > I suspect that it's not a honest Q-in-Q-in-VXLAN but some cheating,
> e. g.
> > pop the VLAN tags and push them back on the other side, while VXLAN
> tunnel
> > carries untagged / single-tagged frames. (Yes this requires per-vlan
> > tunnels and might not work for your need).
> >
> > This is how VLAN-based Martini pseudowires work on those switches
> (even on
> > EX4550, if memory serves). It's officially supported in
> EX-to-EX/QFX-to-QFX
> > mode, where it works out-of-the-box, while in case where you have an
> > MX-like router on the other end, you need to manually push/pop VLAN
> tags
> > and disable control-word on the MX side (discussed many times in
> this list).
> >
> >
> > --
> > Pavel
> > _______________________________________________
> > juniper-nsp mailing list juniper-***@puck.nether.net
> > https://puck.nether.net/mailman/listinfo/juniper-nsp
> >
>
> _______________________________________________
> juniper-nsp mailing list juniper-***@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>
>
> _______________________________________________
> juniper-nsp mailing list juniper-***@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/m