[j-nsp] L3VPN/RR/PE on Same router

[tim tiriche]

Hello,

I have a MPLS PE (L3VPN) router that is acting as full mesh iBGP within the
US. The other routers in the US are not RR and regular iBGP. This router
also acts as RR for Europe and takes in full BGP table. Is there some
caveats to watch out for?
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Alexander Marhold]

Yes, the PE should do next-hop-self, the RR should not do it
Route reflector can also be EBGP-Border Router,
General use of next-hop self can result in inefficient forwarding

 use next-hop self only for EBGP learned routes

policy-statement bgp-export {
term ebgp {
from route-type external;
then {
next-hop self;
accept;
}
}
term ibgp {
from route-type internal;
then accept;
}
}

regards

alexander


-----Ursprüngliche Nachricht-----
Von: juniper-nsp [mailto:juniper-nsp-***@puck.nether.net] Im Auftrag von tim tiriche
Gesendet: Donnerstag, 16. August 2018 16:40
An: juniper-***@puck.nether.net
Betreff: [j-nsp] L3VPN/RR/PE on Same router

Hello,

I have a MPLS PE (L3VPN) router that is acting as full mesh iBGP within the
US. The other routers in the US are not RR and regular iBGP. This router
also acts as RR for Europe and takes in full BGP table. Is there some
caveats to watch out for?
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nethe

[a***@netconsultings.com]

Of tim tiriche
Sent: Thursday, August 16, 2018 3:40 PM
Subject: [j-nsp] L3VPN/RR/PE on Same router
Hello,
I have a MPLS PE (L3VPN) router that is acting as full mesh iBGP within the
US. The other routers in the US are not RR and regular iBGP. This router

also

acts as RR for Europe and takes in full BGP table. Is there some caveats

to

watch out for?

Yes a good practice is to separate internet routes from internal/services
l3vpn routes onto separate BGP control planes (different sessions at least)
so that malformed bgp msg will affect just one part of your overall BGP
infrastructure.

adam

netconsultings.com
::carrier-class solutions for the telecommunications industry::


_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Mark Tinka]

Post by a***@netconsultings.com
Yes a good practice is to separate internet routes from internal/services
l3vpn routes onto separate BGP control planes (different sessions at least)
so that malformed bgp msg will affect just one part of your overall BGP
infrastructure.

I see you've been giving this advice for quite some time now.

We've been running all address families on the same RR's (different
sessions, obviously, but same hardware) for almost 5 years. The only
reason sessions have gone down is due to hardware problems. It didn't
disrupt services because there are always 2 RR's, but we haven't seen an
outage due to protocol problems in one address family spilling over into
other address families.

Of course, I see your concern, but from our own experience over several
years, I've not seen this issue.

I mention this because introducing this kind of separation is onerous.

Mark.
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Robert Raszuk]

Hey Mark,

It has been a while ....

Post by Mark Tinka
We've been running all address families on the same RR's (different
sessions, obviously, but same hardware)

Out of pure curiosity how are you setting up different BGP sessions to the
same RR ?

I think what Adam is proposing is real TCP session isolation, what you may
be doing is just same single TCP session, but different SAFIs which is not
the same.

Sure you can configure parallel iBGP sessions on the TCP level say between
different loopback addresses to the same RR, but what would that really buy
you ? You could even be more brave and use BGP multisession code path (if
happens to be even supported by your vendor) which in most implementations
I have seen is full of holes like swiss cheese but is this what you are
doing ?

Cheers,
R,.

PS. Have not been reading -nsp aliases for a while, but now I see that I
missed a lot ! Btw do we really need per vendor aliases here ? Wouldn't it
be much easier to just have single nsp list ? After all we all most likely
have all of the vendors in our networks (including Nokia !) and we are all
likely reading all the lists :) Or maybe there is one already ?

Post by Mark Tinka

Post by a***@netconsultings.com
Yes a good practice is to separate internet routes from internal/services
l3vpn routes onto separate BGP control planes (different sessions at

least)

Post by a***@netconsultings.com
so that malformed bgp msg will affect just one part of your overall BGP
infrastructure.

I see you've been giving this advice for quite some time now.
We've been running all address families on the same RR's (different
sessions, obviously, but same hardware) for almost 5 years. The only
reason sessions have gone down is due to hardware problems. It didn't
disrupt services because there are always 2 RR's, but we haven't seen an
outage due to protocol problems in one address family spilling over into
other address families.
Of course, I see your concern, but from our own experience over several
years, I've not seen this issue.
I mention this because introducing this kind of separation is onerous.
Mark.
_______________________________________________
https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Mark Tinka]

Post by Robert Raszuk
Hey Mark,
It has been a while ....

It has, mate. Good to see you in these parts again :-)...

Post by Robert Raszuk
Out of pure curiosity how are you setting up different BGP sessions to
the same RR ? 
I think what Adam is proposing is real TCP session isolation, what you
may be doing is just same single TCP session, but different SAFIs
which is not the same.

You're right; I should have clarified that better - we are, indeed,
running one TCP session with multiple SAFI's.

The only uniqueness between BGP sessions at a TCP level would be by IP
protocol, i.e., IPv4 and IPv6. But even within IPv6, we carry multiple
SAFI's across a single TCP session.

Post by Robert Raszuk
Sure you can configure parallel iBGP sessions on the TCP level say
between different loopback addresses to the same RR, but what would
that really buy you ? You could even be more brave and use BGP
multisession code path (if happens to be even supported by your
vendor) which in most implementations I have seen is full of holes
like swiss cheese but is this what you are doing ?

I'm not that brave :-).

But to your point, the complete hardware and Layer 4 separation of BGP
sessions, perhaps going one step further and having separate planes for
different SAFI's, is overkill, IMHO. But that's just me.

As I mentioned before, we've had our setup since 2014. With the
exception of x86 hardware being more sensitive to temperature
situations, causing related failures, we haven't had any issues at all.

Post by Robert Raszuk
PS.  Have not been reading -nsp aliases for a while, but now I see
that I missed a lot !  Btw do we really need per vendor aliases here ?
Wouldn't it be much easier to just have single nsp list ? After all we
all most likely have all of the vendors in our networks (including
Nokia !) and we are all likely reading all the lists :) Or maybe there
is one already ?

There isn't one to rule them all, AFAIK. In fact, Arista-NSP went live
just yesterday, if I'm not mistaken :-).

I think there is value in having separate lists for the different
vendors. I wouldn't say all network operators have each of them to make
one list the ideal. Besides, there are a lot of things I have zero
interest in on one list that I wish I could filter out (SRX on j-nsp,
ASA on c-nsp, as examples). You can imagine what that'd be like on a
single list :-).

Mark.
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net

[s***@nethelp.no]

Post by Robert Raszuk
PS. Have not been reading -nsp aliases for a while, but now I see that I
missed a lot ! Btw do we really need per vendor aliases here ? Wouldn't it
be much easier to just have single nsp list ? After all we all most likely
have all of the vendors in our networks (including Nokia !) and we are all
likely reading all the lists :) Or maybe there is one already ?

Disagree. I follow several of the -nsp lists, but some of them much
more closely than others. Having them all mixed up would definitely
make this more difficult.

Steinar Haug, Nethelp consulting, ***@nethelp.no
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Robert Raszuk]

Just to clarify ... I was not really worried about how to follow various
lists - mail client does a good job to combine them into one folder, filter
duplicates etc ...

But when writing general reply/question to Mark today about BGP sessions I
noticed it only had j-nsp - but oh the question is general so where do I
post ? I added c-nsp ... that was the trigger for the above comment.

On a similar note I would love to hear comments from all of the members on
what linux tools they use to test pps on the routers ... which list should
I post it to ?

Cheers
R.

Post by Robert Raszuk

Post by Robert Raszuk
PS. Have not been reading -nsp aliases for a while, but now I see that I
missed a lot ! Btw do we really need per vendor aliases here ? Wouldn't

it

Post by Robert Raszuk
be much easier to just have single nsp list ? After all we all most

likely

Post by Robert Raszuk
have all of the vendors in our networks (including Nokia !) and we are

all

Post by Robert Raszuk
likely reading all the lists :) Or maybe there is one already ?

Disagree. I follow several of the -nsp lists, but some of them much
more closely than others. Having them all mixed up would definitely
make this more difficult.

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[a***@netconsultings.com]

Sent: Friday, August 17, 2018 9:57 AM
To: Mark Tinka
Subject: Re: [j-nsp] L3VPN/RR/PE on Same router
Hey Mark,
It has been a while ....

Post by Mark Tinka
We've been running all address families on the same RR's (different
sessions, obviously, but same hardware)

Out of pure curiosity how are you setting up different BGP sessions to the
same RR ?
I think what Adam is proposing is real TCP session isolation, what you may be
doing is just same single TCP session, but different SAFIs which is not the
same.

Yes Robert, I was indeed proposing separate TCP sessions at least -as that's the only way to protect against the default behaviour of terminating session upon malformed bgp update reception.

Sure you can configure parallel iBGP sessions on the TCP level say between
different loopback addresses to the same RR, but what would that really buy
you ? You could even be more brave and use BGP multisession code path (if
happens to be even supported by your vendor) which in most
implementations I have seen is full of holes like swiss cheese but is this what
you are doing ?

Another alternative would be to spin up a separate BGP process, which I think is supported only in XR, but once again that somewhat places one on the outskirts of the common deployment graph.
But I know Mark is using csr1k -so depending on the available NFVI resources (I guess dedicated servers in this case), I think it's not that onerous to spin up yet another VM right?


adam

netconsultings.com
::carrier-class solutions for the telecommunications industry::

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Mark Tinka]

Post by a***@netconsultings.com
Another alternative would be to spin up a separate BGP process, which I think is supported only in XR, but once again that somewhat places one on the outskirts of the common deployment graph.
But I know Mark is using csr1k -so depending on the available NFVI resources (I guess dedicated servers in this case), I think it's not that onerous to spin up yet another VM right?

Turning another one up? Not a problem at all.

The issue is in operating more than you need to.

Mark.
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Saku Ytti]

Post by a***@netconsultings.com
Another alternative would be to spin up a separate BGP process, which I think is supported only in XR, but once again that somewhat places one on the outskirts of the common deployment graph.
But I know Mark is using csr1k -so depending on the available NFVI resources (I guess dedicated servers in this case), I think it's not that onerous to spin up yet another VM right?

I think many vendors support this. JunOS supports multiple RPD
instances in single FreeBSD instance and multiple FreeBSD instances in
single Linux instance.

Personally I'd run the RRs for different AFIs in separate hardware,
because RR hardware is not really where the costs are.

[a***@netconsultings.com]

Sent: Saturday, August 18, 2018 11:44 AM

Post by a***@netconsultings.com
Another alternative would be to spin up a separate BGP process, which I

think is supported only in XR, but once again that somewhat places one on
the outskirts of the common deployment graph.

Post by a***@netconsultings.com
But I know Mark is using csr1k -so depending on the available NFVI

resources (I guess dedicated servers in this case), I think it's not that onerous
to spin up yet another VM right?
I think many vendors support this. JunOS supports multiple RPD instances in
single FreeBSD instance and multiple FreeBSD instances in single Linux
instance.

Really? Interesting, didn't know that, are these features documented anywhere? I could not find anything looking for multi instance RPD.
Are the RPD instances as ships in night, each maintaining its own set of tables and protocols?

Personally I'd run the RRs for different AFIs in separate hardware, because
RR hardware is not really where the costs are.

Yeah all the costs are in edge devices, even core routers HW is order of magnitude more expensive than that of RRs

adam

netconsultings.com
::carrier-class solutions for the telecommunications industry::

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Saku Ytti]

Post by a***@netconsultings.com
Really? Interesting, didn't know that, are these features documented anywhere? I could not find anything looking for multi instance RPD.
Are the RPD instances as ships in night, each maintaining its own set of tables and protocols?

Yes. It's old old feature:

https://www.juniper.net/documentation/en_US/junos/topics/concept/logical-systems-overview-solutions.html

The ability to run multiple FreeBSD KVM on Linux guest is very new. It
also allows you to connect the separate instances via virtual fabric
interface, so you don't need to eat physical interface to connect the
separate instances.
This is really 100% separate JunOS, they only share the Linux
hypervisor from software POV.

https://www.juniper.net/documentation/en_US/junos/information-products/pathway-pages/junos-node-slicing/junos-node-slicing.html

Use case might be buy 128GB RE, collapse edge + pure-core without
having hardware.
Or Megaport on steroids, drop Calient optical switch in pops and MX,
have customers build via API their entire global backbone in seconds,
with active devices and optical network to connect them.

Post by a***@netconsultings.com
Yeah all the costs are in edge devices, even core routers HW is order of magnitude more expensive than that of RRs

Depends, for access networks, 5 year costs tend to be dominated on
MRC's on the backhauls.

[a***@netconsultings.com]

Sent: Saturday, August 18, 2018 12:15 PM

Post by a***@netconsultings.com
Really? Interesting, didn't know that, are these features documented

anywhere? I could not find anything looking for multi instance RPD.

Post by a***@netconsultings.com
Are the RPD instances as ships in night, each maintaining its own set of

tables and protocols?
https://www.juniper.net/documentation/en_US/junos/topics/concept/logi
cal-systems-overview-solutions.html
The ability to run multiple FreeBSD KVM on Linux guest is very new. It also
allows you to connect the separate instances via virtual fabric interface, so
you don't need to eat physical interface to connect the separate instances.
This is really 100% separate JunOS, they only share the Linux hypervisor from
software POV.
https://www.juniper.net/documentation/en_US/junos/information-
products/pathway-pages/junos-node-slicing/junos-node-slicing.html
Use case might be buy 128GB RE, collapse edge + pure-core without having
hardware.
Or Megaport on steroids, drop Calient optical switch in pops and MX, have
customers build via API their entire global backbone in seconds, with active
devices and optical network to connect them.

Aah these two! I feel embarrassed now haha :)

I actually have high hopes for the node-slicing thing -in particular the ability to run CP externally on x86 based COTS HW, hopefully it will get adopted by all the vendors soon (and if the API between CP and DP gets standardized well be living in an SDN utopia).

I'm actually having this "architect personality crises" cause I can't make up my mind between centralizing and decentralizing on the PE edge i.e. whether to have couple big monolithic chassis to fit/scale for all service types (thus scale up/vertically) or rather decentralize and create many smaller maybe even specialized node types and scale by adding more types or more of the same from each type (thus scale out/horizontally).
And I'm very aware of this whole industry swinging back and forth between the two extremes over the years.

There is a finite limit to how much CP state can one fit onto these huge monolithic chassis you can have a big N slot chassis full even though there are just a few line-cards in it, just because you ran out of CP resources because of too many routes/VRFs/BGP sessions. (this problem gets even more pronounced with multi-chassis systems). This is the problem we're facing currently.
I mean this idea of additional CP HW is not new, it's been around since juniper T Matrix and cisco CRS1.
CRS1 -introducing Distributed Route Processors (DRPs) onto which one could offload say an instance of additional BGP process (but one could have just one additional per main system or just two DRPs per SDR).
Another example from past was the T Matrix with JCS1200 so you could have separate CP for each logical system (but again one could have just two REs per logical system).
And then with the advent of MX and ASR9k -we kind of lost the ability to scale out CP.

So I really welcome this flexibility of being able to scale CP up (assigning more CPUs to the CP VM) or scale CP out (more CP VMs), while maintaining single monolithic DP.
This would help me address one of the biggest drawbacks of decentralization (many smaller PEs) -that is separate cooling/power/chassis for each PE -really eats up rack space and is power inefficient.
On the other hand I'm mindful of added complexity in terms of maintaining resiliency in this external CP environment.

Biggest problem is though that this node-slicing/external x86 based CP is very fresh and I'm not aware of same thing available in cisco or other vendors and unfortunately I need to provide solutions now,



adam

netconsultings.com
::carrier-class solutions for the telecommunications industry::


_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Saku Ytti]

Hey Mark,

Post by Mark Tinka

Post by a***@netconsultings.com
Yes a good practice is to separate internet routes from internal/services
l3vpn routes onto separate BGP control planes (different sessions at least)
so that malformed bgp msg will affect just one part of your overall BGP
infrastructure.

I see you've been giving this advice for quite some time now.

I'm siding with Adam here. His disaster scenario actually happed to me
in 3292. We ran for years VXR VPN route-reflectors, after we changed
them to MX240 we added lot more RR's, with some hard justifications to
management why we need more when we've had no trouble with the count
we had.
After about 3 months of running MX240 reflectors, we got bad BGP
UPDATE and crashed each reflector, which was unprecedented outage in
the history of the network. And tough to explain to management,
considering we just had made the reflection more redundant with some
significant investment. I'm sure they believed we just had cocked it
up, as people don't really believe in chance/randomness, evident how
people justify that things can't be broken, by explaining how in
previous moment in time it wasn't broken, implying that transitioning
from non-broken to broken is impossible.

Note, this is not to trash on Juniper, all vendors have bad BGP
implementations and I'm sure one can fuzz any of them to find crash
bugs.

Not only is it CAPEX irrelevant to have separate RR for IPv4 and IPv6,
but you also get faster convergence, as more CPU cycles, fewer BGP
neighbours, less routes. I view it as cheap insurance as well as very
simple horizontal scaling.

[a***@netconsultings.com]

Sent: Friday, August 17, 2018 2:38 PM
To: Mark Tinka
Subject: Re: [j-nsp] L3VPN/RR/PE on Same router
Hey Mark,

Post by Mark Tinka

Post by a***@netconsultings.com
Yes a good practice is to separate internet routes from
internal/services l3vpn routes onto separate BGP control planes
(different sessions at least) so that malformed bgp msg will affect
just one part of your overall BGP infrastructure.

I see you've been giving this advice for quite some time now.

I'm siding with Adam here. His disaster scenario actually happed to me in
3292. We ran for years VXR VPN route-reflectors, after we changed them to
MX240 we added lot more RR's, with some hard justifications to
management why we need more when we've had no trouble with the count
we had.
After about 3 months of running MX240 reflectors, we got bad BGP UPDATE
and crashed each reflector, which was unprecedented outage in the history
of the network. And tough to explain to management, considering we just
had made the reflection more redundant with some significant investment.
I'm sure they believed we just had cocked it up, as people don't really
believe in chance/randomness, evident how people justify that things can't
be broken, by explaining how in previous moment in time it wasn't broken,
implying that transitioning from non-broken to broken is impossible.
Note, this is not to trash on Juniper, all vendors have bad BGP
implementations and I'm sure one can fuzz any of them to find crash bugs.

Oh yeah for sure, the XR RRs too were crashing upon reception of malformed BGP updates in the past.

Currently XR BGP is *somewhat protected by the "BGP Attribute Filter and Enhanced Attribute Error
Handling" (now RFC 7606) which already proved itself to me (just got a log msg informing me the malformed attribute was deleted instead of an important transit session reset).
Unfortunately can't enable it on junos as the code we run would instead of session reset crashed the rpd due to a bug if the RFC 7606 feature would be enabled.

*But still I'd be haunted by what could happen if RFC 7606 would have missed something and that thing would then crash BGP on RRs, can't afford that happening.

Not only is it CAPEX irrelevant to have separate RR for IPv4 and IPv6, but you
also get faster convergence, as more CPU cycles, fewer BGP neighbours, less
routes. I view it as cheap insurance as well as very simple horizontal scaling.

And going virtual this really is a marginal spend in the grand scheme of things.

adam

netconsultings.com
::carrier-class solutions for the telecommunications industry::


_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Robert Raszuk]

Post by a***@netconsultings.com
and that thing would then crash BGP on RRs, can't afford that happening.

Then best thing is to run two or three RRs in parallel each using different
BGP code base - even for the same AFI/SAFI pair

I am seeing number of networks running single vendor RRs and when things
melt they run around and claim that the problem was was really so rear and
unexpected :) Well usually bugs are of unexpected nature ....

Thx,
R.

Post by a***@netconsultings.com

Sent: Friday, August 17, 2018 2:38 PM
To: Mark Tinka
Subject: Re: [j-nsp] L3VPN/RR/PE on Same router
Hey Mark,

Post by Mark Tinka

Post by a***@netconsultings.com
Yes a good practice is to separate internet routes from
internal/services l3vpn routes onto separate BGP control planes
(different sessions at least) so that malformed bgp msg will affect
just one part of your overall BGP infrastructure.

I see you've been giving this advice for quite some time now.

I'm siding with Adam here. His disaster scenario actually happed to me in
3292. We ran for years VXR VPN route-reflectors, after we changed them to
MX240 we added lot more RR's, with some hard justifications to
management why we need more when we've had no trouble with the count
we had.
After about 3 months of running MX240 reflectors, we got bad BGP UPDATE
and crashed each reflector, which was unprecedented outage in the history
of the network. And tough to explain to management, considering we just
had made the reflection more redundant with some significant investment.
I'm sure they believed we just had cocked it up, as people don't really
believe in chance/randomness, evident how people justify that things

can't

be broken, by explaining how in previous moment in time it wasn't broken,
implying that transitioning from non-broken to broken is impossible.
Note, this is not to trash on Juniper, all vendors have bad BGP
implementations and I'm sure one can fuzz any of them to find crash bugs.

Oh yeah for sure, the XR RRs too were crashing upon reception of malformed
BGP updates in the past.
Currently XR BGP is *somewhat protected by the "BGP Attribute Filter and
Enhanced Attribute Error
Handling" (now RFC 7606) which already proved itself to me (just got a log
msg informing me the malformed attribute was deleted instead of an
important transit session reset).
Unfortunately can't enable it on junos as the code we run would instead of
session reset crashed the rpd due to a bug if the RFC 7606 feature would be
enabled.
*But still I'd be haunted by what could happen if RFC 7606 would have
missed something and that thing would then crash BGP on RRs, can't afford
that happening.

Not only is it CAPEX irrelevant to have separate RR for IPv4 and IPv6,

but you

also get faster convergence, as more CPU cycles, fewer BGP neighbours,

less

routes. I view it as cheap insurance as well as very simple horizontal

scaling.
And going virtual this really is a marginal spend in the grand scheme of things.
adam
netconsultings.com
_______________________________________________
https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Youssef Bengelloun-Zahr]

Hi,

Post by Robert Raszuk

Post by a***@netconsultings.com
and that thing would then crash BGP on RRs, can't afford that happening.

Then best thing is to run two or three RRs in parallel each using different
BGP code base - even for the same AFI/SAFI pair
I am seeing number of networks running single vendor RRs and when things
melt they run around and claim that the problem was was really so rear and
unexpected :) Well usually bugs are of unexpected nature ....

And I have seen the opposite, ie networks running multiple vendor RRs, ending up with crashs because of buggy BGP implementations.

At the end of the day, it is a question of tossing a coin and hopping it will fall on the right side.

Post by Robert Raszuk
Thx,
R.

Post by a***@netconsultings.com

Sent: Friday, August 17, 2018 2:38 PM
To: Mark Tinka
Subject: Re: [j-nsp] L3VPN/RR/PE on Same router
Hey Mark,

Post by Mark Tinka

Post by a***@netconsultings.com
Yes a good practice is to separate internet routes from
internal/services l3vpn routes onto separate BGP control planes
(different sessions at least) so that malformed bgp msg will affect
just one part of your overall BGP infrastructure.

I see you've been giving this advice for quite some time now.

I'm siding with Adam here. His disaster scenario actually happed to me in
3292. We ran for years VXR VPN route-reflectors, after we changed them to
MX240 we added lot more RR's, with some hard justifications to
management why we need more when we've had no trouble with the count
we had.
After about 3 months of running MX240 reflectors, we got bad BGP UPDATE
and crashed each reflector, which was unprecedented outage in the history
of the network. And tough to explain to management, considering we just
had made the reflection more redundant with some significant investment.
I'm sure they believed we just had cocked it up, as people don't really
believe in chance/randomness, evident how people justify that things

can't

be broken, by explaining how in previous moment in time it wasn't broken,
implying that transitioning from non-broken to broken is impossible.
Note, this is not to trash on Juniper, all vendors have bad BGP
implementations and I'm sure one can fuzz any of them to find crash bugs.

Oh yeah for sure, the XR RRs too were crashing upon reception of malformed
BGP updates in the past.
Currently XR BGP is *somewhat protected by the "BGP Attribute Filter and
Enhanced Attribute Error
Handling" (now RFC 7606) which already proved itself to me (just got a log
msg informing me the malformed attribute was deleted instead of an
important transit session reset).
Unfortunately can't enable it on junos as the code we run would instead of
session reset crashed the rpd due to a bug if the RFC 7606 feature would be
enabled.
*But still I'd be haunted by what could happen if RFC 7606 would have
missed something and that thing would then crash BGP on RRs, can't afford
that happening.

Not only is it CAPEX irrelevant to have separate RR for IPv4 and IPv6,

but you

also get faster convergence, as more CPU cycles, fewer BGP neighbours,

less

routes. I view it as cheap insurance as well as very simple horizontal

scaling.
And going virtual this really is a marginal spend in the grand scheme of things.
adam
netconsultings.com
_______________________________________________
https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.

[Robert Raszuk]

Post by Youssef Bengelloun-Zahr
And I have seen the opposite, ie networks running multiple vendor RRs,
ending up with crashs because of buggy BGP implementations.

Hmmm since usually IBGP RRs do not talk to each other (leave alone RR
hierarchy aside) what you are essentially endorsing is single vendor
networks right ?

If I have Cisco PEs and Juniper RRs by your description it may crash ...
hence better to avoid it - right ?.

Good thing this is thread about iBGP not eBGP :):)

Thx
R.

Post by Youssef Bengelloun-Zahr
Hi,

Post by Robert Raszuk

Post by a***@netconsultings.com
and that thing would then crash BGP on RRs, can't afford that happening.

Then best thing is to run two or three RRs in parallel each using

different

Post by Robert Raszuk
BGP code base - even for the same AFI/SAFI pair
I am seeing number of networks running single vendor RRs and when things
melt they run around and claim that the problem was was really so rear

and

Post by Robert Raszuk
unexpected :) Well usually bugs are of unexpected nature ....

And I have seen the opposite, ie networks running multiple vendor RRs,
ending up with crashs because of buggy BGP implementations.
At the end of the day, it is a question of tossing a coin and hopping it
will fall on the right side.

Post by Robert Raszuk
Thx,
R.

Post by a***@netconsultings.com

Sent: Friday, August 17, 2018 2:38 PM
To: Mark Tinka
Subject: Re: [j-nsp] L3VPN/RR/PE on Same router
Hey Mark,

Post by Mark Tinka

Post by a***@netconsultings.com
Yes a good practice is to separate internet routes from
internal/services l3vpn routes onto separate BGP control planes
(different sessions at least) so that malformed bgp msg will affect
just one part of your overall BGP infrastructure.

I see you've been giving this advice for quite some time now.

I'm siding with Adam here. His disaster scenario actually happed to me

in

Post by Robert Raszuk

Post by a***@netconsultings.com

3292. We ran for years VXR VPN route-reflectors, after we changed them

to

Post by Robert Raszuk

Post by a***@netconsultings.com

MX240 we added lot more RR's, with some hard justifications to
management why we need more when we've had no trouble with the count
we had.
After about 3 months of running MX240 reflectors, we got bad BGP UPDATE
and crashed each reflector, which was unprecedented outage in the

history

Post by Robert Raszuk

Post by a***@netconsultings.com

of the network. And tough to explain to management, considering we just
had made the reflection more redundant with some significant

investment.

Post by Robert Raszuk

Post by a***@netconsultings.com

I'm sure they believed we just had cocked it up, as people don't really
believe in chance/randomness, evident how people justify that things

can't

be broken, by explaining how in previous moment in time it wasn't

broken,

Post by Robert Raszuk

Post by a***@netconsultings.com

implying that transitioning from non-broken to broken is impossible.
Note, this is not to trash on Juniper, all vendors have bad BGP
implementations and I'm sure one can fuzz any of them to find crash

bugs.

Post by Robert Raszuk

Post by a***@netconsultings.com
Oh yeah for sure, the XR RRs too were crashing upon reception of

malformed

Post by Robert Raszuk

Post by a***@netconsultings.com
BGP updates in the past.
Currently XR BGP is *somewhat protected by the "BGP Attribute Filter and
Enhanced Attribute Error
Handling" (now RFC 7606) which already proved itself to me (just got a

log

Post by Robert Raszuk

Post by a***@netconsultings.com
msg informing me the malformed attribute was deleted instead of an
important transit session reset).
Unfortunately can't enable it on junos as the code we run would instead

of

Post by Robert Raszuk

Post by a***@netconsultings.com
session reset crashed the rpd due to a bug if the RFC 7606 feature

would be

Post by Robert Raszuk

Post by a***@netconsultings.com
enabled.
*But still I'd be haunted by what could happen if RFC 7606 would have
missed something and that thing would then crash BGP on RRs, can't

afford

Post by Robert Raszuk

Post by a***@netconsultings.com
that happening.

Not only is it CAPEX irrelevant to have separate RR for IPv4 and IPv6,

but you

also get faster convergence, as more CPU cycles, fewer BGP neighbours,

less

routes. I view it as cheap insurance as well as very simple horizontal

scaling.
And going virtual this really is a marginal spend in the grand scheme of things.
adam
netconsultings.com
_______________________________________________
https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://p

[a***@netconsultings.com]

Sent: Friday, August 17, 2018 3:43 PM
To: Robert Raszuk
Subject: Re: [j-nsp] L3VPN/RR/PE on Same router
Hi,

Post by Robert Raszuk

Post by a***@netconsultings.com
and that thing would then crash BGP on RRs, can't afford that happening.

Then best thing is to run two or three RRs in parallel each using
different BGP code base - even for the same AFI/SAFI pair
I am seeing number of networks running single vendor RRs and when
things melt they run around and claim that the problem was was really
so rear and unexpected :) Well usually bugs are of unexpected nature ....

And I have seen the opposite, ie networks running multiple vendor RRs,
ending up with crashs because of buggy BGP implementations.
At the end of the day, it is a question of tossing a coin and hopping it will fall
on the right side.

It's about increasing the odds of it to fall on the right side,

Yes you can have multiple vendors that have the same quagga/bird BGP message parser code,
But comparing say XR and Junos, judging from the rest of the inner workings I could experience empirically, I'd say they are sufficiently different implementations.


adam

netconsultings.com
::carrier-class solutions for the telecommunications industry::

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https:/

[Robert Raszuk]

Post by a***@netconsultings.com
It's about increasing the odds of it to fall on the right side,

Exactly !

Post by a***@netconsultings.com
But comparing say XR and Junos, judging from the rest of the inner workings

I could experience empirically, I'd say they are sufficiently different

Post by a***@netconsultings.com
implementations.

True. In fact even XE & XR BGP code core is quite different in spite of
number of failed
attempts to at least make new bgp features to share code.

The bottom line is that if you get badly malformed update, broken
attribute, illegal NLRIs, mistaken blast of BGP-LS etc .. the probability
that all of those BGP implementations crash is much less likely then when
compared that your chosen one will work even if you run two instances of
it.
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Mark Tinka]

Post by Robert Raszuk
Then best thing is to run two or three RRs in parallel each using
different BGP code base - even for the same AFI/SAFI pair

With experience, I could consider this. But for now, it does seem like
overkill.

Either, not totally opposed to the idea.

Mark.
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Mark Tinka]

Post by a***@netconsultings.com
And going virtual this really is a marginal spend in the grand scheme of things.

If you're not going virtual for RR's, you are missing out.

Mark.
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Mark Tinka]

Post by Saku Ytti
I'm siding with Adam here. His disaster scenario actually happed to me
in 3292. We ran for years VXR VPN route-reflectors, after we changed
them to MX240 we added lot more RR's, with some hard justifications to
management why we need more when we've had no trouble with the count
we had.

Agreed, and I see the use-case as a possible pain-point, particularly
based on your experience.

We'll need to evaluate this on our side, as the path to this road is
onerous.

Post by Saku Ytti
Not only is it CAPEX irrelevant to have separate RR for IPv4 and IPv6,
but you also get faster convergence, as more CPU cycles, fewer BGP
neighbours, less routes. I view it as cheap insurance as well as very
simple horizontal scaling.

This might be the case when using real routers as RR's. But having used
CSR1000v on x86 hardware for years now, you are not lacking for CPU
performance, even with 10's of sessions per RR, many of them with full
tables and several SAFI's.

Mark.
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Alexander Arseniev via juniper-nsp]

Hello,

Yes there is

https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/advertise-from-main-vpn-table-edit-protocols-bgp.html

Also, either don't configure "family route-target" on this combined
PE/RR at all, or configure "family route-target advertise-default" in
order to be able to receive routes from all VRFs in Your network.

HTH

Thanks

Alex

Post by tim tiriche
Hello,
I have a MPLS PE (L3VPN) router that is acting as full mesh iBGP within the
US. The other routers in the US are not RR and regular iBGP. This router
also acts as RR for Europe and takes in full BGP table. Is there some
caveats to watch out for?
_______________________________________________
https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Sebastian Wiesinger]

Post by tim tiriche
Hello,
I have a MPLS PE (L3VPN) router that is acting as full mesh iBGP within the
US. The other routers in the US are not RR and regular iBGP. This router
also acts as RR for Europe and takes in full BGP table. Is there some
caveats to watch out for?

You will run into problems if you try to have a Hub-and-Spoke L3VPN
with a downstream hub on the PE. Routes do not get advertised because
they already got exported from one routing table to another. Or how
Juniper states it in:

https://www.juniper.net/documentation/en_US/junos/topics/example/bgp-vpn-session-flap-prevention.html

"The route export is not performed if the route in instance.inet.0 is
a secondary route. In Junos OS, a route is only exported one time from
one routing table as a primary route to another routing table as a
secondary route. "

Same when you enable the "advertise-from-main-vpn-tables" knob in
global bgp config.

Regards

Sebastian

[Antti Ristimäki]

Hi,

There might be some corner cases where running a combined RR/PE can cause mysterious issues. For example, there was (or is - I'm not sure whether it's fixed or not) an issue that a RR didn't advertise iBGP learned VPLS routes when the RR itself had a local attachment circuit in the given VPLS instance.

Antti

Post by tim tiriche
Hello,
I have a MPLS PE (L3VPN) router that is acting as full mesh iBGP within the
US. The other routers in the US are not RR and regular iBGP. This router
also acts as RR for Europe and takes in full BGP table. Is there some
caveats to watch out for?
_______________________________________________
https://puck.nether.net/mailman/listinfo/juniper-nsp
--

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp