[j-nsp] I've got some bone head problem on an srx...but I don't see it.

Discussion:

Morgan McLean

2013-06-12 01:29:08 UTC

I have an SRX cluster at an office with a single connection to the web at
the moment. It has a couple ipsec connections out to our datacenters, and a
couple local subnets hanging on RETH interfaces.

For the life of me, I can't figure out why I'm unable to ping out from this
system. Even if I try to ping the point to point between us and Verizon, a
direct route, it won't work unless I specify the source address as our
local interface address.

Outbound nat from clients behind the SRX works fine. The loopback is in
trust, and I have a couple zones + trust with a source nat rule using the
verizon interface IP as the egress point. Destination nat rules work.

So everything seems to work...except from the SRX. As a result, we cannot
ping the SRX remotely...but again IPSEC works.

Any great tips? None of our other SRX's behave like this...and its driving
me nuts!

--
Thanks,
Morgan

Morgan McLean

2013-06-12 02:12:27 UTC

Permalink

--- JUNOS 10.4R7.5 built 2011-09-08 07:12:35 UTC
{primary:node0}
mmclean at srxhost> show route 4.2.2.2

inet.0: 248 destinations, 251 routes (244 active, 4 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0 *[Static/5] 2w5d 01:07:06

to x.x.x.237 via ge-2/0/23.0

{primary:node0}
mmclean at srxhost > ping 4.2.2.2
PING 4.2.2.2 (4.2.2.2): 56 data bytes
^C
--- 4.2.2.2 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

{primary:node0}
mmclean at srxhost> ping 4.2.2.2 source x.x.x.238
PING 4.2.2.2 (4.2.2.2): 56 data bytes
64 bytes from 4.2.2.2: icmp_seq=0 ttl=59 time=12.020 ms
^C
--- 4.2.2.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 12.020/12.020/12.020/0.000 ms

{primary:node0}

I've gotten a couple replies off list. There is an any policy from trust
to untrust, and the untrust zone does have host inbound traffic ping
enabled. I think the ping not responding is a byproduct of whatever is
going on, though.
Morgan

Post by Morgan McLean
I have an SRX cluster at an office with a single connection to the web at
the moment. It has a couple ipsec connections out to our datacenters, and a
couple local subnets hanging on RETH interfaces.
For the life of me, I can't figure out why I'm unable to ping out from
this system. Even if I try to ping the point to point between us and
Verizon, a direct route, it won't work unless I specify the source address
as our local interface address.
Outbound nat from clients behind the SRX works fine. The loopback is in
trust, and I have a couple zones + trust with a source nat rule using the
verizon interface IP as the egress point. Destination nat rules work.
So everything seems to work...except from the SRX. As a result, we cannot
ping the SRX remotely...but again IPSEC works.
Any great tips? None of our other SRX's behave like this...and its
driving me nuts!
--
Thanks,
Morgan

--
Thanks,
Morgan

Andrew Jones

2013-06-12 03:29:06 UTC

Permalink

Turn on some flow traceoptions and look at what source address the
packets contain... it sounds like the srx might be picking an
inappropriate source address, and finding out what that address is will
aid troubleshooting.

Post by Morgan McLean
--- JUNOS 10.4R7.5 built 2011-09-08 07:12:35 UTC
{primary:node0}
mmclean at srxhost> show route 4.2.2.2
inet.0: 248 destinations, 251 routes (244 active, 4 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
0.0.0.0/0 *[Static/5] 2w5d 01:07:06

to x.x.x.237 via ge-2/0/23.0

{primary:node0}
mmclean at srxhost > ping 4.2.2.2
PING 4.2.2.2 (4.2.2.2): 56 data bytes
^C
--- 4.2.2.2 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
{primary:node0}
mmclean at srxhost> ping 4.2.2.2 source x.x.x.238
PING 4.2.2.2 (4.2.2.2): 56 data bytes
64 bytes from 4.2.2.2: icmp_seq=0 ttl=59 time=12.020 ms
^C
--- 4.2.2.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 12.020/12.020/12.020/0.000 ms
{primary:node0}

Post by Morgan McLean
I have an SRX cluster at an office with a single connection to the web at
the moment. It has a couple ipsec connections out to our
datacenters, and a
couple local subnets hanging on RETH interfaces.
For the life of me, I can't figure out why I'm unable to ping out from
this system. Even if I try to ping the point to point between us and
Verizon, a direct route, it won't work unless I specify the source address
as our local interface address.
Outbound nat from clients behind the SRX works fine. The loopback is in
trust, and I have a couple zones + trust with a source nat rule using the
verizon interface IP as the egress point. Destination nat rules work.
So everything seems to work...except from the SRX. As a result, we cannot
ping the SRX remotely...but again IPSEC works.
Any great tips? None of our other SRX's behave like this...and its
driving me nuts!
--
Thanks,
Morgan

--
Thanks,
Morgan

Morgan McLean

2013-06-12 02:09:05 UTC

Permalink

I've gotten a couple replies off list. There is an any policy from trust to
untrust, and the untrust zone does have host inbound traffic ping enabled.
I think the ping not responding is a byproduct of whatever is going on,
though.

Morgan

--
Thanks,
Morgan

Ben Dale

2013-06-12 04:47:13 UTC

Permalink

Post by Morgan McLean
I have an SRX cluster at an office with a single connection to the web at
the moment. It has a couple ipsec connections out to our datacenters, and a
couple local subnets hanging on RETH interfaces.
For the life of me, I can't figure out why I'm unable to ping out from this
system. Even if I try to ping the point to point between us and Verizon, a
direct route, it won't work unless I specify the source address as our
local interface address.
Outbound nat from clients behind the SRX works fine. The loopback is in
trust, and I have a couple zones + trust with a source nat rule using the
verizon interface IP as the egress point. Destination nat rules work.
So everything seems to work...except from the SRX. As a result, we cannot
ping the SRX remotely...but again IPSEC works.
Any great tips? None of our other SRX's behave like this...and its driving
me nuts!

Being a cluster, this isn't fxp0 interface shenanigans is it? Do you have the managemen function zone applied to fxp0?

Ben

Morgan McLean

2013-06-12 04:50:50 UTC

Permalink

The issue is I had default address selection enabled. I don't use FXP on
any of my devices, I don't believe in it. I think routed loopback is a more
sound management option in the event of a failure.

Regardless, the weird thing is the loopback was in trust zone, which is
allowed to get to untrust, is part of the source nat rule, etc. No idea why
it wasn't working. I'd forgot I enabled it, and it escaped my eyes when I
looked at the config block.

I might re-enable it and try a packet trace...which I had done before but
saw nothing. I'll report back in a few minutes.

Morgan

Post by Morgan McLean

Post by Morgan McLean
I have an SRX cluster at an office with a single connection to the web at
the moment. It has a couple ipsec connections out to our datacenters,

and a

Post by Morgan McLean
couple local subnets hanging on RETH interfaces.
For the life of me, I can't figure out why I'm unable to ping out from

this

Post by Morgan McLean
system. Even if I try to ping the point to point between us and Verizon,

Post by Morgan McLean
direct route, it won't work unless I specify the source address as our
local interface address.
Outbound nat from clients behind the SRX works fine. The loopback is in
trust, and I have a couple zones + trust with a source nat rule using the
verizon interface IP as the egress point. Destination nat rules work.
So everything seems to work...except from the SRX. As a result, we cannot
ping the SRX remotely...but again IPSEC works.
Any great tips? None of our other SRX's behave like this...and its

driving

Post by Morgan McLean
me nuts!

Being a cluster, this isn't fxp0 interface shenanigans is it? Do you have
the managemen function zone applied to fxp0?
Ben

--
Thanks,
Morgan

Morgan McLean

2013-06-12 04:59:46 UTC

Permalink

I rolled back and ran a ping to a host out on the net. Heres the trace...is
the fact that its coming from junos-self screwing things up?

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: chose interface .local..0 as
incoming nat if.

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:flow_first_rule_dst_xlate: packet
192.168.29.11->192.81.130.21 nsp2 0.0.0.0->192.81.130.21.

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:flow_first_routing: call
flow_route_lookup(): src_ip 192.168.29.11, x_dst_ip 192.81.130.21, in ifp
.local..0, out ifp N/A sp 8, dp 207, ip_proto 1, tos 0

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:Doing DESTINATION addr
route-lookup

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: routed (x_dst_ip 192.81.130.21)
from junos-self (.local..0 in 0) to ge-2/0/23.0, Next-hop: 157.130.198.237

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: policy search from zone
junos-self-> zone untrust (0x0,0x800cf,0xcf)

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: app 0, timeout 60s, curr ageout
60s

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:flow_first_src_xlate:
nat_src_xlated: False, nat_src_xlate_failed: False

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:flow_first_src_xlate: src nat
returns status: 0, rule/pool id: 0/0, pst_nat: False.

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: dip id = 0/0, 192.168.29.11/8->
192.168.29.11/8

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: choose interface ge-2/0/23.0 as
outgoing phy if

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:is_loop_pak: No loop: on ifp:
ge-2/0/23.0, addr: 192.81.130.21, rtt_idx:0

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: check nsrp pak fwd: in_tun=0x0,
VSD 0 for out ifp ge-2/0/23.0

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: vsd 0 is active

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:jsf sess interest check. regd
plugins 13

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: Allocating plugin info block for
12 plugin(s) from OL

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:-jsf int check: plugin id 1,
svc_req 0x0. rc 4

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:-jsf int check: plugin id 2,
svc_req 0x2. rc 4

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:-jsf int check: plugin id 3,
svc_req 0x0. rc 4

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:-jsf int check: plugin id 5,
svc_req 0x0. rc 4

Jun 11 21:51:22
21:51:21.1472397:CID-1:RT:+++++++++++jsf_test_plugin_data_evh: 3

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:-jsf int check: plugin id 6,
svc_req 0x0. rc 4

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:-jsf int check: plugin id 7,
svc_req 0x0. rc 4

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:-jsf int check: plugin id 8,
svc_req 0x0. rc 4

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:-jsf int check: plugin id 10,
svc_req 0x0. rc 4

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:-jsf int check: plugin id 11,
svc_req 0x0. rc 2

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: No JSF plugins enabled for
session

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: Releasing plugin info block for
12 plugin(s) to OL

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:flow_first_service_lookup():
natp(0x58c92bc8): app_id, 0(0).

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: service lookup identified
service 0.

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: flow_first_final_check: in
<.local..0>, out <ge-2/0/23.0>

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:construct v4 vector for nsp2

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: existing vector list
220-4942b678.

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: Session (id:104814) created for
first pak 220

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:
flow_first_install_session======> 0x58c92bc8

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: nsp 0x58c92bc8, nsp2 0x58c92c2c

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: make_nsp_ready_no_resolve()

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: route lookup: dest-ip
192.168.29.11 orig ifp .local..0 output_ifp .local..0 orig-zone 2 out-zone
2 vsd 0

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: route to 192.168.29.11

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:Installing c2s NP session wing

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:Installing s2c NP session wing

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: flow got session.

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: flow session id 104814

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: vector bits 0x220 vector
0x4942b678

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: vsd 0 is active

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:mbuf 0x44f31b80, exit nh 0x150010

Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:flow_process_pkt_exception:
Freeing lpak 51990930 associated with mbuf 0x44f31b80

Post by Morgan McLean

Post by Morgan McLean
I have an SRX cluster at an office with a single connection to the web at
the moment. It has a couple ipsec connections out to our datacenters,

and a

Post by Morgan McLean
couple local subnets hanging on RETH interfaces.
For the life of me, I can't figure out why I'm unable to ping out from

this

Post by Morgan McLean
system. Even if I try to ping the point to point between us and Verizon,

driving

Post by Morgan McLean
me nuts!

Being a cluster, this isn't fxp0 interface shenanigans is it? Do you have
the managemen function zone applied to fxp0?
Ben

--
Thanks,
Morgan

pkc_mls

2013-06-12 14:28:26 UTC

Permalink

Can you run a "show security flow session destination-prefix
192.81.130.21" to make sure session
is correctly established and NAT is performed aswell ?

Pavel Lunin

2013-06-13 16:25:14 UTC

Permalink

Post by Morgan McLean
I rolled back and ran a ping to a host out on the net. Heres the trace...is
the fact that its coming from junos-self screwing things up?
Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:flow_first_routing: call
flow_route_lookup(): src_ip 192.168.29.11, x_dst_ip 192.81.130.21, in ifp
.local..0, out ifp N/A sp 8, dp 207, ip_proto 1, tos 0

[...]

Post by Morgan McLean
nat_src_xlated: False, nat_src_xlate_failed: False
Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:flow_first_src_xlate: src nat
returns status: 0, rule/pool id: 0/0, pst_nat: False.
Jun 11 21:51:22 21:51:21.1472397:CID-1:RT: dip id = 0/0, 192.168.29.11/8->
192.168.29.11/8

This means you were sending packets to the Internet from the source IP
192.168.29.11.

Morgan McLean

2013-06-13 17:20:54 UTC

Permalink

Yes. the issue appears that I was not putting junos-self or junos-host at
the source security zone for the nat rule. I have yet to try it, will test
today.

Thanks!
Morgan

Post by Morgan McLean

Post by Morgan McLean
I rolled back and ran a ping to a host out on the net. Heres the

trace...is

Post by Morgan McLean
the fact that its coming from junos-self screwing things up?
Jun 11 21:51:22 21:51:21.1472397:CID-1:RT:flow_first_routing: call
flow_route_lookup(): src_ip 192.168.29.11, x_dst_ip 192.81.130.21, in ifp
.local..0, out ifp N/A sp 8, dp 207, ip_proto 1, tos 0

[...]

192.168.29.11/8->

Post by Morgan McLean
192.168.29.11/8

This means you were sending packets to the Internet from the source IP
192.168.29.11.
_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

--
Thanks,
Morgan