[j-nsp] 32-Bit JunOS on the 64-Bit Routing Engines

Discussion:

Thomas Eichhorn

2011-08-24 07:12:16 UTC

Hi all,

I just discussed the following with my SE:

I wanted to get new 64Bit REs with some new gear,
but run the 32-Bit JunOS on them - he denied that
this is possible.

I tried to research that, but have not yet found
something in the docs - does anybody here have some clue
about that?

As the REs are 'only' standard PCs, I do not see any reason
for them to be not capable of running 'legacy' 32Bit JunOS.

I would be really glad if someone has some clue about that and
could unearth the truth.

Thanks,
Tom

Johannes Resch

2011-08-24 08:28:19 UTC

Permalink

Hi,

Post by Thomas Eichhorn
Hi all,
I wanted to get new 64Bit REs with some new gear,
but run the 32-Bit JunOS on them - he denied that
this is possible.
I tried to research that, but have not yet found
something in the docs - does anybody here have some clue
about that?
As the REs are 'only' standard PCs, I do not see any reason
for them to be not capable of running 'legacy' 32Bit JunOS.
I would be really glad if someone has some clue about that and
could unearth the truth.

We are in the same situation, and have received the same -disappointing-
answer from our SE.

I've just given this a quick try on a MX running 10.4S1.1 (64bit) with
RE-S-1800X4-16G, and it seems it is not possible to get 32bit JunOS
installed using a jinstall image.

jresch at testlab-RR03-re0> request system software add no-validate reboot
re1 /var/tmp/jinstall-11.2R1.10-domestic-signed.tgz
Pushing bundle to re1
Installing package '/var/tmp/jinstall-11.2R1.10-domestic-signed.tgz' ...
Verified jinstall-11.2R1.10-domestic.tgz signed by PackageProduction_11_2_0
Adding jinstall...
ERROR: Package jinstall is not compatible - i386 vs {amd64}
ERROR: jinstall fails requirements check
ERROR: jinstall-11.2R1.10-domestic-signed fails post-install
Installation failed for package
'/var/tmp/jinstall-11.2R1.10-domestic-signed.tgz'

Interestingly, on T640/T1600 32bit JunOS _is_ supported on
a 64bit RE-DUO-C1800-8G.
These REs will also support 64bit JunOS in the future.

Both REs types use Intel Xeon CPUs.
So I would assume the limitation on MX is non-technical.

RE-DUO-C1800-8G has
CPU: Intel(R) Xeon(R) CPU L5215 @ 1.86GHz (1862.01-MHz 686-class CPU)

RE-S-1800X4-16G has
CPU: Intel(R) Xeon(R) CPU C5518 @ 1.73GHz (1729.11-MHz K8-class CPU)

cheers,
-jr

Trond Hastad

2011-08-25 06:11:18 UTC

Permalink

Post by Johannes Resch
Hi,

Did you try with the force switch?

request system software add
/var/tmp/jinstall-11.2R1.10-domestic-signed.tgz force

I do not think it is a supported combination but it should work.

--
regards
Trond

Keegan Holley

2011-08-24 12:19:50 UTC

Permalink

Interestingly enough my SE told us this is possible at lease on our Mx480 and MX960 boxes. Our lab boxes are otherwise engaged at the moment so we havent tested. One note regarding general computing though. The processor can only address 4G (3.8 or so actually) of ram with a 32 bit word size. So even if you get the re's running the 32 bit code they will only register 4G of the precious 16G.

Sent from my iPhone

Thomas Eichhorn

2011-08-24 12:27:14 UTC

Permalink

Yeah, that is clear - my original point is:

I do not trust the 64bit software - I have more faith in the 32bit software.

As per now, it has equal cost to order an MX960 with 32b-4G-RE or
64b-16G-RE.

So of course I would order the bigger RE but only if I can use the
the matured software...

Tom

Post by Keegan Holley
Interestingly enough my SE told us this is possible at lease on our Mx480 and MX960 boxes. Our lab boxes are otherwise engaged at the moment so we havent tested. One note regarding general computing though. The processor can only address 4G (3.8 or so actually) of ram with a 32 bit word size. So even if you get the re's running the 32 bit code they will only register 4G of the precious 16G.
Sent from my iPhone

Robert Raszuk

2011-08-24 12:45:13 UTC

Permalink

Hi Thomas,

I agree with your reasoning - it is very practical :)

However I am not sure what you exactly call by "software". AFAIK the BSD
kernel has been 64 bit enabled and capable years ago so from that point
of view this is mature.

For a change various processes for example RPD last time I checked was
still 32 bit for most platforms other then JCS.

I am not sure what is the status with porting other processes to be 64
bit enabled and honestly I am not sure it is needed across the board.

So I would recommend checking with your SE if the 64 Junos you are
getting to run on 64bit RE is really fully rewritten across or processes
first. Maybe you will find out that it is not.

Just looking below it seems that jinstall is failing because it detects
cpu mismatch "i386 vs {amd64}" To fix that I think the best way would be
to talk to friends in Juniper to recompile/publish junos for your
platform/RE.

Cheers,
R.

Post by Thomas Eichhorn
I do not trust the 64bit software - I have more faith in the 32bit software.
As per now, it has equal cost to order an MX960 with 32b-4G-RE or
64b-16G-RE.
So of course I would order the bigger RE but only if I can use the
the matured software...
Tom

Post by Keegan Holley
Interestingly enough my SE told us this is possible at lease on our
Mx480 and MX960 boxes. Our lab boxes are otherwise engaged at the
moment so we havent tested. One note regarding general computing
though. The processor can only address 4G (3.8 or so actually) of ram
with a 32 bit word size. So even if you get the re's running the 32
bit code they will only register 4G of the precious 16G.
Sent from my iPhone

_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Chris Adams

2011-08-24 13:13:15 UTC

Permalink

Well, that isn't entirely true. Intel added the Physical Address
Extension to the Pentium Pro many years ago (and virtually everything
claiming to be i686 compatible has PAE). PAE allows the OS kernel to
address up to 36 bits of RAM (64G), just not all at once. In general, a
given program can still only see 32 bits, unless it does special bank
switching.

I don't know about PAE support on FreeBSD or JUNOS, but it does exist in
all x86 Juniper hardware.

--
Chris Adams <cmadams at hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

Keegan Holley

2011-08-24 13:25:37 UTC

Permalink

Sent from my iPhone

Post by Chris Adams

I've never heard of this actually being used though. Maybe I'm wrong though. Most people just modified their code to support 64 bit and stopped there. I also haven't seen any boards RE's or regular Mobo's with 32 bit procs and support for more that the 4G of RAM.

Post by Chris Adams
In general, a
given program can still only see 32 bits, unless it does special bank
switching.
I don't know about PAE support on FreeBSD or JUNOS, but it does exist in
all x86 Juniper hardware.

Interesting.

Post by Chris Adams
--
Chris Adams <cmadams at hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Chris Adams

2011-08-24 13:32:14 UTC

Permalink

Post by Keegan Holley

Post by Chris Adams
Well, that isn't entirely true. Intel added the Physical Address
Extension to the Pentium Pro many years ago (and virtually everything
claiming to be i686 compatible has PAE). PAE allows the OS kernel to
address up to 36 bits of RAM (64G), just not all at once.

A good number of 32-bit server motherboards supported PAE, and the Linux
kernel has supported it for years. IIRC about the only program that
directly used PAE under Linux was Oracle (before 64 bit x86 platforms
were widely used).

--
Chris Adams <cmadams at hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

Joel jaeggli

2011-08-25 17:47:07 UTC

Permalink

Post by Keegan Holley
Sent from my iPhone

Post by Chris Adams

There are plenty of machines that do. virtually every intel system since
the pentium pro (except the atom) has the hardware if not the bios
support for doing so, that's not germain to the question of whether it's
feasible/useful in an embedded system. In particular, in a system (like
for example a firewall) where kernel datastructures may represent the
overwhelming source of memory utilization, the PAE performance hit may
trivially overwhelm the value of any memory that can otherwise be freed
up for userspace.

64bitness has been the prefered approach for intel based servers since
about 2003, but the embedded lifecycle runs on it's own timeline.

Post by Keegan Holley

Interesting.

_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Jonas Frey (Probe Networks)

2011-08-26 00:56:47 UTC

Permalink

Thats not completely accurate, for example the Intel Atom D525 does run
64bit code.

Post by Joel jaeggli
There are plenty of machines that do. virtually every intel system since
the pentium pro (except the atom) has the hardware if not the bios
support for doing so, that's not germain to the question of whether it's
feasible/useful in an embedded system. In particular, in a system (like
for example a firewall) where kernel datastructures may represent the
overwhelming source of memory utilization, the PAE performance hit may
trivially overwhelm the value of any memory that can otherwise be freed
up for userspace.
64bitness has been the prefered approach for intel based servers since
about 2003, but the embedded lifecycle runs on it's own timeline.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <https://puck.nether.net/pipermail/juniper-nsp/attachments/20110826/e16a200b/attachment.pgp>

Joel jaeggli

2011-08-26 02:13:47 UTC

Permalink

Post by Jonas Frey (Probe Networks)
Thats not completely accurate, for example the Intel Atom D525 does run
64bit code.

there are a number of atoms the support 64bit, I think that the
observation I was making was that there are atoms that don't support
PAE, by virtue of not supporting 4 or more GB of ram.

Post by Jonas Frey (Probe Networks)

Jonas Frey (Probe Networks)

2011-08-26 04:49:17 UTC

Permalink

Well, i do have Atoms running here with 32bit and PAE and 8 GB of ram.
Even the rather slow N270 does support this. The Intel board D525MW
(which includes a soldered D525 Atom) officially (intel website)
supports only 4GB but will run fine with 8 GB.

Of course there might be older Atoms that do not support either PAE
and/or x64. But newer Gen's pretty much do.

Not sure if a Atom is useful in a router anyway...its not that powerfull
as a normal Core i3/i5. With Juniper now using 55xx Xeon's in the
multi-core RE's i guess they decided to beef up the power on them
because of growing IPv6 route tables as well as large (multiple)
VRF/IPv4 tables.

Post by Joel jaeggli

Post by Jonas Frey (Probe Networks)
Thats not completely accurate, for example the Intel Atom D525 does run
64bit code.

there are a number of atoms the support 64bit, I think that the
observation I was making was that there are atoms that don't support
PAE, by virtue of not supporting 4 or more GB of ram.

Post by Jonas Frey (Probe Networks)

Hannes Viertel

2011-08-24 13:32:09 UTC

Permalink

Thomas,

the hardware itself could theoretically run both 32/64 bit junos.

But your SE was completely right. Only 64 bit Junos is systested and therefore supported on the new RE's.

As Robert pointed out, 32/64 bit use the same codebase and at the current point in time only the kernel is enabled to handle the additional memory. Therefore some of the memory maps/footprints changed slightly. No other daemons have moved to 64bit.

But it is still recommended to handle/test/verify 32/64bit sw of the same junos release as separate pieces of software. There might be architectural issues that just affect one piece of sw and not the other.

hope that helps

/hannes

------ Weitergeleitete Nachricht
Von: Thomas Eichhorn <te at te3networks.de>
Datum: Wed, 24 Aug 2011 13:27:14 +0100
An: Keegan Holley <keegan.holley at sungard.com>
Cc: <juniper-nsp at puck.nether.net>
Betreff: Re: [j-nsp] 32-Bit JunOS on the 64-Bit Routing Engines
I do not trust the 64bit software - I have more faith in the 32bit software.
As per now, it has equal cost to order an MX960 with 32b-4G-RE or
64b-16G-RE.
So of course I would order the bigger RE but only if I can use the
the matured software...
Tom

_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
------ Ende der weitergeleiteten Nachricht

Keegan Holley

2011-08-24 23:52:54 UTC

Permalink

Post by Hannes Viertel
As Robert pointed out, 32/64 bit use the same codebase and at the current
point in time only the kernel is enabled to handle the additional memory.
Therefore some of the memory maps/footprints changed slightly. No other
daemons have moved to 64bit.

They are saying that the new 16G RE's can handle 250M routes. How is this
possible if none of the daemons are 64bit?

Post by Hannes Viertel

software.

As per now, it has equal cost to order an MX960 with 32b-4G-RE or
64b-16G-RE.
So of course I would order the bigger RE but only if I can use the
the matured software...
Tom

Post by Keegan Holley
Interestingly enough my SE told us this is possible at lease on our

Mx480 and MX960 boxes. Our lab boxes are otherwise engaged at the moment so
we havent tested. One note regarding general computing though. The
processor can only address 4G (3.8 or so actually) of ram with a 32 bit word
size. So even if you get the re's running the 32 bit code they will only
register 4G of the precious 16G.

Post by Keegan Holley
Sent from my iPhone
On Aug 24, 2011, at 3:12 AM, Thomas Eichhorn<te at te3networks.de>

_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
------ Ende der weitergeleiteten Nachricht

_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Daniel Roesen

2011-08-25 05:32:20 UTC

Permalink

Post by Keegan Holley
They are saying that the new 16G RE's can handle 250M routes. How is this
possible if none of the daemons are 64bit?

Multiple logical-system instances (== multiple rpd processes)? :-)

Best regards,
Daniel

--
CLUE-RIPE -- Jabber: dr at cluenet.de -- dr at IRCnet -- PGP: 0xA85C8AA0

Keegan Holley

2011-08-25 06:18:10 UTC

Permalink

2011/8/25 Daniel Roesen <dr at cluenet.de>

Post by Daniel Roesen

Post by Keegan Holley
They are saying that the new 16G RE's can handle 250M routes. How is

this

Post by Keegan Holley
possible if none of the daemons are 64bit?

Multiple logical-system instances (== multiple rpd processes)? :-)

lol. a) try splitting your table between multiple routers virtual or
otherwise and let me know how that works out. b) So someone shows up and
pitches a router where the specs are based on the power of a single logical
router x the number of logical routers I can create. If my response is to
purchase said contraption and put it in my network, whatever happens then is
my fault. c) It would be extremely ironic if the EOL'd the 16G RE and
release one with a bigger proc before the extra ram becomes useful like they
did with the first 2G RE.

Joel jaeggli

2011-08-25 06:45:29 UTC

Permalink

Post by Keegan Holley
2011/8/25 Daniel Roesen <dr at cluenet.de>

Post by Daniel Roesen

Post by Keegan Holley
They are saying that the new 16G RE's can handle 250M routes. How is

this

Post by Keegan Holley
possible if none of the daemons are 64bit?

Multiple logical-system instances (== multiple rpd processes)? :-)

The people for whom a 16GB RE is genuinely useful right this second are
few, they pretty much know who they are.

I will say that having worked at a big security vendor even with 32 bit
userspace a 64 bit kernel buys you dramatically more headroom to
scale-userspace on the box, and the extra ram is basically the cheapest
part of the whole exercise.

Post by Keegan Holley
--
CLUE-RIPE -- Jabber: dr at cluenet.de -- dr at IRCnet -- PGP: 0xA85C8AA0
_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Hannes Viertel

2011-08-25 06:47:41 UTC

Permalink

Hi Keegan,

i cannot confirm the 250M routes. What has been tested are 21m bgp v4 Prefixes in a simple "load it up" test.

Basically the imminent scaling improvements on this type of RE come from the change of the memory maps ( kernel can grow beyond 1 gb, and process memory is now upto 3 gb with a maximum deamon size of 2gb ).

hope that helps

/hannes

Post by Hannes Viertel
As Robert pointed out, 32/64 bit use the same codebase and at the current point in time only the kernel is enabled to handle the additional memory. Therefore some of the memory maps/footprints changed slightly. No other daemons have moved to 64bit.
They are saying that the new 16G RE's can handle 250M routes. How is this possible if none of the daemons are 64bit?

_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
------ Ende der weitergeleiteten Nachricht

_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Robert Raszuk

2011-08-25 09:28:23 UTC

Permalink

Hi Keegan,

Post by Keegan Holley
They are saying that the new 16G RE's can handle 250M routes. How is this
possible if none of the daemons are 64bit?

The only real practical scaling use case I am aware of in the range of
5M routes today is for vpnv4 route reflectors.

Another possible scaling point would be perhaps in poorly implemented IX
route server functionality where you would need to copy entire table to
each RS peer in order to execute per peer policy. So indeed if you have
500K of v4 routes and 500 peers it would indeed result in 250M prefixes
to be handled.

Other then that just from routing point of view I am not sure what's the
practical use of such RE or 250M of routes on a real router. I think
control plane can scale and 64bit routing stacks migration is already in
progress (or even completed and shipping few years back on some other
platforms), but forwarding I am afraid is far from that range. So you
are left there using either simple-va like approach, come back to old
good caching or worse process switching on the RE ;-)

Cheers,
R.