[j-nsp] Configuration database stuck with mgd crashing

[Tore Anderson]

One of my routers (a MX240 running 16.1R6-S2.3) have gotten stuck in a
state where it believes the configuration database has been modified,
and if I try to configure it anyway, mgd crashes and is respawned:

***@router> configure exclusive
error: configuration database modified

***@router> configure private
error: shared configuration database modified

***@router> configure
Entering configuration mode

Message from ***@router at Aug 31 13:38:57 ...
router mgd[20554]: ../../../../../../src/ui/lib/access/model.c:238: insist 'model > 0 && model <= MODEL_MAX' failed

error: session failure: unexpected termination
error: remote side unexpectedly closed connection
Connection to router closed.

At this point PID 20554 goes away from the process list. However if I
log back in I can see a «ghost» reference to it:

router> configure exclusive
Users currently editing the configuration:
tore terminal pts/0 (pid 20554) on since 2018-08-31 13:38:57 CEST, idle 00:01:25
error: configuration database modified

"request system logout user tore all" will get rid of that reference,
but the fundamental defective state of the configuration database
remains.

Any suggestions on how to correct this problem without requiring
any downtime? I have of course tried "restart management", but
that didn't help. NETCONF is impacted too.

Tore


_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.ne

[Luis Balbinot]

As root you can “mgd -I” to fix that. We also had the same issue and it’s
been fixed on 16.1R7. We never opened a JTAC case for that because we knew
the answer would be software upgrade since 16.1R7 was already out. The
cause was Netconf and it randomly occurred.

Strangely this was caused by the service release version. Earlier 16.1R6
images were fine.

Post by Tore Anderson
One of my routers (a MX240 running 16.1R6-S2.3) have gotten stuck in a
state where it believes the configuration database has been modified,
error: configuration database modified
error: shared configuration database modified
Entering configuration mode
router mgd[20554]: ../../../../../../src/ui/lib/access/model.c:238: insist
'model > 0 && model <= MODEL_MAX' failed
error: session failure: unexpected termination
error: remote side unexpectedly closed connection
Connection to router closed.
At this point PID 20554 goes away from the process list. However if I
router> configure exclusive
tore terminal pts/0 (pid 20554) on since 2018-08-31 13:38:57 CEST, idle 00:01:25
error: configuration database modified
"request system logout user tore all" will get rid of that reference,
but the fundamental defective state of the configuration database
remains.
Any suggestions on how to correct this problem without requiring
any downtime? I have of course tried "restart management", but
that didn't help. NETCONF is impacted too.
Tore
_______________________________________________
https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net

[Aaron Gould]

Maybe "commit full"

https://kb.juniper.net/InfoCenter/index?page=content&id=KB15410

- Aaron

-----Original Message-----
From: juniper-nsp [mailto:juniper-nsp-***@puck.nether.net] On Behalf Of Tore Anderson
Sent: Friday, August 31, 2018 6:48 AM
To: juniper-***@puck.nether.net
Subject: [j-nsp] Configuration database stuck with mgd crashing

One of my routers (a MX240 running 16.1R6-S2.3) have gotten stuck in a
state where it believes the configuration database has been modified,
and if I try to configure it anyway, mgd crashes and is respawned:

***@router> configure exclusive
error: configuration database modified

***@router> configure private
error: shared configuration database modified

***@router> configure
Entering configuration mode

Message from ***@router at Aug 31 13:38:57 ...
router mgd[20554]: ../../../../../../src/ui/lib/access/model.c:238: insist 'model > 0 && model <= MODEL_MAX' failed

error: session failure: unexpected termination
error: remote side unexpectedly closed connection
Connection to router closed.

At this point PID 20554 goes away from the process list. However if I
log back in I can see a «ghost» reference to it:

router> configure exclusive
Users currently editing the configuration:
tore terminal pts/0 (pid 20554) on since 2018-08-31 13:38:57 CEST, idle 00:01:25
error: configuration database modified

"request system logout user tore all" will get rid of that reference,
but the fundamental defective state of the configuration database
remains.

Any suggestions on how to correct this problem without requiring
any downtime? I have of course tried "restart management", but
that didn't help. NETCONF is impacted too.

Tore


_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/l

[Tore Anderson]

* Aaron Gould

Post by Aaron Gould
Maybe "commit full"

Thank you for the suggestion! I was however unable to get into configure
mode in the first place, so I couldn't issue any kind of "commit".

Luis's suggestion of «mgd -I» from a root shell did the trick, though.

Tore
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinf

[Phil Shafer]

Post by Aaron Gould
Maybe "commit full"

"commit full" helps when daemons miss config changes (which they
shouldn't) or if you just want to say "because I said so", but it
needs a functioning database, provided by MGD. In this case, MGD
has corrupted the database (due to a software bug) and the assert
means that it's unable to do anything useful with the database since
it's corrupted and cannot be trusted. "mgd -I" is the "nuke the
entire site from orbit" option. It rebuilds the schema and the
database from scratch and reloads the entire contents. It's the
only way to be sure.

Thanks,
Phil
_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[Sebastian Wiesinger]

Post by Phil Shafer
"commit full" helps when daemons miss config changes (which they
shouldn't) or if you just want to say "because I said so", but it
needs a functioning database, provided by MGD. In this case, MGD
has corrupted the database (due to a software bug) and the assert
means that it's unable to do anything useful with the database since
it's corrupted and cannot be trusted. "mgd -I" is the "nuke the
entire site from orbit" option. It rebuilds the schema and the
database from scratch and reloads the entire contents. It's the
only way to be sure.

What operational impact does mgd -I have?

Regards

Sebastian

[Luis Balbinot]

Mini heart attacks :-)

Now seriously, I’ve seen none so far.

Post by Sebastian Wiesinger

Post by Phil Shafer
"commit full" helps when daemons miss config changes (which they
shouldn't) or if you just want to say "because I said so", but it
needs a functioning database, provided by MGD. In this case, MGD
has corrupted the database (due to a software bug) and the assert
means that it's unable to do anything useful with the database since
it's corrupted and cannot be trusted. "mgd -I" is the "nuke the
entire site from orbit" option. It rebuilds the schema and the
database from scratch and reloads the entire contents. It's the
only way to be sure.

What operational impact does mgd -I have?
Regards
Sebastian
--
GPG Key: 0x93A0B9CE (F4F6 B1A3 866B 26E9 450A 9D82 58A2 D94A 93A0 B9CE)
'Are you Death?' ... IT'S THE SCYTHE, ISN'T IT? PEOPLE ALWAYS NOTICE THE SCYTHE.
-- Terry Pratchett, The Fifth Elephant
_______________________________________________
https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
juniper-nsp mailing list juniper-***@puck.nether.net
https://puck.nether.net/mailma