I have couple of those never ending cases with support and NW 8.0.x.  From time to time I get those "you should go to 8.1.1.x" as best advice coming from engineering.  I don't buy that and usually end up in discussion with support person wasting my time (or he/she does so arguing with 1st line support person from vendor side). Let me repeat what I have said on several occasions: difference between NW7 and NW8 is huge; difference between NW8 and NW8.1 is even bigger.  Don't get fooled by numbers.  And don't go for latest and greatest as price to pay might be too high if you find stability to be precious.  Some issues are no big deal, but when they happen so obvious you start to question Q&A product goes through.  This is one such case.

 

As per roadmap we have in 2014, in Q2 we started with update of small servers to 8.1.1.4.  In past I would never go to new NW version before SP3 or SP4 with subpatch 4, but things have changes and I no longer can use old approach as NetWorker release management has changed too.  After updating to 8.1.1.4 my daemon.raw on server was flooded with following messages:

 

nsrexecd RAP critical csp configuration service not initialized

 

I could not find much on workflow here, but CSP stands here for Cryptographic Service Provider.  In case of EMC that is of course their own RSA.

 

Stuff used by NetWorker can be found in cst folders which usually is:

  • /opt/nsr/cst for NetWorker (UNIX/Linux)
  • /opt/lgtonmc/cst for NMC (UNIX/Linux)

 

What is CST?  It stands for Common Security Toolkit and it is in NetWorker since 7.5 release (NMC and lockbox at least). Now, given that I do not use NMC and that source reporting problem is nsrexecd, my first conclusion is that this is related somewhat to /opt/nsr/cst.  I see literally daemon.raw flooded with these messages whenever NSR runs some call or I try to run command on my own. Still, I do not see any impact on normal operations (note that I do not use lockbox feature so I can't be really 100% sure).  Anyway, back to /opt/nsr/cst - is there any different between the one before and after upgrade (I did upgrade from 8.0.3.1 to 8.1.1.4).  Let's check:

 

8.1.1.48.0.3.1

  20 -rwxr-xr-x root     root        17176 Apr 02 18:06 Config-Template.xml

  16 -rw-r--r-- root     root        13653 Mar 06 00:01 Config.xml.1

  16 -rwxr-xr-x root     root        13653 Apr 02 18:06 Config.xml.template

   8 -rwxr-xr-x root     root         7442 Apr 02 18:06 CSP-Admin.xml

   4 -rwxr-xr-x root     root         2156 Apr 02 18:06 csp-app.xml

   8 -rwxr-xr-x root     root         4157 Apr 02 18:06 CSP-Authn.xml

   4 -rwxr-xr-x root     root         1070 Apr 02 18:06 CSP-CatalogPosix.xml

   4 -rwxr-xr-x root     root          689 Apr 02 18:06 CSP-CatalogService.xml

   4 -rwxr-xr-x root     root         1082 Apr 02 18:06 CSP-CatalogWindows.xml

   4 -rwxr-xr-x root     root         2175 Apr 02 18:06 CSP-CertificateManagement.xml

  20 -rw-r--r-- root     root        19002 Feb 27 2013 csp.clb

   4 -rw-r--r-- root     root         2378 Dec 18 2011 csp.clb.bak

   4 -rw-r--r-- root     root            2 Dec 18 2011 csp.clb.bak.FCD

   4 -rw-r--r-- root     root           76 Feb 27 2013 csp.clb.FCD

   4 -rwxr-xr-x root     root          863 Apr 02 18:06 CSP-Config.xml

   4 -rwxr-xr-x root     root          668 Apr 02 18:06 CSP-DestinationAlert.xml

   4 -rwxr-xr-x root     root         1439 Apr 02 18:06 CSP-DestinationEventLog.xml

   4 -rwxr-xr-x root     root         1827 Apr 02 18:06 CSP-DestinationFile.xml

   4 -rwxr-xr-x root     root         1376 Apr 02 18:06 CSP-DestinationSyslog.xml

   4 -rwxr-xr-x root     root          575 Apr 02 18:06 CSP-LayoutFlat.xml

   4 -rwxr-xr-x root     root          569 Apr 02 18:06 CSP-LayoutRaw.xml

   4 -rwxr-xr-x root     root          569 Apr 02 18:06 CSP-LayoutXML.xml

   8 -rwxr-xr-x root     root         5902 Apr 02 18:06 CSP-LDAP.xml

   4 -rwxr-xr-x root     root         2966 Apr 02 18:06 CSP-LocalDirectory.xml

   4 -rwxr-xr-x root     root          642 Apr 02 18:06 CSP-LocalizationService.xml

   4 -rwxr-xr-x root     root         3376 Apr 02 18:06 CSP-Logging.xml

   4 -rwxr-xr-x root     root          882 Apr 02 18:06 CSP-OSLogin.xml

   4 -rwxr-xr-x root     root          996 Apr 02 18:06 CSP-PAM.xml

   4 -rwxr-xr-x root     root          632 Apr 02 18:06 CSP-PositionalFormatter.xml

   4 -rwxr-xr-x root     root         2433 Apr 02 18:06 CSP-RoleManagement.xml

  12 -rwxr-xr-x root     root         8496 Apr 02 18:06 CSP-saml-schema-assertion-1.1.xsd

  16 -rwxr-xr-x root     root        13061 Apr 02 18:06 CSP-saml-schema-assertion-2.0.xsd

   4 -rwxr-xr-x root     root         2564 Apr 02 18:06 CSP-SecurID.xml

   4 -rwxr-xr-x root     root          420 Apr 02 18:06 CSP-SM.xml

   4 -rwxr-xr-x root     root          614 Apr 02 18:06 CSP-SprintfFormatter.xml

   4 -rwxr-xr-x root     root          742 Apr 02 18:06 CSP-TrustedUser.xml

   4 -rwxr-xr-x root     root         1474 Apr 02 18:06 CSP-UserManagement.xml

   8 -rwxr-xr-x root     root         5182 Apr 02 18:06 CSP-xenc-schema.xsd

   8 -rwxr-xr-x root     root         4185 Apr 02 18:06 CSP.xml

   0 -rw-r--r-- root     root            0 Apr 16 09:50 upgrade_cst.tag

  12 -rwxr-xr-x root     root        10611 Apr 02 18:06 xmldsig-core-schema.xsd

20 -rwxr-xr-x root     root        17176 Jan 04 03:38 Config-Template.xml

  16 -rwxr-xr-x root     root        13653 Jan 04 03:38 Config.xml

  16 -rw-r--r-- root     root        13653 Mar 06 00:01 Config.xml.1

   8 -rwxr-xr-x root     root         7442 Jan 04 03:38 CSP-Admin.xml

   4 -rwxr-xr-x root     root         2156 Jan 04 03:38 csp-app.xml

   8 -rwxr-xr-x root     root         4157 Jan 04 03:38 CSP-Authn.xml

   4 -rwxr-xr-x root     root         1070 Jan 04 03:38 CSP-CatalogPosix.xml

   4 -rwxr-xr-x root     root          689 Jan 04 03:38 CSP-CatalogService.xml

   4 -rwxr-xr-x root     root         1082 Jan 04 03:38 CSP-CatalogWindows.xml

   4 -rwxr-xr-x root     root         2175 Jan 04 03:38 CSP-CertificateManagement.xml*

  20 -rw-r--r-- root     root        19002 Feb 27 2013 csp.clb

   4 -rw-r--r-- root     root         2378 Dec 18 2011 csp.clb.bak

   4 -rw-r--r-- root     root            2 Dec 18 2011 csp.clb.bak.FCD

   4 -rw-r--r-- root     root           76 Feb 27 2013 csp.clb.FCD

   4 -rwxr-xr-x root     root          863 Jan 04 03:38 CSP-Config.xml

   4 -rwxr-xr-x root     root          668 Jan 04 03:38 CSP-DestinationAlert.xml

   4 -rwxr-xr-x root     root         1439 Jan 04 03:38 CSP-DestinationEventLog.xml

   4 -rwxr-xr-x root     root         1827 Jan 04 03:38 CSP-DestinationFile.xml

   4 -rwxr-xr-x root     root         1376 Jan 04 03:38 CSP-DestinationSyslog.xml

   4 -rwxr-xr-x root     root          575 Jan 04 03:38 CSP-LayoutFlat.xml

   4 -rwxr-xr-x root     root          569 Jan 04 03:38 CSP-LayoutRaw.xml

   4 -rwxr-xr-x root     root          569 Jan 04 03:38 CSP-LayoutXML.xml

   8 -rwxr-xr-x root     root         5902 Jan 04 03:38 CSP-LDAP.xml

   4 -rwxr-xr-x root     root         2966 Jan 04 03:38 CSP-LocalDirectory.xml

   4 -rwxr-xr-x root     root          642 Jan 04 03:38 CSP-LocalizationService.xml

   4 -rwxr-xr-x root     root         3376 Jan 04 03:38 CSP-Logging.xml

   4 -rwxr-xr-x root     root          882 Jan 04 03:38 CSP-OSLogin.xml

   4 -rwxr-xr-x root     root          996 Jan 04 03:38 CSP-PAM.xml

   4 -rwxr-xr-x root     root          632 Jan 04 03:38 CSP-PositionalFormatter.xml

   4 -rwxr-xr-x root     root         2433 Jan 04 03:38 CSP-RoleManagement.xml

  12 -rwxr-xr-x root     root         8496 Jan 04 03:38 CSP-saml-schema-assertion-1.1.xsd

  16 -rwxr-xr-x root     root        13061 Jan 04 03:38 CSP-saml-schema-assertion-2.0.xsd

   4 -rwxr-xr-x root     root         2564 Jan 04 03:38 CSP-SecurID.xml

   4 -rwxr-xr-x root     root          420 Jan 04 03:38 CSP-SM.xml

   4 -rwxr-xr-x root     root          614 Jan 04 03:38 CSP-SprintfFormatter.xml

   4 -rwxr-xr-x root     root          742 Jan 04 03:38 CSP-TrustedUser.xml

   4 -rwxr-xr-x root     root         1474 Jan 04 03:38 CSP-UserManagement.xml

   8 -rwxr-xr-x root     root         5182 Jan 04 03:38 CSP-xenc-schema.xsd

   8 -rwxr-xr-x root     root         4185 Jan 04 03:38 CSP.xml

   0 -rw-r--r-- root     root            0 Jan 18 10:31 upgrade_cst.tag

  12 -rwxr-xr-x root     root        10611 Jan 04 03:38 xmldsig-core-schema.xsd

 

Do you see the difference?  Except timestamps of course.  If you look first and last line, not really, but look more closely. Yes, lines 2 and 3 are different.  More important, the file which seems to be missing is the key here - Config.xml.  You may think, based on above table, that Config.xml has somehow become Config.xml.template, but let me assure you this is not the case - here is the output after I restored missing file on separate (different) system:

 

blog027.JPG.jpg

 

As you can see above, Config.xml has different size than other files.  I admit I spent 0 time figuring out this files - perhaps I should have, but as hinted above - restoring this file does the trick.  Here is live log from the system based on which above table was made:

 

[root@hcrvelin ~]# cd /opt/nsr/cst/

[root@hcrvelin cst]# recover

Current working directory is /opt/nsr/cst/

recover> ll

total 264

  20 -rwxr-xr-x root     root        17176 Apr 02 18:06 Config-Template.xml*

  16 -rw-r--r-- root     root        13653 Mar 06 00:01 Config.xml.1

  16 -rwxr-xr-x root     root        13653 Apr 02 18:06 Config.xml.template*

   8 -rwxr-xr-x root     root         7442 Apr 02 18:06 CSP-Admin.xml*

   4 -rwxr-xr-x root     root         2156 Apr 02 18:06 csp-app.xml*

   8 -rwxr-xr-x root     root         4157 Apr 02 18:06 CSP-Authn.xml*

   4 -rwxr-xr-x root     root         1070 Apr 02 18:06 CSP-CatalogPosix.xml*

   4 -rwxr-xr-x root     root          689 Apr 02 18:06 CSP-CatalogService.xml*

   4 -rwxr-xr-x root     root         1082 Apr 02 18:06 CSP-CatalogWindows.xml*

   4 -rwxr-xr-x root     root         2175 Apr 02 18:06 CSP-CertificateManagement.xml*

  20 -rw-r--r-- root     root        19002 Feb 27 2013 csp.clb

   4 -rw-r--r-- root     root         2378 Dec 18 2011 csp.clb.bak

   4 -rw-r--r-- root     root            2 Dec 18 2011 csp.clb.bak.FCD

   4 -rw-r--r-- root     root           76 Feb 27 2013 csp.clb.FCD

   4 -rwxr-xr-x root     root          863 Apr 02 18:06 CSP-Config.xml*

   4 -rwxr-xr-x root     root          668 Apr 02 18:06 CSP-DestinationAlert.xml*

   4 -rwxr-xr-x root     root         1439 Apr 02 18:06 CSP-DestinationEventLog.xml*

   4 -rwxr-xr-x root     root         1827 Apr 02 18:06 CSP-DestinationFile.xml*

   4 -rwxr-xr-x root     root         1376 Apr 02 18:06 CSP-DestinationSyslog.xml*

   4 -rwxr-xr-x root     root          575 Apr 02 18:06 CSP-LayoutFlat.xml*

   4 -rwxr-xr-x root     root          569 Apr 02 18:06 CSP-LayoutRaw.xml*

   4 -rwxr-xr-x root     root          569 Apr 02 18:06 CSP-LayoutXML.xml*

   8 -rwxr-xr-x root     root         5902 Apr 02 18:06 CSP-LDAP.xml*

   4 -rwxr-xr-x root     root         2966 Apr 02 18:06 CSP-LocalDirectory.xml*

   4 -rwxr-xr-x root     root          642 Apr 02 18:06 CSP-LocalizationService.xml*

   4 -rwxr-xr-x root     root         3376 Apr 02 18:06 CSP-Logging.xml*

   4 -rwxr-xr-x root     root          882 Apr 02 18:06 CSP-OSLogin.xml*

   4 -rwxr-xr-x root     root          996 Apr 02 18:06 CSP-PAM.xml*

   4 -rwxr-xr-x root     root          632 Apr 02 18:06 CSP-PositionalFormatter.xml*

   4 -rwxr-xr-x root     root         2433 Apr 02 18:06 CSP-RoleManagement.xml*

  12 -rwxr-xr-x root     root         8496 Apr 02 18:06 CSP-saml-schema-assertion-1.1.xsd*

  16 -rwxr-xr-x root     root        13061 Apr 02 18:06 CSP-saml-schema-assertion-2.0.xsd*

   4 -rwxr-xr-x root     root         2564 Apr 02 18:06 CSP-SecurID.xml*

   4 -rwxr-xr-x root     root          420 Apr 02 18:06 CSP-SM.xml*

   4 -rwxr-xr-x root     root          614 Apr 02 18:06 CSP-SprintfFormatter.xml*

   4 -rwxr-xr-x root     root          742 Apr 02 18:06 CSP-TrustedUser.xml*

   4 -rwxr-xr-x root     root         1474 Apr 02 18:06 CSP-UserManagement.xml*

   8 -rwxr-xr-x root     root         5182 Apr 02 18:06 CSP-xenc-schema.xsd*

   8 -rwxr-xr-x root     root         4185 Apr 02 18:06 CSP.xml*

   0 -rw-r--r-- root     root            0 Apr 16 09:50 upgrade_cst.tag

  12 -rwxr-xr-x root     root        10611 Apr 02 18:06 xmldsig-core-schema.xsd*

recover> cd ..

recover> versions cst

 

Versions of `/opt/nsr/cst/':

 

   4 drwxr-xr-x root     root         4096 Apr 16 09:49 cst/

     save time:  Mon 21 Apr 2014 02:00:06 AM CEST                

      location:  dalaL.004 at DD_dataL004

 

   4 drwxr-xr-x root     root         4096 Apr 16 09:49 cst/

     save time:  Thu 17 Apr 2014 02:00:08 AM CEST                

      location:  dalaL.004 at DD_dataL004

 

   4 drwxr-xr-x root     root         4096 Jan 18 10:31 cst/

     save time:  Mon 14 Apr 2014 02:00:11 AM CEST                

      location:  dalaL.003 at DD_dataL003

 

   4 drwxr-xr-x root     root         4096 Jan 18 10:31 cst/

     save time:  Mon 07 Apr 2014 02:00:07 AM CEST                

      location:  dalaL.004 at DD_dataL004

 

recover> changetime Mon 14 Apr 2014 02:00:11 AM CEST

6497:recover: time changed to Mon 14 Apr 2014 02:00:11 AM CEST

recover> cd cst

recover> ll

total 264

  20 -rwxr-xr-x root     root        17176 Jan 04 03:38 Config-Template.xml*

  16 -rwxr-xr-x root     root        13653 Jan 04 03:38 Config.xml*

  16 -rw-r--r-- root     root        13653 Mar 06 00:01 Config.xml.1

   8 -rwxr-xr-x root     root         7442 Jan 04 03:38 CSP-Admin.xml*

   4 -rwxr-xr-x root     root         2156 Jan 04 03:38 csp-app.xml*

   8 -rwxr-xr-x root     root         4157 Jan 04 03:38 CSP-Authn.xml*

   4 -rwxr-xr-x root     root         1070 Jan 04 03:38 CSP-CatalogPosix.xml*

   4 -rwxr-xr-x root     root          689 Jan 04 03:38 CSP-CatalogService.xml*

   4 -rwxr-xr-x root     root         1082 Jan 04 03:38 CSP-CatalogWindows.xml*

   4 -rwxr-xr-x root     root         2175 Jan 04 03:38 CSP-CertificateManagement.xml*

  20 -rw-r--r-- root     root        19002 Feb 27 2013 csp.clb

   4 -rw-r--r-- root     root         2378 Dec 18 2011 csp.clb.bak

   4 -rw-r--r-- root     root            2 Dec 18 2011 csp.clb.bak.FCD

   4 -rw-r--r-- root     root           76 Feb 27 2013 csp.clb.FCD

   4 -rwxr-xr-x root     root          863 Jan 04 03:38 CSP-Config.xml*

   4 -rwxr-xr-x root     root          668 Jan 04 03:38 CSP-DestinationAlert.xml*

   4 -rwxr-xr-x root     root         1439 Jan 04 03:38 CSP-DestinationEventLog.xml*

   4 -rwxr-xr-x root     root         1827 Jan 04 03:38 CSP-DestinationFile.xml*

   4 -rwxr-xr-x root     root         1376 Jan 04 03:38 CSP-DestinationSyslog.xml*

   4 -rwxr-xr-x root     root          575 Jan 04 03:38 CSP-LayoutFlat.xml*

   4 -rwxr-xr-x root     root          569 Jan 04 03:38 CSP-LayoutRaw.xml*

   4 -rwxr-xr-x root     root          569 Jan 04 03:38 CSP-LayoutXML.xml*

   8 -rwxr-xr-x root     root         5902 Jan 04 03:38 CSP-LDAP.xml*

   4 -rwxr-xr-x root     root         2966 Jan 04 03:38 CSP-LocalDirectory.xml*

   4 -rwxr-xr-x root     root          642 Jan 04 03:38 CSP-LocalizationService.xml*

   4 -rwxr-xr-x root     root         3376 Jan 04 03:38 CSP-Logging.xml*

   4 -rwxr-xr-x root     root          882 Jan 04 03:38 CSP-OSLogin.xml*

   4 -rwxr-xr-x root     root          996 Jan 04 03:38 CSP-PAM.xml*

   4 -rwxr-xr-x root     root          632 Jan 04 03:38 CSP-PositionalFormatter.xml*

   4 -rwxr-xr-x root     root         2433 Jan 04 03:38 CSP-RoleManagement.xml*

  12 -rwxr-xr-x root     root         8496 Jan 04 03:38 CSP-saml-schema-assertion-1.1.xsd*

  16 -rwxr-xr-x root     root        13061 Jan 04 03:38 CSP-saml-schema-assertion-2.0.xsd*

   4 -rwxr-xr-x root     root         2564 Jan 04 03:38 CSP-SecurID.xml*

   4 -rwxr-xr-x root     root          420 Jan 04 03:38 CSP-SM.xml*

   4 -rwxr-xr-x root     root          614 Jan 04 03:38 CSP-SprintfFormatter.xml*

   4 -rwxr-xr-x root     root          742 Jan 04 03:38 CSP-TrustedUser.xml*

   4 -rwxr-xr-x root     root         1474 Jan 04 03:38 CSP-UserManagement.xml*

   8 -rwxr-xr-x root     root         5182 Jan 04 03:38 CSP-xenc-schema.xsd*

   8 -rwxr-xr-x root     root         4185 Jan 04 03:38 CSP.xml*

   0 -rw-r--r-- root     root            0 Jan 18 10:31 upgrade_cst.tag

  12 -rwxr-xr-x root     root        10611 Jan 04 03:38 xmldsig-core-schema.xsd*

 

recover> add Config.xml

1 file(s) marked for recovery

recover> recover

Recovering 1 file into its original location

Volumes needed (all on-line):

        dalaL.003 at DD_dataL003

Total estimated disk space needed for recover is 16 KB.

Requesting 1 file(s), this may take a while...

Recover start time: Wed 23 Apr 2014 04:17:39 PM CEST

Requesting 1 recover session(s) from server.

libDDBoost version: major: 2, minor: 6, patch: 2, engineering: 0, build: 410681

Successfully established client DDCL session for recovering save-set ID '21701894'.

./Config.xml

Received 1 file(s) from NSR server `hcrvelin'

Recover completion time: Wed 23 Apr 2014 04:17:41 PM CEST

recover> q

 

Guess what?  As soon as you restore missing file, flooding of log with what it says to be critical messages is - over.

 

I'm confused as to how this patches are tested by folks at EMC.  It takes so little to see this.  Just do upgrade and check daemon.raw. Isn't that first step to be made?  If someone wants to believe they did this and concluded there is no harm - I don't buy it. No article about it, no release notes - no business case neither. And you still want me to use latest and greatest on heavy duty servers?  Dream on - I will stick to my conservative approach and wait a bit more.  Much more (Q4).

 

I would lie if I would say that I was not aware of other cases (and source upgrade versions) being affected by this very same thing - I was. After I hit this problem I started to ask question around and I have heard of couple of same issues observed with some non-sense responses by 1st level support (if same folks are doing Q&A that might explain some things). I'd like to say thanks to JC for all the things I didn't mention here and her support on this issue.