I am a bit behind so I'm catching up here a bit with missed stuff.  Those familiar with NetWorker know that each NetWorker code tree gets patch released one month apart (approximately).  Currently we have following code trees:

  • 8.1SP3 with latest patch being 8.1.3.10 (build 611)
  • 8.2SP3 with latest patch being 8.2.3.6 (build 1159)
  • 9.0 with latest patches being 9.0.0.8 (build 524)
  • 9.0SP1 with latest patches being 9.0.1.2 (build 645)

 

I'm going to list those patches I missed since last blog post (summer+).  Where applicable, I will list associated modules as well (that includes NMSAP and NMDA as new practice since NW9SP1).

 

Patches for 8.1.3.X since last blog post are:

Ticket/Escalation/ID8.1.3.9 - Description
259579/26339/nsrjobd getting core dump on Linux x64 platform with NW 8.2.2.x version
259056//Fix the regression caused by 247484
257773/26176/Stale nsrmmd left behind after NetWorker Service stops, Customer needs to kill manually "stale nsrmmd" cause more downtime window for large datazone service recycle.  If NetWorker nsrexecd Service stops and restarts immediately the nsrmmd process is not killed (it require more time after nsrexecd is stopped to go down gracefully).
257305/26031/Multiple nsrmmds created for same tape devices, causing contention and failures. Failures occur due contention in devices’ activities.
257079/25977/Virtual Synthetic Full - nsrrecopy is ignoring the server network interface setting.  Server Network Interface has been configured for client hosts with reference to the backup network. However, when the nsrconsolidate operation runs, it ignores the Server Network Interface setting.
256012//liblocal function "lgwstat" can fail for locked files On Windows
255113/23751/

OS buffering of tape writes may result in out of order writes with writing filemarks with CDI.  Error:

Duplicate volume name <volume_name>. Select a new name or remove the original volume.

254947//NW savegrp fails to handle add job due to case sensitive comparison for client name
254924/25330/Scheduled cloning job sometimes hangs and does not select other volumes on DD devices.  Clone jobs intermittently stall and wait until a recyclable volume is manually relabeled. The subsequent writes for the same clone job do not require manual intervention.
254786/25721/Duplicated session info records created for session updates with rm_jobid of 0.  Stale sessions for backups are shown in NetWorker Management Console.
253647//Save - SAP Oracle CPR backup always restarts from the first failed backup, not the last failed backup if restarted more than once
253392/25661/nsrmmd core dump.

The following error message is logged while the mentioned SSID does exist in media database:

nsrmmd NSR notice Stat of the saveset file for ssid <SSID> failed during precommit check with error 'The operation cannot proceed since the connection table to Data Domain server <NULL> is missing.

252810/24416/Scheduled NDMP cloning loops for 18 hours and hangs the system even though both source/destination clients are available.
250774//Change the locate.mk: Provide a version number to libpal for packaging purposes
249002/24105/Support the extended attributes for Solaris
247146/25063/nsrim does not recycle tape volumes with incomplete and suspect savesets. The suspect or incomplete savesets are not removed from volume even after nsrim -V <volume_name> is run. Tape volumes will not become recyclable. These savesets need to be manually deleted using nsrmm in order for the volumes to be recycled.
245187/23025/Cloning from DD to LTO volume intermittently hung. Error : Media Notice: Volume `xxx' ineligible for this operation; Need a different volume from pool `xxx'
239119//NW server losses ldap configuration if jukebox is setup
257921/26048/NMC does not display correct information regarding space utilization for DD.
256795/25883/Multiple GUI Defects. Some options in NetWorker Management Console menus are greyed out. Inconsistent behavior in the contents context menus in NMC.
255077//NMC UI: Unselecting table columns in any page causes error msg to be displayed in NMC Management Console homepage
246854//Help Window is not launched when clicked on "?" in "Perform the Recovery" page of Recovery wizard
Ticket/Escalation/ID8.1.3.10 - Description
265228/26313/PSS recovery performance is very low when executed from server side (NMC) in contrast to client side recovery (Winworker)
264343/26181/Scanner dumps core for NFS mount AFTD on Solaris Sparc
263111/26736/Virtual synthetic error top 8.2.3.2 with server network interface
262441/25854/AFTD operation fails when NFS path used for device access
262006//CID 50958: Resource leak: reported by coverity in "nsr_dataprotectionpolicy.c"
261723/26495/Recurrent nsrd core dump
261190/26426/Networker ignores 'Recovery Storage Node' field and tries to read via wrong node
250070/25234/'Data Domain Backup' value is applied to Clone target, causing 'no matching IP interface data domain devices for save of client' error
239310/23943/unknown operation messages logged in daemon.raw following nsrim -X running
236868/23893/If the PreCmd script fails, NetWorker runs the script for each SaveSet
235643/23835/bootstrap ssids remain on aftd beyond clretent
195797//NW00160531Savegrp Message "savegroup notice: SKIP completed, Total 4 client(s), 4 Succeeded, Cloning failed" need to be modified
261167//Client Creation fails even if remote access updated with valid attributes " *@* "

 

To download 8.1.3.x binaris go to ftp://ftp.legato.com/pub/NetWorker/Cumulative_Hotfixes/8.1

 

For 8.2.3.x tree below is the list of patches for 8.2.3.2 until 8.2.3.6 which is latest one at the time of this writing.

 

Ticket/Escalation/ID8.2.3.2 - Description
259496//Disable AMS cloning (large save set slicing) by default
257773/26176/Stale nsrmmd left behind after NetWorker Service stops, Customer needs to kill manually "stale nsrmmd" cause more downtime window for large datazone service recycle
257305/26031/Multiple nsrmmds created for same tape devices, causing contention and failures
257179/26143/NW 8.2.3 nsrwatch dumps core on HP-UX ia64.
257079/25977/Virtual Synthetic Full - nsrrecopy is ignoring the server network interface setting.
257053//Sqlite media database feature is not enabled on all HP platform
256975//(REOPENED) nsrmmdbd leaks memory in add_ss_vol_deletion()
256873//nsrmmdbd leaks memory in svcmmdb_update_ss6_6()
256752//Reduce the amount of memory reserved for getpwnam() , getpwuid() and getgrgid() liblocal_t_varp buffers
256576//nsrmmdbd leaks memory in priv_ngc_fetchss_range()
256333/24826/NW: Backup renamed directories field is disabled when creating a client using 'nsradmin -i' command
256096//nsrwatch Does not display VMware Protection Policies
256012//liblocal function "lgwstat" can fail for locked files On Windows
255113/23751/(REOPENED) OS buffering of tape writes may result in out of order writes with writing filemarks with CDI
255058/25750/Networker server is hanging and most services and commands become unusable
254716//AutoSAT: 'nsr_render_log -H' command does not work
253392/25661/nsrmmd core dump
253035//Policy creation issues after removing "Vmware actions" from an existing policy
257921/26048/NMC does not display correct information regarding space utilization for DD
256795/25883/Multiple GUI Defects
256328//NMC UI: Java NullPointerException is thrown when User click on Group Properties in VMWare view
255077//NMC UI: Unselecting table columns in any page causes error msg to be displayed in NMC Management Console homepage
252014//NMC hangs while browsing large saveset in Client Properties.
246854//Help Window is not launched when clicked on "?" in "Perform the Recovery" page of Recovery wizard
Ticket/Escalation/ID8.2.3.3 - Description
261282/26472/NDMP backup is failing when the it tries to continue to second tape. NDMP backups of Netapp clusters to tape devices attached to another Netapp cluster fail when a continuation volume is required to continue the backup. After a continuation tape is loaded and mounted in the tape device the following messages are seen in the savegrp debug output or workflow debug logs:

42619:nsrndmp_save: NDMP Service Error: DUMP: Write to socket failed

42619:nsrndmp_save: NDMP Service Error: DUMP: DUMP IS ABORTED

 

 

From the NetWorker server Application event log:

Error Faulting application nsrndmp_save.exe, version 8.2.2.1, time stamp 0x561e79e6, faulting module nsrndmp_save.exe, version 8.2.2.1, time stamp 0x561e79e6, exception code 0xc0000005, fault offset 0x0000000000006441, process id 0x1790, application start time 0x01d1af7273e48075.

 

 

 

The issue is due to nsrndmp_save.exe crashing on the NetWorker server when a continuation tape is required to continue the NDMP backup.          

261190/26426/(REOPENED) Networker ignores 'Recovery Storage Node' field and tries to read via wrong node. NetWorker ignores 2nd entry and beyond in 'Recovery Storage Node' list and may attempt to use any node. If the node NetWorker attempts to use cannot reach the recovery destination client (due to network / firewall restrictions), the recovery stalls and does not progress - with no error messages.
261005//nsrvba_save crashes in Policy_Ref::process_jobs_tree()
260489/26185/Since upgrading to v8.2.3.x, cloned savesets are changed to 'eligible for recycling' after nsrim runs. Tape cloned save sets change to 'undef' and become expired after writing. Daemon log shows messages like:

 

nsrmmdbd NSR notice Adjusting clone 4097201600/1463322246 retention (1497563839) based on attribute data (1463321925:0)

 

 

Note the ":0" at the end. Any other number is normal. Issue only affects clone copies on Tape or FTD, clones on Data Domain or AFTD are OK.

260310/26296/savegrp is spending 2 hours in mmdb lookup before starting the backup command with NMM SQL VDI clients. When save group starts, it spends hours before anything happens for groups containing SQL VDI clients. The first incremental backup after a full goes well but subsequent attempts progressively get slower although the amount of data is small. When savegrp for SQL starts and before the nsrsqlsv command is requested from the client, it takes an enormous amount of time for savegrp/nsrjobd to do lookups in the Media DB.
260207//MMDB_ADD_FRAGMENT RPC does not always update the *ss clone retention attribute in mdb_saveset table
259962/26325/Since upgrade to networker 8.2.3.1, networker cloning exceeding max defined sessions and causing backups to fail. Clone jobs use more cloning sessions than the defined maximum sessions.
259767//nsrhang can incorrectly match nsrdispd's PID instead of nsrd
259537/26197/

Clones stall after a few hours (CPU reduces to 0%). At this time pending backups (typically index sessions) stall. There is no waiting for media errors, just devices 'writing, idle'. As soon as the clones are stopped the backup sessions resume. The daemon log has thousands of messages like:

 

 

ansrd NSR notice Unable to find save-set for host <host_name> and save time '1461739445'.

259355/26099/Networker 8.2.3.0 nsrclone core dump in libDDBoost.dll. The nsrmmdbd process crashes, preventing backups to complete. The nsrclone process dumps core.
259056//Fix the regression caused by 247484.
259029/26235/nsrwatch not showing correct save size information. The size is displayed in EB instead of GB.
257859//nsrsnmd fails nsrrpcinfo -t tests
257531/25662/nsrexecd core dump after upgrade to 8.2.2.4 client on AIX 7.1 CST/RSA.  nsrexecd cores after updating to NetWorker 8.2.2.4. nsrexecd cores immediately at start up. The problem appears to impact newer version of AIX 7.1. The problem is not present on AIX 7.1 TL1, but shows in AIX 7.1 TL4.
255113/23751/OS buffering of tape writes may result in out of order writes with writing filemarks with CDI. Error:

Duplicate volume name <volume_name>. Select a new name or remove the original volume.

252745/25530/Networker server (nsrexecd) mutex hangs in msg_post(). The nsrd becomes unresponsive every few days. All commands (nsrwatch, nsradmin) stop responding. NetWorker services need to be recycled to provide relief.
250533//multiple nsrclone MLKs with DD devices
249002/24105/Support the extended attributes for Solaris
248895//Add option to force IPv4 for index based recovery
256795/25883/Multiple GUI Defects. Some options in NetWorker Management Console menus are greyed out. Inconsistent behavior in the contents context menus in NMC.
250223/25284/NMC gui not able to add vm to policy VBA there are no polices available. The VMware view in NMC does not reports any VM-to-Policy assignment. Trying to add any VM to any Policy a popup immediately appears with "There are no Policies available" message.
249905//[regress] Client recreating with shortname
Ticket/Escalation/ID8.2.3.4 - Description
262514//nsrwatch fails to unmount file type devices
262074/26469/Automatic staging fails - 8.2.3.1 - ERROR: Failed to build/verify following ssid(s) to continue cloning due to '?'
261945/25776/Scheduled clone shows success in NMC and jobsdb despite aborted clone with 'some of the clones failed' message in logs
261723/26495/Recurrent nsrd core dump
261708/26107/(REOPENED!) ngc code for mmdb barcode queries is incomplete
261417//nsrwatch doesn't clear residue from previous display text when new text is shorter
260598//nsrd could try to make a connection to nsrmmdbd indefinitely after it's crashed preventing nsrd from restarting nsrmmdbd
259414//nsrwatch uses excessive CPU when groups info window is open
254924/25330/Scheduled cloning job sometimes hangs and does not select other volumes on DD devices
Ticket/Escalation/ID8.2.3.5 - Description
263910//sysdesc.c:get_total_mb_used() erroneously calls donefstab(), prematurely freeing the global cached file system table
263629/26644/nsrmmdbd clients can erroneously create a connection without a sufficiently high RPC authentication flavor
263357//nsrwatch : policy "get info" for a successful job showing incorrect size for backup data
263112/26743/xcessive mmdb_xxxx messages in logs. Example:

nsrmmdbd NSR warning Request 2892715 (update volume '<volumename>') ended with error: volume <volumename> version mismatch

 

 

These messages are superficial and do not indicate a problem.

263111/26736/Virtual synthetic error top 8.2.3.2 with server network interface
262880//Nsrwatch: linux: extra characters shown on unmounting and mounting of an AFTD device
262800//nsrwatch: not showing correct status of a running group on Unix platforms
262651//A lot of "database is locked" messages output during "mmrecov"(or nsrmmdbasm -r) - customer issue noted in escalation 26485
262441/25854/AFTD operation fails when NFS path used for device access. Backup to NFS-AFTD causes "Permission denied" error. Ports used by  nsrsnmd are not being released after each AFTD device related operation. Eventually number of available ports get exhausted causing subsequent operation to fail with permission denied error.
261509//Installation on Ubuntu 16.04 fails with libxp6 package not found.
261095/26349/nsrim cores or fails with error as a result of a memory leak when examining exchange SS's
260743/25672/Escalation 26473 - recover.exe and winworkr fail directed restore from/to a client, when executed from a different client. Cannot recover files using NetWorker User GUI. There is no error reported in GUI, the recover just hangs.
259579/26339/nsrjobd getting core dump on Linux x64 platform with NW 8.2.2.x version
Ticket/Escalation/ID8.2.3.6 - Description
265655/26589/networker can not accept host names that have number of characters above 46
265587//lg_snprintf() returns -1 on truncation on some platforms
265228/26313/PSS recovery performance is very low when executed from server side (NMC) in contrast to client side recovery (Winworker)
265205//s_clone3.exp test case is failing in nw_8_2_dev
265171/26629/Fix from ESC 26185 does not revert savesets to not-recyclable permanently
265164//Clean up nsrclone messaging when it fails to build a ssid list from a list of volumes or volids
265075//DPRINTF() output is truncated after 600 character
264427/26824/savepnpc coredump during direct save setup
264343/26181/Scanner dumps core for NFS mount AFTD on Solaris Sparc
263676//CID 13416: Resource leak in nsrcap.c
262006//CID 50958: Resource leak: reported by coverity in "nsr_dataprotectionpolicy.c"
264997//NMC wizard for NSM clients, always resets the value for Mount Host
261753//NMC UI: Java Nullpointer Exception is thrown when User try to add valid Policy-VBA
261718//NMC UI: Java Nullpointer Exception is thrown when User click on 'Reset' button in create vmware policy
261574//NMC UI: Java Nullpointer Exception is thrown when User click on 'Add "Add Policy-VBA" option in VMware Table View
261565//Nemo: VMware View: Pop up with Error message is displayed when tried to edit a policy using map display.
261372//NMC Recover: Saved Saveset Recover entry is not configurable as buttons and left pane details are disabled
261360//NMC UI: Java Nullpointer Exception is thrown when User click on 'Policy Properties' option in VMware View
261167//Client Creation fails even if remote access updated with valid attributes " *@* " .
250824//BBB RecoverUI:Redirected recover of savesets cloned to tape is not working
250446//NMC Console: Error handling of Hostname field and text of Error message needs improvement
249407//NMC 8.2.3 fails to 'fetch' NMM 9.0.1 wizard JAR

 

You can find above packages in ftp://ftp.legato.com/pub/NetWorker/Cumulative_Hotfixes/8.2/

 

Along with NW came NMM as well.  Here are NMM patches:

 

Ticket/Escalation/IDNMM 8.2.3.2 - Description
256846/26051/SQL Incremental backup fails and gets promoted to FULL for a Cross domain Mirroring setup.
255477//AD Scheduled backups are failed due to recent policy changes.
255426/25855/Exchange 2010 DAG savegroup is marked as failed but all logs indicate a good backup
254885/25779/SQL Recovery fails with "Error failed to obtained tape session" error when NMM resolves the hostname to a different canonical name than NW's canonical name (FQDN vs shortname)
252428/25276/AD backup completes successfully in NMC but savegrp logs shows "Could not backup following objects" for some items
246601//NMM Backup fails with savefs error when UAC enabled
Ticket/Escalation/IDNMM 8.2.3.3 - Description
260553/26299/NSR Log Viewer unable to open files after upgrading to 8.2.3
259650/26345/Mailbox Restore to PST excludes 'dumpster' folder
259529/26318/Incorrect retention policy applied to manual striped SQL backups
259098/26248/nsrsqlsv crash during backup. dump analysis required
258308/26005/Mailbox Export to PST fails if user has explicit 'Mailbox Import Export' Role assignment. Works if user is a member of Organization management Role group
257222//[Regress] NMM is crashing when we select required volumes option
Ticket/Escalation/IDNMM 8.2.3.5 - Description
263062/26393/NSRBRAGENT launches, but fails to connect. Exchange Dummy Savesets are created for DAG backups performance
262977/26614/Port request for NMSQL fix of ESC 19987 / BUG 220770 into NMM
262634/26403/NMM 9.0.0.6 Fails GLR/RDB browse due to MAPI popup errors
262348/26447/SQL Backup of some DBs failing after upgrade to 8.2.3.1
Ticket/Escalation/IDNMM 8.2.3.5 - Description
266014//NMM 8.2.3 and above to Support Cloud Boost device with nwfs.exe
265937/26338/nsrsqlsv.exe crash due to null username and password from client properties when SQL AAG Backing up
265233//Unauthenticated RCE issue - Additional fix
264154//[Hyper-V GLR]Mount point and hyper-v VHD mount point location appears to be empty in granular tab resulting in recovery failure

 

Above NMM patches can be found at ftp://ftp.legato.com/pub/NetWorker/NMM/Cumulative_Hotfixes/8.2.3/

 

Finally, we have NetWorker 9 which now exists in its initial form and SP1 - each with some post patches.  Below the list for patches since I wrote about them last time.

 

Ticket/Escalation/ID9.0.0.7 - Description
259506//Running nsrlogout cores nsrexecd
259056//Fix the regression caused by 247484
257773/26176/Stale nsrmmd left behind after NetWorker Service stops, Customer needs to kill manually "stale nsrmmd" cause more downtime window for large datazone service recycle.  If NetWorker nsrexecd Service stops and restarts immediately the nsrmmd process is not killed (it require more time after nsrexecd is stopped to go down gracefully).
257312//REG: NMDA Sybase RUI recovery causes nsrjobd to crash
257079/25977/Virtual Synthetic Full - nsrrecopy is ignoring the server network interface setting. Server Network Interface has been configured for client hosts with reference to the backup network. However, when the nsrconsolidate operation runs, it ignores the Server Network Interface setting.
256975//nsrmmdbd leaks memory in add_ss_vol_deletion()
256873//nsrmmdbd leaks memory in svcmmdb_update_ss6_6()
256658//nsrworkflow:If more than one client with same hostname exist, nsrworkflow backup job fails when -c option set to same client hostname
256640//Probe action shows different status for the same instance of the client
256507/26070/

The nsrmmdbd.exe process is crashing after upgrade:

Faulting application name: nsrmmdbd.exe, version: 8.2.3.0, time stamp: 0x56e7dc0c

Faulting module name: nsrmmdbd.exe, version: 8.2.3.0, time stamp: 0x56e7dc0c

256333/24826/NW: Backup renamed directories field is disabled when creating a client using 'nsradmin -i' command. When creating client with nsradmin -i filename, if the intended backup command includes hyphen (save -c, -D5, etc...), the "Backup renamed directories: Enabled" entry in the input file is ignored. If the backup command does not include hyphen (save or savepnpc) the "Backup renamed directories: Enabled" entry in the input file takes effect.
255700/25533/nsrim still does not delete the expired save set on ALL ddboost devices even after apply HF from Port NW#MASTER Bug 185055. Used space on DD Boost devices\volumes will grow, consuming the capacity of the Data Domain. When the 'nsrim -X' runs as part of its regular NetWorker maintenance, the Space Recovery operation is not purging\deleting the expired save sets from the DD Boost devices\volumes.
255113/23751/

OS buffering of tape writes may result in out of order writes with writing filemarks with CDI.  Error:

Duplicate volume name <volume_name>. Select a new name or remove the original volume.

254924/25330/Scheduled cloning job sometimes hangs and does not select other volumes on DD devices
254169//NW 9.0.0.3 - Policy/Workflow independent start
253392/25661/

The nsrmmd process crashed and dumped core. The following error message is logged while the mentioned SSID does exist in media database:

nsrmmd NSR notice Stat of the saveset file for ssid <SSID> failed during precommit check with error 'The operation cannot proceed since the connection table to Data Domain server <NULL> is missing.'

250774//Change the locate.mk: Provide a version number to libpal for packaging purposes
250389// nsrsnmd MLK
249002/24105/Support the extended attributes for Solaris
247433//nsrd cores under Purify
247146/25063/nsrim does not recycle tape volumes with incomplete and suspect savesets
247067//nsrmmd coredump caused by heap corruption through disconnect_mb and Coverity CID 51518 - use after free in function destroy_rreq in nsr/mm/mm_dmux.c. The issue only occurs if oldauth is disabled.
245161//Pool retention policy should take precedence over client retention policy,when Pool retention policy is > client retention policy
242594//savepsm, save, ... exes failing with assertion (some statically linked Windows exes)
258263/25352/a blank grey window with "OK" button only is received when adding/ editing group after upgrade from 9.0.0.2 to 9.0.0.5
257921/26048/NMC does not display correct information regarding space utilization for DD
257890//unable to restore cloud boost backup files from NMC as "Next" button is disabled in recover window
257778//Null pointer exception - Browse saveset button not loading savesets (In Group tab - Client view)
257493//NPE and empty error message window while accessing the workflow action details
257476//NMC UI: Java NullPointerException is thrown when User tries to Start individual client on a deleted Policy workflow
257303//NMC refresh issues and hangs in diagnostic mode
256525//NMC UI: Java NullPointerException is thrown when User tries to create New Group in VMware View
256328//NMC UI: Java NullPointerException is thrown when User click on Group Properties in VMWare view
256141//NMC UI: Java StackOverflowError exception is thrown when User enable/disable View Parallel menu option in Monitoring Tab
256128//NMC Wizard: Parallelism field value is reset to 12 after error message is displayed for Remote client
255472//Client Creation fails if remote access is updated with valid attributes
255304//Workflow and action should not allow creation with more than 64 character (NMC)
255077//NMC UI: Unselecting table columns in any page causes error msg to be displayed in NMC Management Console homepage
254949//PPRP: Modifying the client config wizard, Next button is disabled while selecting EMC PP for Recoverpoint SCM
254280//Error while associating a group to a client ,whose action is expire
254017//SAP HANA : All saveset name is displayed as "NetWorker Module for SAP (backint)" in NMC during database restore
252472//RUI: Saved Configuration fills in missing details with default settings
252021//VMware view topology doesn't show VM machines of an ESXi server
252014//NMC hangs while browsing large saveset in Client Properties.
251766//"Ctrl-O" keyboard shortcut does not work in "Policies" and is common for both "Properties" and "Copy"
251488//Restart Workflows should have a confirmation dialog window similar to start option
Ticket/Escalation/ID9.0.0.8 - Description
261282/26472/NDMP backup is failing when the it tries to continue to second tape.

NDMP backups of Netapp clusters to tape devices attached to another Netapp cluster fail when a continuation volume is required to continue the backup. After a continuation tape is loaded and mounted in the tape device the following messages are seen in the savegrp debug output or workflow debug logs:

42619:nsrndmp_save: NDMP Service Error: DUMP: Write to socket failed

42619:nsrndmp_save: NDMP Service Error: DUMP: DUMP IS ABORTED

 

 

From the NetWorker server Application event log:

Error Faulting application nsrndmp_save.exe, version 8.2.2.1, time stamp 0x561e79e6, faulting module nsrndmp_save.exe, version 8.2.2.1, time stamp 0x561e79e6, exception code 0xc0000005, fault offset 0x0000000000006441, process id 0x1790, application start time 0x01d1af7273e48075.

 

 

The issue is due to nsrndmp_save.exe crashing on the NetWorker server when a continuation tape is required to continue the NDMP backup.

261190/26426/Networker ignores 'Recovery Storage Node' field and tries to read via wrong node. NetWorker ignores 2nd entry and beyond in 'Recovery Storage Node' list and may attempt to use any node. If the node NetWorker attempts to use cannot reach the recovery destination client (due to network / firewall restrictions), the recovery stalls and does not progress - with no error messages.
261154//Error encountered during the installation of licensing software
260489/26185/Since upgrading to v8.2.3.x, cloned savesets are changed to 'eligible for recycling' after nsrim runs
260207//MMDB_ADD_FRAGMENT RPC does not always update the *ss clone retention attribute in mdb_saveset table
259895//NetWorker service status returns "0" even if the services are not running
257987//Empty "Recommends" line should be removed from the DEB source files
257305/26031/Multiple nsrmmds created for same tape devices, causing contention and failures.

Tape cloned save sets change to 'undef' and become expired after writing. Daemon log shows messages like:

nsrmmdbd NSR notice Adjusting clone 4097201600/1463322246 retention (1497563839) based on attribute data (1463321925:0)

 

 

Note the ":0" at the end. Any other number is normal.

Issue only affects clone copies on Tape or FTD, clones on Data Domain or AFTD are OK

245187/23025/

Cloning from Data Domain to Tape volume intermittently halts waiting for different volume, even though appendable volumes are available. Error:

Media Notice: Volume `xxx' ineligible for this operation; Need a different volume from pool <pool_name>

261681/26479/

Getting the  error: "Failed to connect to database"  when opening a report on the NMC. Daily NMC db backup fails after an initial successful backup. There are a lot of idle postgres processes on the NMC

postgres 32721 23746  0 13:46 ?        00:00:00 postgres: lgtogst lgto_gst ::1(49282) idle

Error from  pg_basebackup log:

pg_basebackup: could not connect to server: FATAL:  sorry, too many clients already

FATAL:  sorry, too many clients already

Error from web_output log:

AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

261268//Stale clone sessions -currently running ,displayed the clone details window after completion of the action
260006//Selected objected in group tab does not reflect the updated client list
256955/26008/When configuring a Policy when the backup action has been configured, during adding a cloning action, the Source Storage Node for the policy is grayed out / greyed out as Read only.
Ticket/Escalation/ID9.0.1.1 - Description
263112/26743/

Excessive mmdb_xxxx messages in logs since upgrade to v8.2.3.x, eg:

nsrmmdbd NSR warning Request 2892715 (update volume '<volumename>') ended with error: volume <volumename> version mismatch

262514//nsrwatch fails to unmount file type devices Public
262441/25854/AFTD operation fails when NFS path used for device access
262173//Add ELS license file verification to nsrlic
261945/25776/Scheduled clone shows success in NMC and jobsdb despite aborted clone with 'some of the clones failed' message in logs
261830//Solving Coverity issues: memory leak, uninitialized variables and a return value check
261580//PSR: Inappropriate error message "DOD INTERNET HOST TABLE SPECIFICATION check failure", if auth host name is missing
261509//Installation on Ubuntu 16.04 fails with libxp6 package not found.
261417//nsrwatch doesn't clear residue from previous display text when new text is shorter
261190/26426/Networker ignores 'Recovery Storage Node' field and tries to read via wrong node. NetWorker ignores 2nd entry and beyond in 'Recovery Storage Node' list and may attempt to use any node. If the node NetWorker attempts to use cannot reach the recovery destination client (due to network / firewall restrictions), the recovery stalls and does not progress - with no error messages.
260743/26473/recover.exe and winworkr fail directed restore from/to a client, when executed from a different client
260598//nsrd could try to make a connection to nsrmmdbd indefinitely after it's crashed preventing nsrd from restarting nsrmmdbd
260109/26470/failed client not listed on policy_notification.log
257937//The hostname checks for Authc server should follow the latest RFC docs
261681/26479/

Getting the  error: "Failed to connect to database"  when opening a report on the NMC. Daily NMC db backup fails after an initial successful backup. There are a lot of idle postgres processes on the NMC

postgres 32721 23746  0 13:46 ?        00:00:00 postgres: lgtogst lgto_gst ::1(49282) idle

Error from  pg_basebackup log:

pg_basebackup: could not connect to server: FATAL:  sorry, too many clients already

FATAL:  sorry, too many clients already

Error from web_output log:

AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

Ticket/Escalation/ID9.0.1.2 - Description
265075//DPRINTF() output is truncated after 600 character
264477/26867/NW 9.0.1 - RH7 - Segmentation fault in RH7 following NW9.0.1 installation
263932/26448/Clone Action / nsrclone does not honour Source Pool constraint
263910//sysdesc.c:get_total_mb_used() erroneously calls donefstab(), prematurely freeing the global cached file system table
263629/26644/nsrmmdbd clients can erroneously create a connection without a sufficiently high RPC authentication flavor
263357//nsrwatch : policy "get info" for a successful job showing incorrect size for backup data
263111/26736/Virtual synthetic error top 8.2.3.2 with server network interface
262880//Nsrwatch: linux: extra characters shown on unmounting and mounting of an AFTD device
262800//nsrwatch: not showing correct status of a running group on Unix platforms
265002//gstd.exe is crashing after each successful backup.
261372//NMC Recover: Saved Saveset Recover entry is not configurable as buttons and left pane details are disabled
259044//NMC Wizard: Recovery wizard page should display destination client to make it consistent with Property help page

 

You can find above patches in ftp://ftp.legato.com/pub/NetWorker/Cumulative_Hotfixes/9.0/

 

As of NW9SP1, we see also that NSAMP and NMDA modules are getting regular patch releases.  So here is the list of those released since last blog post.

 

Ticket/Escalation/IDNMM 9.0.0.7 - Description
170643//NW149616NWC Configuration wizard for NMM, snapshot policy default setting should be "Serverless"
258308/26005/Mailbox Export to PST fails if user has explicit 'Mailbox Import Export' Role assignment. Works if user is a member of Organization management Role group
258183/26206/Exchange backups fail if database is specified in the saveset list
258173//[SP] Pause Search while doing backup
257222//[Regress] NMM is crashing when we select required volumes option
257078//NMM 901 earlier builds gets installed in the setup where NMM 901 Build 540 is already installed.
256846/26051/SQL Incremental backup fails and gets promoted to FULL for a Cross domain Mirroring setup.
256836/26018/Mailbox restore does not restore 'recoverable items' aka dumpster
256024//[Config Wizard] AG and non-AG level savesets getting created in windows cluster and AG listener resources respectively.
255477//AD Scheduled backups are failed due to recent policy changes.
255361//SCVMM add in: Special character VM is not showing under recover page for recovery
254885/25779/SQL Recovery fails with "Error failed to obtained tape session" error when NMM resolves the hostname to a different canonical name than NW's canonical name (FQDN vs shortname)
253335//All special characters are not browsable in NMM gui
251723//Hyper-V CSV backup pass but NMC shows failed for some savesets
248155//Backup fails on SQL machines when client is configured with IP instead of host name
243348//NMM GLR mailbox browsing. Check the status of user during mailbox browse, to improve GUI response for deleted mailbox restore
Ticket/Escalation/IDNMM 9.0.0.8 - Description
261679//[SP] Pause Search failure causes backup to fail
259650/26345/Mailbox Restore to PST excludes 'dumpster' folder
259529/26318/Incorrect retention policy applied to manual striped SQL backups
259413/25811/Exchange admin tool does not work with Parent Child Domain configuration
255894//NMM GUI closes immediately after selecting GLR pane after rollforward restore
Ticket/Escalation/IDNMM 9.0.1.1 - Description
62987//Port (Bug 253239) Exchange PowerShell interface error reporting to 9.0.1 cumulative and nw_9_0_dev
262634/26403/NMM 9.0.0.6 Fails GLR/RDB browse due to MAPI popup errors
259529/26318/Incorrect retention policy applied to manual striped SQL backups
254809/25811/NMM Exchange Admin configuration does not work in a parent child domain configuration.
Ticket/Escalation/IDNMM 9.0.1.2 - Description
262977/26614/Port request for NMSQL fix of 19987 / 220770 into NMM
262348/26447/SQL Backup of some DBs failing after upgrade to 8.2.3.1
256979//RFE: Provide a user option to disable CBT after every FULL backup, so as to not run the VM off the differencing disk.
Ticket/Escalation/IDNMSAP 9.0.1.2 - Description
264422//The tmp utl files produced by SAPO scheduled backup (/nsr/applogs/tmp) and SAPO RUI ($ORACLE_HOME/dbs) should be relocated to /nsr/applogs
264326//SAP HANA : Delta backups always promoted to full with HANA SPS 12 clean installation
263069//Backups failing when dedup is enabled
262404/26523/SAP HANA logs for MDC taking too long consistently
Ticket/Escalation/IDNMDA 9.0.1.2 - Description
260650/26424/NSR_RMAN_OUTPUT=SHELL causes backup to fail for NMDA Oracle backup in NetWorker 9

 

Above NMM patches you can locate in ftp://ftp.legato.com/pub/NetWorker/NMM/Cumulative_Hotfixes/

Above NMDA patches you can locate in ftp://ftp.legato.com/pub/NetWorker/NMDA/9.0.1/9.0.1.2/

Above NMSAP patches you can locate in ftp://ftp.legato.com/pub/NetWorker/NMSAP/9.0.1/9.0.1.2/