Data Protection Advisor (DPA) - Error 500 returned when running reports


   Article Number:     536550                                   Article Version: 2     Article Type:    Break Fix 




Data Protection Advisor





When attempting to run a Client Summary or Job Summary report a popup window appears with an Error 500. The reports never run.   
    Examining the Data Protection Advisor (DPA) Application's server log files, there were many warning messages stating that it could not insert data into tables or failing to execute queries.   
    For example:   

      " Failed executing updatesql=insert into bkup_ddclone .......Caused by: java.sql.SQLException: Error"   
      "Unhandled exception thrown from onMessage: java.lang.IllegalStateException: Failed processing agent report: BaseTransaction.rollback - ARJUNA016074: no transaction! ..........Caused by: Failed executing querysql=select * from backupevent "    
    Examining the DPA Datastore logs, the following errors were observed:   
      "ERROR: could not extend file "pg_tblspc/16390/PG_9.6_201608131/16385/35561.12": No space left on device"     
      "ERROR: could not open file "pg_tblspc/16389/PG_9.6_201608131/16385/35570": No such file or directory".   
    Another error message in the Datastore logs is observed after a Datastore restart:   
    "2019-08-07 13:22:26.172 EDT 11996 5d4b08d0.2edc 1 apollo:ERROR: could not open file "pg_tblspc/16389/PG_9.6_201608131/16385/35558.5" (target block 2322516): previous segment is only 13128 blocks "   
    Examining the Datastore file system it is found that the file system where the database resides (...\services\datastore\data) has no space left.   






The Error 500 popup window is a generic error from the DPA Application, stating that it failed to perform the last action or command.   
    The cause of the inability to run reports is that the database is out of disk space.   
    When the file system fills up there are multiple issues that can occur.   
    If the DPA UI is already running, misbehavior and failures will be seen in multiple areas in the UI, including with running reports. This is because the DPA Application needs to be able to query the database for information and data. This is not normally possible when the database file system is full.   
    The database could become corrupted or damaged in some way. Postgres caches some amount of data in shared memory at any point in time. If the file system fills up, then Postgres cannot commit (write) this data to the file system as it should. This can cause data that was to be inserted to be lost. It can also cause database files to be incomplete and damaged. Lastly, other data may not be written as it should, including database control files.   
    It is also likely that the Datastore may not be able to be restarted once the file system is full.






To resolve this problem requires that more file system space (disk) be provided to the Datastore database. This can be done in any number of ways, including extending the file system or adding a new disk. Please consult with the server's Administrators for assistance with adding space.   
    Note: There is a common misconception that data can simply be deleted from the Datastore database to recover space. This is often not possible for a number of reasons. One reason being that one cannot start the Datastore once the file system is full. Another being that deleting data from a database is a complex process that requires time consuming investigation into the nature and amount of data in any number of tables. This is often not a task that can be accomplished in the short term.   
    Once disk space has been added, one can then start the Datastore database again.    
    On restart, the database will attempt to perform a recovery from the previous abnormal termination. The recovery startup will attempt to make any corrections it can so the database is functional and operating. This  may or may not result in a normal and functioning database. It could result in a database that appears to run normally, but has internal structure issues or damage. Also as previously mentioned it likely will result in some amount of lost data.   
    After restarting the Datastore database, the database should be examined by Dell EMC Technical Support to ensure that it is functioning normally and that there are no other after effects or issues.   
    It is also recommended that a Datastore export is performed to ensure that there is a viable copy of the database available going forwards. Preferably run the export command with the "--debug" switch added to it, so it provides debug output. Redirect this output to a file, so it can be reviewed afterwards for any issues.   
    Please contact Dell EMC Technical Support for further information and assistance.