ViPR SRM: Large File processing delay on APG database


   Article Number:     518763                                   Article Version: 3     Article Type:    Break Fix 




ViPR SRM 4.1,ViPR SRM 4.0,ViPR SRM 3.7,ViPR SRM 3.7 SP1,ViPR SRM 3.7 SP2,ViPR SRM 4.0.1,ViPR SRM 4.0.2,ViPR SRM 4.0.3,ViPR SRM 4.1.1,ViPR SRM 4.2,ViPR SRM





- In the following report Report Library>>EMC M&R Health>>Misc. Reports>>Backends File Processing, Large File processing delay on APG1 database was found.   
    User-added image   
    - Also many alerts regarding Processing Delay were found in All Alerts report.   
    - Metric count on the APG1 was in acceptable range (Less than 1 million metrics).   
    - No errors are seen in APG-Backend-apg1 cache logs.   
    - In Task-Scheduler logs the below info were seen:   

      INFO – [2017-10-30 12:00:00 EDT] – : Task Import Variable Status-apg4 – Task can not be executed on schedule. It is delayed by an execution of the task Import Variable Status-apg1.         
          INFO – [2017-10-30 12:00:00 EDT] – : Task Import Variable Status-apg2 – Task can not be executed on schedule. It is delayed by an execution of the task Import Variable Status-apg1.
        - Checked the tmp directory ( /opt/APG/Backends/APG-Backend/apgx/tmp) for APG1 and could see some corrupted files and alot of .failed files

      rw-r--r-- 1 apg apg 1965864 Nov  2  2015 group_0_1446465600_14464814461601171259281.failed-rw-r--r-- 1 apg apg  926103 Nov  2  2015 group_0_1446465600_14464816745941172798277.failed-rw-r--r-- 1 apg apg 2744197 Nov  2  2015 group_0_1446465600_14464819200211161640559.failed?????????? ? ?   ?         ?            ? property_15072984563671298243387.mysql?????????? ? ?   ?         ?            ? property_15072984755561297473889.mysql    








- The issue is due to .failed and corrupted files found in /opt/APG/Backends/APG-Backend/apgx/tmp directory.                                                           






To solve the above issue, Follow the below steps:   
        1- Go to
(where x is the number of database which the issue is found on)   
    2- Type ls –la command   
    3-Check if you have output similar to the below:   

ls: cannot access property_15072984563671298243387.mysql: No such file or directoryls: cannot access property_15072984755561297473889.mysql: No such file or directory?????????? ? ?   ?         ?            ? property_15072984563671298243387.mysql?????????? ? ?   ?         ?            ? property_15072984755561297473889.mysql    
    4- Remove the above corrupted files from the tmp directory if you have any output like them. Also remove the .failed files if found   
    5- After that do mysql-database-check on all apgs   
        Enter username: [apg]       
        Enter database: [apg]       
        <Database Name>       
        Enter password:       
      6- Restart all Backend services.
    /opt/APG/bin/ service update all     
      /opt/APG/bin/ service start all