ViPR SRM: Large File processing delay on APG database

           

   Article Number:     518763                                   Article Version: 3     Article Type:    Break Fix 
   

 


Product:

 

ViPR SRM 4.1,ViPR SRM 4.0,ViPR SRM 3.7,ViPR SRM 3.7 SP1,ViPR SRM 3.7 SP2,ViPR SRM 4.0.1,ViPR SRM 4.0.2,ViPR SRM 4.0.3,ViPR SRM 4.1.1,ViPR SRM 4.2,ViPR SRM

 

Issue:

 

 

- In the following report Report Library>>EMC M&R Health>>Misc. Reports>>Backends File Processing, Large File processing delay on APG1 database was found.   
   
    User-added image   
   
    - Also many alerts regarding Processing Delay were found in All Alerts report.   
    - Metric count on the APG1 was in acceptable range (Less than 1 million metrics).   
    - No errors are seen in APG-Backend-apg1 cache logs.   
    - In Task-Scheduler logs the below info were seen:   

      INFO – [2017-10-30 12:00:00 EDT] – : Task Import Variable Status-apg4 – Task can not be executed on schedule. It is delayed by an execution of the task Import Variable Status-apg1.         
          INFO – [2017-10-30 12:00:00 EDT] – : Task Import Variable Status-apg2 – Task can not be executed on schedule. It is delayed by an execution of the task Import Variable Status-apg1.
       
       
        - Checked the tmp directory ( /opt/APG/Backends/APG-Backend/apgx/tmp) for APG1 and could see some corrupted files and alot of .failed files
     
          

   
      rw-r--r-- 1 apg apg 1965864 Nov  2  2015 group_0_1446465600_14464814461601171259281.failed-rw-r--r-- 1 apg apg  926103 Nov  2  2015 group_0_1446465600_14464816745941172798277.failed-rw-r--r-- 1 apg apg 2744197 Nov  2  2015 group_0_1446465600_14464819200211161640559.failed?????????? ? ?   ?         ?            ? property_15072984563671298243387.mysql?????????? ? ?   ?         ?            ? property_15072984755561297473889.mysql    
   

          

                                                             

 

 

Cause:

 

 

- The issue is due to .failed and corrupted files found in /opt/APG/Backends/APG-Backend/apgx/tmp directory.                                                           

 

 

Resolution:

 

 

To solve the above issue, Follow the below steps:   
            
        1- Go to
/opt/APG/Backends/APG-Backend/apgx/tmp
(where x is the number of database which the issue is found on)   
    2- Type ls –la command   
    3-Check if you have output similar to the below:   
        

ls: cannot access property_15072984563671298243387.mysql: No such file or directoryls: cannot access property_15072984755561297473889.mysql: No such file or directory?????????? ? ?   ?         ?            ? property_15072984563671298243387.mysql?????????? ? ?   ?         ?            ? property_15072984755561297473889.mysql    
   
    4- Remove the above corrupted files from the tmp directory if you have any output like them. Also remove the .failed files if found   
   
    5- After that do mysql-database-check on all apgs   
   
    /opt/APG/bin/mysql-database-check.sh       
        Enter username: [apg]       
        root       
        Enter database: [apg]       
        <Database Name>       
        Enter password:       
        watch4net
     
     
      6- Restart all Backend services.
   
   
    /opt/APG/bin/manage-modules.sh service update all     
      /opt/APG/bin/manage-modules.sh service start all