Powerpath for Linux: Powerpath does not start after reboot and needs to be manually started


   Article Number:     499077                                   Article Version: 6     Article Type:    Break Fix 




PowerPath for Linux,PowerPath





Operating System: RHEL 7.x, OEL 7.x & SLES 12.   
    This does not apply for versions RHEL 5.x, 6,x & SLES 11   
    EMC Software: PowerPath for Linux 6.x   
    PowerPath does not start automatically after the reboot of the host   
    PowerPath can be started manually without any issue.   
    No errors are logged in boot.log when Powerpath service starts and stops   
    After system reboots, you would see that powerpath fails with "Initialization Error" since the service was unable to start   

# powermt display dev=allERROR: Cannot open PowerPath. Initialization error    
# cat /var/log/boot.log |grep PowerStarting PowerPath startup script...[ OK ] Stopped PowerPath startup script. <---Starting PowerPath Management Daemon...Starting PowerPathPost startup script...[ OK ] Started PowerPathPost startup script.[ OK ] Started PowerPath Management Daemon.    







      On start, PowerPath has a few checks initially before it proceeds with actually starting.   One of these is to check when PowerPath was last started and this is determined by calculating the age of a marker file. On bootup, the system clock is initially synced from the hardware clock.  However, if a NTP client is configured on the host, then when that starts, the system clock is updated from a NTP server.      
      The problem arises if the hardware clock is too much out of sync with that of the NTP clock, usually when it is much behind.   On host bootup, PowerPath starts much before the NTP client.  Hence, when it tries to determine when it tries to calculate the age of the marker file, it will be a negative value and PowerPath exits immediately without proceeding with the start.   







    Update the hwclock to the system clock using the following command on the host:   
    hwclock --systohc   
    This will update the hardware clock to the correct time on the host.   
    Permanent Fix:   
    The issue will be fixed in PowerPath for Linux 6.3






You can run the below command to check the time when powerpath was last started.   
    'systemctl status PowerPath.service'     
      Usually you would see that the last time when powerpath was active has a huge time difference.

systemctl status PowerPath.service● PowerPath.service - PowerPath startup scriptLoaded: loaded (/usr/lib/systemd/system/PowerPath.service; enabled; vendor preset: disabled)Active: inactive (dead) since Thu 2017-04-06 12:00:15 GET; 4h 0min agoProcess: 2075 ExecStopPost=/usr/lib/PowerPath/PowerPath check_service_stop (code=exited, status=0/SUCCESS)Process: 1807 ExecStartPost=/usr/lib/PowerPath/PowerPath check_service_start (code=killed, signal=TERM)Process: 1665 ExecStart=/usr/lib/PowerPath/PowerPath start (code=exited, status=0/SUCCESS)Main PID: 1665 (code=exited, status=0/SUCCESS)    
    Another way to find is the enable '-x' flag in Powerpath startup script.   
    For OEL 7.x, the startup scripts are    
    # head -5 /usr/lib/PowerPath/PowerPath   
    #!/bin/sh -x   
    Once you reboot the host, the logs would be captured in boot.log. Also run the below command where last_boot would have the extract of the same   
    journalctl -b 0 > last_boot   
    In the below verbose logs which was collected above, we see the time lag of -13991   
    Apr 20 10:23:23 revdb1.mfg.ge PowerPath[1751]: + '[' yes == yes -a -f /tmp/.pp_systemctl_check ']'       
        Apr 20 10:23:23 revdb1.mfg.ge PowerPath[1751]: ++ date +%s       
        Apr 20 10:23:23 revdb1.mfg.ge PowerPath[1751]: ++ date +%s -r /tmp/.pp_systemctl_check       
        Apr 20 10:23:23 revdb1.mfg.ge PowerPath[1751]: + '[' -13991 -lt 5 ']'        
        Apr 20 10:23:23 revdb1.mfg.ge PowerPath[1751]: + rm -f /tmp/.pp_systemctl_check       
        Apr 20 10:23:23 revdb1.mfg.ge PowerPath[1751]: + /bin/true       
        Apr 20 10:23:23 revdb1.mfg.ge PowerPath[1751]: + rc_exit       
        Apr 20 10:23:23 revdb1.mfg.ge PowerPath[1751]: + exit 0