Troubleshooting Steps to add managed systems to DD Management Center

           

   Article Number:     487552                                   Article Version: 3     Article Type:    Break Fix 
   

 


Product:

 

Data Domain Management Center

 

Issue:

 

 

Unable to add specific Data Domain devices to DDMC.   
   
    Example:   
   
    SE@DDMC01##  managed-system add phxdd01.tmp.medtronic.com force   
    The SHA1 fingerprint for the remote host's CA certificate is   
    F1:D2:22:95:7C:45:C9:69:CB:76:25:18:C7:33:30:43:7A:CA:98:B9   
    Do you want to trust this certificate?  Are you sure? (yes|no) [no]: yes   
   
    ** Once added, all "admin"  role users on this DD Management Center   
       will operate on "phxdd01.tmp.medtronic.com" system with "admin" role.   
   
    To allow "phxdd01.tmp.medtronic.com" to be managed by this DD Management Center,   
    Enter "phxdd01.tmp.medtronic.com" sysadmin password:   
    ok, proceeding.   
    *** Add phxdd01.tmp.medtronic.com failed:   
   
    System "phxdd01.tmp.medtronic.com" is in the "unknown" state. Data collection is disabled   
   
    Another error message a customer may get when trying to add a new managed system to DDMC is as follows:   

**** managed-dd.example.com: Error communicating with host ddmc.example.com: error occurred in the SSL/TLS handshake.    
                                                             

 

 

Cause:

 

 

It can be due to various reasons.   
   
     - Connectivity Issue.   
    - Invalid entries in DD   
    - Invalid entries on DDMC   
    - Required port is not open   
    - SSL/TLS protocol version mismatch between the DDMC and the DD to be managed   
   
   
    Logs:   
   
    DDMC:   
    Messages.engineering:   
    ul 29 19:04:36 MSPjDDMC01 sms: NOTICE: Trust with host phxdd01.medtronic.com has been added   
    Jul 29 19:09:42 MSPjDDMC01 -ddsh: NOTICE: MSG-DDSH-00017: (tty=pts/0, session=8899) tassos1: command "managed-system add phxdd01.tmp.medtronic.com force" exited with code: 95   
    Jul 29 20:58:37 MSPjDDMC01 -ddsh: NOTICE: MSG-DDSH-00009: (tty=pts/0, session=8899) tassos1: command "managed-system add phxdd01.tmp.medtronic.com force"   
    Jul 29 21:04:36 MSPjDDMC01 sms: WARNING: ems_post_event: Failed to initialize event: Incompatible managed system version. EVT-OBJ::SystemName=phxdd01.tmp.medtronic.com EVT-INFO::DetectedVersion=   
    Jul 29 21:23:32 MSPjDDMC01 sms: NOTICE: Trust with host phxdd01.medtronic.com has been added   
    Jul 29 21:47:24 MSPjDDMC01 -ddsh: NOTICE: MSG-DDSH-00017: (tty=pts/0, session=8899) tassos1: command "managed-system add phxdd01.tmp.medtronic.com force" exited with code: 245   
   
    sms.info   
    07/29 21:04:36.487 (tid 0x6ffbca0): **** Error communicating with host phxdd01.tmp.medtronic.com: Error communicating with host phxdd01.tmp.medtronic.com:   
    error occurred in the SSL/TLS handshake.   
    07/29 21:04:36.509 (tid 0x6ffbca0): Workflow Getting system data (ID 1434912) starts child workflow (ID 1434913) to   
    get current node config & status info for host "phxdd01.tmp.medtronic.com"   
    07/29 21:04:36.521 (tid 0x70005a0): Workflow (ID 1434913) begin to get_node_info   
    for host "phxdd01.tmp.medtronic.com"   
    07/29 21:04:36.716 (tid 0x70005a0): **** Error communicating with host phxdd01.tmp.medtronic.com: error occurred in the   
    SSL/TLS handshake.   
    07/29 21:04:36.723 (tid 0x70005a0): Workflow (ID 1434913) detected host "phxdd01.tmp.medtronic.com" is unreachable. No data collection is   
    performed.   
    07/29 21:04:36.733 (tid 0x70005a0): WARNING: ems_post_event: Failed to initialize event: Incompatible managed system version. EVT-   
    OBJ::SystemName=phxdd01.tmp.medtronic.com EVT-INFO::DetectedVersion=   
     
                                                           

 

 

Resolution:

 

 

Below are the troubleshooting steps that can be followed to resolve the issue, however if the error received is about "error occurred in the SSL/TLS handshake", this is the result of security hardening for later DDMC releases, which was not made to older DDOS versions, and hence despite the DDMC / DDOS combination may be a supported one as per the matrix, it will not work. KB article Does DDOS / DDMC support TLS versions 1.1 and 1.2? has all the technical details, however, the problem occurs when using DDMC 6.1 to manage DDs in versions older than DDOS 5.7.4.0, hence this issue is resolved by upgrading the managed DD to DDOS 5.7.4.0 or later.   
   
    For other possible causes of problems, follow the troubleshooting steps below:   
   
    1. Check  the connectivity between DD and DDMC via ping and net lookup commands both ways.   
   
    2. Add appropriate host entries if required to make ping and lookup successful.   
   
    3. From DDMC, also check connection to DD by running below command:   
   
    #managed-system check-connection <DD Hostname>   
   
    4. Access SE mode and Check 3009 port is open both ways by using telnet:   
   
    on both DDR and DDMC access SE mode by opening a SSH command line connection [with putty for example]    
   
    1) at login execute command system show serialno   
    2) execute priv set se   
    3) authenticate using the serial number from step 1   
   
    On DD    
    #se telnet <DDMC IP> 3009   
   
    On DDMC   
    #se telnet <DD IP> 3009   
   
    5. Compare the fingerprint that DDMC is fetching while adding DD to DDMC with that of CA certificate of DD .    
    DDMC should pick up correct DD fingerprint.   
   
    SE@phxdd01#adminaccess certificate show detailed   
    Type:                host   
    Cert Type:           Host Certificate   
    Application:         https   
    Subject/Issued To:   phxdd01.tmp.medtronic.com   
    Issued By:           phxdd01.tmp.medtronic.com   
    Valid From:          Sat Aug  1 01:30:36 2015   
    Valid Until:         Wed Jul 25 08:30:36 2046   
    Fingerprint:         7F:81:11:BC:F5:10:40:83:68:87:81:F5:97:77:EF:6C:EF:02:74:82   
   
    Type:                ca   
    Cert Type:           Root CA   
    Application:         trusted-ca   
    Subject/Issued To:   phxdd01.tmp.medtronic.com   
    Issued By:           phxdd01.tmp.medtronic.com   
    Valid From:          Sun Aug  2 08:30:36 2015   
    Valid Until:         Wed Jul 25 08:30:36 2046   
    Fingerprint:         F1:D2:22:95:7C:45:C9:69:CB:76:25:18:C7:33:30:43:7A:CA:98:B9    
   
    SE@DDMC01##  managed-system add phxdd01.tmp.medtronic.com force   
    The SHA1 fingerprint for the remote host's CA certificate is   
    F1:D2:22:95:7C:45:C9:69:CB:76:25:18:C7:33:30:43:7A:CA:98:B9   
    Do you want to trust this certificate?  Are you sure? (yes|no) [no]: yes   
   
    6. On DD , Check the Hostnames for Host as well as CA certificate under Subject column. It should be same unlike below:   
   
    tassos1@jaxdd01# hostname   
    The Hostname is: jaxdd01.corp.medtronic.com   
    assos1@jaxdd01# adminaccess certificate show   
    Subject                              Type   Application   Valid From                 Valid Until                Fingerprint   
    ----------------------------------   ----   -----------   ------------------------   ------------------------   ------------------------------------------------------------   
    jaxdd01.corp.medtronic.com           host   https         Sun Dec  8 12:16:08 2013   Wed Nov 30 18:16:08 2044   2A:21:3E:1E:43:C9:77:F7:20:EF:E5:DF:D9:C9:9A:F8:4C:33:5E:0B   
    jaxdd01.jaxdd01.ent.medtronics.com   ca     trusted-ca    Wed Feb 22 12:41:58 2012   Sat Feb 14 12:41:58 2043   AE:AF:8A:E9:0D:0C:F3:53:B5:A7:BF:D8:38:BC:2D:DA:CF:E5:E9:C8   
    ----------------------------------   ----   -----------   ------------------------   ------------------------   ------------------------------------------------------------   
   
    If you see mismatch in hostname as in above output, then you need to re-generate certificate on DD.   
   
    To re-generate DD Certificate , TSE needs to goto BASH Mode. This step is required to be handled by Data Domain Support Engineer.   
   
    Once done, you will see the certificate as below:   
   
    # ddsh -a adminaccess certificate show   
    Subject                      Type   Application   Valid From                 Valid Until                Fingerprint   
    --------------------------   ----   -----------   ------------------------   ------------------------   ------------------------------------------------------------   
    jaxdd01.corp.medtronic.com   host   https         Sat Aug  8 06:39:31 2015   Wed Aug  1 10:39:31 2046   D5:26:79:20:3A:2F:73:41:7E:A8:5C:9B:69:54:11:8B:33:E9:BD:D9   
    jaxdd01.corp.medtronic.com   ca     trusted-ca    Sun Aug  9 11:39:31 2015   Wed Aug  1 10:39:31 2046   02:A0:F7:49:E1:16:BC:8E:FD:47:E4:24:C3:AE:45:7D:B1:8B:0C:3D   
    --------------------------   ----   -----------   ------------------------   ------------------------   -----------------------------   
   
    7. On DDMC, make sure all valid hostnames are added as Managed-system and under trust.   
    #adminaccess trust show   
    #managed-system show   
   
    - You can even compare outputs of both the commands above and see if there is any mis-match.   
    - After figuring out, invalid/incorrect/not in use DD's hostname, you can now proceed to delete their trust from DDMC.   
   
    Run on DDMC   
    Remove DDR trust, run this CLI   
    Command :#adminaccess trust del host <Invalid Hostname>   
   
    8. Now try to re-add the DD to DDMC via GUI/CLI with force option   
    #managed-system add <DD Hostname> force   
   
    9. Sync command can be used anytime when you want to sync managed systems on DDMC:   
    #managed-system sync   
    #managed-system show