Neutrino: MySQL Galera does not come-up properly after a graceful restart

Neutrino: MySQL Galera does not come-up properly after a graceful restart

Environment:

Neutrino

VxRack System 1000 Neutrino Nodes

Neutrino Software

 

Description:

After a graceful shutdown and restart of the POD, cannot login to either the Neutrino UI or Horizon.

 

keystone logs and show issue with SQL Connections:

 

2016-06-02 15:43:20.083 35 TRACE keystone.common.wsgi DBConnectionError: (OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') None None

2016-06-02 15:43:20.083 35 TRACE keystone.common.wsgi

2016-06-02 15:43:20.090 35 WARNING oslo_db.sqlalchemy.session [-] SQL connection failed. 10 attempts left.

The container on boston keep restarting with these error messages:

 

160602 15:50:17 [ERROR] WSREP: failed to open gcomm backend connection: 131: invalid UUID: 00000000 (FATAL)

         at gcomm/src/pc.cpp:PC():270

160602 15:50:17 [ERROR] WSREP: gcs/src/gcs_core.cpp:long int gcs_core_open(gcs_core_t*, const char*, const char*, bool)():206: Failed to open backend connection: -131 (State not recoverable)

160602 15:50:17 [ERROR] WSREP: gcs/src/gcs.cpp:long int gcs_open(gcs_conn_t*, const char*, const char*, bool)():1379: Failed to open channel 'galera_cluster' at 'gcomm://10.xxx.xx.11,10.xxx.xx.15,10.xxx.xx.19,': -131 (State not recoverable)

160602 15:50:17 [ERROR] WSREP: gcs connect failed: State not recoverable

160602 15:50:17 [ERROR] WSREP: wsrep::connect() failed: 7

160602 15:50:17 [ERROR] Aborting

File gvwstate.dat is empty or missing:

 

  /opt/emc/caspian/mysql-galera/data/gvwstate.dat

 

 

 

Resolution:

The solution requires stopping mysql containers, manually creating the gvwstate.dat files on the node and restarting the container. Once done the galera cluster will return to normal state and be able to login UI.

For a detailed step by step resolution please refer to EMC Support Solution 485445 https://support.emc.com/kb/485445