Dell EMC VxRail: VxRail Manager postgresqlGMT.log growing too large, eventually consumes storage leading to VxRail manager inaccessible

           

   Article Number:     540375                                   Article Version: 3     Article Type:    Break Fix 
   

 


Product:

 

VxRail Appliance Family

 

Issue:

 

 

Vxrail manager postgresqlGMT.log growing too large, eventually consumes storage leading to VxRail manager inaccessible.   
   
    a) Take Putty to VxRail manager    
    b) Run df -h command and you will see similar output.   
   
    Evidence : vxmgmt353a:/ # df -h   
    Filesystem Size Used Avail Use% Mounted on   
    devtmpfs 3.9G 0 3.9G 0% /dev   
    tmpfs 3.9G 4.0K 3.9G 1% /dev/shm   
    tmpfs 3.9G 369M 3.6G 10% /run   
    tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup   
    /dev/sda3 16G 12G 2.8G 81% /   
    /dev/sda1 124M 35M 83M 30% /boot   
    /dev/mapper/data_vg-store2 14G 39M 13G 1% /data/store2   
    /dev/mapper/data_vg-store1 2.0G 3.1M 1.9G 1% /data/store1   
    tmpfs 799M 0 799M 0% /run/user/498   
    tmpfs 799M 0 799M 0% /run/user/1000
                                                           

 

 

Cause:

 

 

Log rotation is not happening on the below file leading to disk space being utilized   
   
    Path : /var/lib/pgsql/data/pg_log/postgresqlGMT.log    
   
     
                                                           

 

 

Change:

 

 

Navigateavigate to the path :/var/lib/pgsql/data # cat postgresql.conf.   
   
    Once you cat the file you will be able to find the below output   
   
    listen_addresses='127.0.0.1' # Listen on local Unix domain and TCP/IP socket   
    logging_collector=on # Redirect output to pg_log directory   
   
    log_line_prefix='%t [%p]: [%l-1] user=%u,db=%d ' # log timestamp, process-id, session log line#, user and database   
    log_min_messages=warning # Write messages above warning levels to the server log.   
    log_min_error_statement=warning # Write SQL statements that cause an warning to the server log.   
    log_min_duration_statement=3000 # log SQL which takes longer than 3000ms.   
    log_lock_waits=on # A log message will be produced when a session waits longer than deadlock_timeout.   
    log_temp_files=1024 # log temp files with size >= 1024kb   
    log_statement=all # All SQL statements will be logged.   
   
    log_filename='postgresql-%Y-%m-%d_%H%M%S.log' # name without date/time so it can be rotated using Linux logrotate   
    log_rotation_age='0' # disable log rotation as it is handled by Linux logrotate   
    log_rotation_size=100MB # disable log rotation.   
   
    client_min_messages=warning # Do not report debug and notice level messages.   
   
    max_connections=100   
   
    effective_cache_size=128MB   
    shared_buffers=8MB   
    work_mem=1MB   
    maintenance_work_mem=16MB   
    wal_buffers=64kB   
    port=5432
                                                           

 

 

Resolution:

 

 

Please modify the lines accordingly as following and add log_truncate_on_rotation=on as below, before modification, please save a copy of the orignal postgresql.conf file:   
   
    log_filename='postgresql-%H.log' ( Hear H refers to Hours , basically we are setting rotation based on hours )    
    log_rotation_age=60             # disable log rotation as it is handled by Linux logrotate   
    log_rotation_size=100MB              # disable log rotation.   
    log_truncate_on_rotation=on   
   
    After the changes are made please delete the file  postgresqlGMT.log and restart the Vxrail manager   
   
    Path for the log file is as mentioned : /var/lib/pgsql/data/pg_log/postgresqlGMT.log 
                                                           

 

 

Notes:

 

 

Also we have a Bug : 17868