How to configure remote logging from an Isilon OneFS cluster to a remote server

NOTE: This topic is part of the Uptime Information Hub.

 

Would  you like to configure your EMC® Isilon® cluster to send system  logs to your central syslog server or to an external syslog server like Splunk®  or Elasticsearch® for syslog data analysis? This article provides instructions on how to configure OneFS for this activity.

 

Configure OneFS for remote logging

 

The system logging daemon, syslogd, is used to collect  messages from a number of different programs. Typically, syslogd will log to a  local file, but it can also be configured to log over a network to a remote  logging server. The following steps show you how to configure OneFS to send system logs to a remote server.

 

Note: Make sure the remote server is set up to allow remote logging.

 

Send syslog events other than audit and protocol events to a remote server

  1. Open an SSH connection on any node in the  cluster and log on using the "root" account.
  2. Run the following command at the command line replacing <hostname/IP address> with the hostname or IP address of the remote server:
    isi_log_server add <hostname/IP address>
  3. You can also add the filtering syntax in the command as follows:
    isi_log_server add <hostname/IP address> <filter>
       
        The default filtering syntax is:
    '*.warn;*.notice;kern.*;ifs.info;istat.none'
       
        For example, you can configure all syslog level 4 warning messages with the  following:
    isi_log_server add <hostname/IP address> '*.warn;'

Notes:

  • This setting will persist throughout node or cluster reboots.
  • IPv6 addresses are supported only in OneFS 8.0 and later.

 

Send audit and protocol syslog events to a remote server

 

Currently the isi_log_server  command will not modify the audit and protocol part of the syslog configuration. However, by applying custom filters, you can forward these entries to the remote system.

 

You can enable remote logging of syslog events for audit and protocol activity by editing the /etc/mcp/templates/syslog.conf file so that those syslog events are sent to a defined server.

 

IMPORTANT: Do not edit the /etc/syslog.conf file. This file is automatically generated and should not be edited directly.

Use the following procedure to send audit and protocol syslog events to a remote server. These steps must be followed in order.

CAUTION: Improper editing of the /etc/mcp/templates/syslog.conf file could disable logging or have other unexpected results.

    1. Open an SSH connection on any node in the cluster and log on using the "root" account.
    2. Run the following command to back up the /etc/mcp/templates/syslog.conf file:
      cp /etc/mcp/templates/syslog.conf  /etc/mcp/templates/syslog.conf.bku1
    3. Open the /etc/mcp/templates/syslog.conf file in a text editor such as vi, edit, or nano.
    4. Add a custom filter with your syslog host. For example:
      *.warn;*.notice;kern.*;ifs.info;istat.none  @<hostname/IP address>
         
          A filter of *.* will  generate a lot of traffic.
    5. To enable remote logging of syslog events for configuration and protocol auditing, find these sections of the /etc/mcp/templates/syslog.conf file.
      !audit_config
      *.*                                         /var/log/audit_config.log
      !audit_protocol
      *.*                                         /var/log/audit_protocol.log
    6. Add a line for remote syslog servers, so that the resulting sections of the file will now look like this (this is assuming that you have enabled syslog for auditing):
      !audit_config
      *.*                                              /var/log/audit_config.log
      *.*                                              @<hostname/IP address>
      !audit_protocol
      *.*                                              /var/log/audit_protocol.log
      *.*                                              @<hostname/IP address>
    7. Save the file and exit from your editor. The  master control process (MCP) will push out your changes from the template file  into /etc/syslog.conf a short time later.
    8. Reload the configuration by sending the hang-up signal to the syslogd process:
      isi_for_array 'killall -HUP syslogd'

For detailed information about editing the /etc/mcp/templates/syslog.conf file, refer to the following:

    • View the man page by running the following  command from the command line:
      man syslog.conf
    • Refer to the "Configuring System  Logging" section in the FreeBSD Handbook available here.

Revert changes made to the syslog.conf file

 

These steps will revert the syslog.conf file to the backup version you created in the previous section.

  1. Open an SSH connection on any node in the  cluster and log on using the "root" account.
  2. Run the following command to copy and then rename the original backup of the syslog.conf file.
    cp /etc/mcp/templates/syslog.conf.bku1  /etc/mcp/templates/syslog.conf

Verify the configuration change

 

You can test the configuration change by sending a test message and using the tcpdump command to view the transmission.

  1. Open an SSH connection on any node in the cluster and log in using the "root" account.
  2. Run the following command to start tcpdump:
    tcpdump -i em0 -v port 514
  3. Open another SSH connection to the same node in step 1.
  4. Run the following command to log a test message:
    logger "this is just a test"
  5. Return to the SSH session you open in step 1. Output similar to the following appears:
    07:45:56.651890 IP (tos 0x0, ttl 64, id  xxxxx, offset 0, flags [none], proto UDP (17), length 118) xxx.xxx.xxx.xxx.52604  > yyy.yyy.yyy.yyy.syslog: SYSLOG, length: 90
    Facility news (7), Severity info  (6)
    Msg: 014-04-02T07:45:56-05:00  (ifs)/boot/kernel.amd64/kernel: em0: promiscuous mode enabled
    07:46:02.900633 IP (tos 0x0, ttl 64, id xxxxx, offset 0, flags [none], proto  UDP (17), length 87) xxx.xxx.xxx.xxx.52604 > yyy.yyy.yyy.yyy.syslog: SYSLOG,  length: 59
    Facility cron (9), Severity  emergency (0)
    Msg: 014-04-02T07:46:02-05:00 (ifs)root: this is just a test
  6. Type Ctrl-C to stop tcpdump.


NOTE: The remote server must allow remote logging for this operation to succeed. You can usually configure this by appending the
–r  parameter to the SYSLOGD_OPTIONS line in the syslog startup routine on the remote server and then restarting the local syslog daemon.