In prior releases, SyncIQ did not have built in over-the-wire encryption, leaving the door open for potential man-in-the-middle attacks when replicating datasets over untrusted networks. SyncIQ encryption, introduced in OneFS 8.2, protects the data in-flight during inter-cluster replication over the WAN.


synciq_encryption_1.png

 

SyncIQ encryption helps to secure data transfer between OneFS clusters, benefiting customers who undergo regular security audits and/or government regulations. In OneFS 8.2:


  • SyncIQ policies now support end-to-end encryption for cross-cluster communications.
  • Certificates are easy to manage with the newly added SyncIQ certificate store.
  • Certificate revocation is supported through the use of an external OCSP responder.
  • Clusters can now require that all incoming and outgoing SyncIQ policies be encrypted through a simple configuration change in the SyncIQ global settings.


SyncIQ encryption relies on cryptography, using a public and private key pair to encrypt and decrypt replication sessions. These keys are mathematically related: Data encrypted with one key is decrypted with other key, confirming the identity of each cluster. SyncIQ uses the common X.509 Public Key Infrastructure (PKI) standard which defines certificate requirements.


A Certificate Authority (CA) serves as a trusted 3rd party, which issues and revokes certificates. Each cluster’s certificate store has the CA, it’s certificate, and the peer’s certificate, establishing a trusted ‘passport’ mechanism.


synciq_encryption_2.png


A SyncIQ job can attempt either an encrypted or unencrypted handshake:


synciq_encryption_3.png

 

Under the hood, SyncIQ utilizes TLS protocol version 1.2 and OpenSSL version: 1.0.2o. Customers are responsible for creating their own X.509 certificates, and SyncIQ peers must store each other’s end entity certificates. A TLS authentication failure will cause the corresponding SyncIQ job to immediately fail, and a new CELOG event has been added in 8.2 to notify the user of a SyncIQ encryption failure.


One the source cluster, the SyncIQ job’s coordinator process passes the target cluster’s public cert to its primary worker (pworker) processes. The target monitor and secondary worker (sworker) processes receive a list of approved source cluster certs. The pworkers can then establish secure connections with their corresponding sworkers.


synciq_encryption_4.png

 

SyncIQ traffic encryption is enabled on a per-policy basis. To support this in OneFS 8.2, the CLI now includes the ‘isi certificates’ and ‘isi sync certificates’ commands for the configuration of TLS certificates:


# isi cert -h

Description:

    Configure cluster TLS certificates.

 

Required Privileges:

    ISI_PRIV_CERTIFICATE

 

Usage:

    isi certificate <subcommand>

        [--timeout <integer>]

        [{--help | -h}]

 

Subcommands:

  Certificate Management:

    authority Configure cluster TLS certificate authorities.

    server Configure cluster TLS server certificates.

    settings Configure cluster TLS certificate settings.

The following new policy configuration fields are also introduced in 8.2:

Config Field

Detail

--target-certificate-id <string>

The ID of the target cluster certificate being used for encryption.

--ocsp_issuer_certificate-id <string>

The ID of the certificate authority that issued the certificate whose revocation status is being checked.

--ocsp-address <string>

The address of the OCSP responder to which to connect.

--encryption-cipher-list <string>

The cipher list being used with encryption. For SyncIQ targets, this list serves as a list of supported ciphers. For SyncIQ sources, the list of ciphers will be attempted to be used in order.

In order to configure a policy for encryption the ‘--target-certificate-id’ must be specified. The users will input the ID of the desired certificate as is defined in the certificate manager. If self-signed certificates are being utilized, then they will have been manually copied to their peer cluster’s certificate store. For authentication, there is a strict comparison of the public certs to the expected values. If a cert chain (that has been signed by the CA) is selected to authenticate the connection, the chain of certificates will need to be added to the cluster’s certificate authority store. Both methods use the ‘SSL VERIFY FAIL IF NO PEER CERT’ option when establishing the SSL context. Note that once encryption is enabled (by setting the appropriate policy fields), modification of the certificate IDs is allowed. However, removal and reverting to unencrypted syncs will prompt for confirmation before proceeding.

 

The following procedure can be used to configure SyncIQ encryption on a pair of clusters running OneFS 8.2:


1)     First, create an X.509 certificates, one for each of the source and target clusters, and signed by a certificate authority.


Certificate Type

Abbreviation

Certificate Authority

<ca_cert_id>

Source Cluster Certificate

<src_cert_id>

Target Cluster Certificate

<tgt_cert_id>

 

These can be generated using publicly available tools, such as OpenSSL:

http://slproweb.com/products/Win32OpenSSL.html.


2)     Next, add the certificates to the appropriate source cluster stores. Each cluster gets certificate authority, its own certificate, and its peer’s certificate:


# isi sync certificates server import <src_cert_id> <src_key>

# isi sync certificates peer import <tgt_cert_id>

# isi cert authority import <ca_cert_id>


3)     On the source cluster, set the SyncIQ cluster certificate:


# isi sync settings modify --cluster-certificate-id=<src_cert_id>


4)     Add the certificates to the appropriate target cluster stores:


# isi sync certificates server import <tgt_cert_id> <tgt_key>

# isi sync certificates peer import <src_cert_id>

# isi cert authority import <ca_cert_id>


5)     On the target cluster, set the SyncIQ cluster certificate:


# isi sync settings modify --cluster-certificate-id=<tgt_cert_id>


6)     On the source cluster, create an encrypted SyncIQ policy:


# isi sync policies create <pol_name> sync <src_dir> <target_ip> <tgt_dir> --target-certificate-id=<tgt_cert_id>


Or modify an existing policy on the source cluster:


# isi sync policies modify <pol_name> --target-certificate-id=<tgt_cert_id>


So that’s what’s required to get encryption configured across a pair of clusters both running OneFS 8.2. There are several addition optional encryption configuration parameters available. These include:


  • Updating the policy to use a specified SSL cipher suite:


# isi sync policies modify <pol_name> --encryption-cipher-list=<suite>


  • Configuring the target cluster to check the revocation status of incoming certificates:


# isi sync settings modify --ocsp-address=<address> --ocsp-issuer-certificate-id=<ca_cert_id>

 

  • Modifying how frequently encrypted connections are renegotiated on a cluster:


# isi sync settings modify --renegotiation-period=24H


  • Requiring that all incoming and outgoing SyncIQ policies are encrypted:


# isi sync settings modify --encryption-required=True


To troubleshoot SyncIQ encryption, first check the reports for the SyncIQ policy in question. The reason for the failure should be indicated in the report. If the issue was due to a TLS authentication failure, then the error message from the TLS library will also be provided in the report. Also, more detailed information can often be found in /var/log/messages on the source and target clusters, including:


  • ID of the certificate that caused the failure.
  • Subject name of the certificate that caused the failure.
  • Depth at which the failure occurred in the certificate chain.
  • Error code and reason for the failure.


Before enabling SyncIQ encryption, be aware of the potential performance implications. While encryption only adds minimal overhead to the transmission, it may still negatively impact a production workflow. Be sure to test encrypted replication in a lab environment that emulates the environment before deploying in production.


Note that both the source and target cluster must be upgraded and committed to OneFS 8.2, prior to configuring SyncIQ encryption.