In this second article in this AntiVirus series, we'll take a look at policies, exclusions, global configuration, plus some monitoring and sizing ideas.


The OneFS WebUI and CLI can be used to configure antivirus policies, adjust settings, and manage antivirus scans and reports.


icap_3.png

 

Antivirus scanning can be enabled or disabled via the check-box at the top of the page. Similarly, the AV settings can be viewed and changed via the CLI.


# isi antivirus settings view

           Fail Open: Yes

        Glob Filters: -

Glob Filters Enabled: No

Glob Filters Include: No

       Path Prefixes: /ifs/data

              Repair: Yes

       Report Expiry: 1Y

       Scan On Close: Yes

        Scan On Open: Yes

Scan Cloudpool Files: No

   Scan Size Maximum: 1.00G

             Service: Yes

          Quarantine: Yes

            Truncate: No

 

For example, the following syntax will change the maximum file scanning size from 1GB to 100 MB:


# isi antivirus settings modify --scan-size-maximum 100M


To prevent specific files from being scanned by antivirus scans, from the WebUI navigate to Data Protection > Antivirus > Settings and configure filters based on file size, name, etc.


icap_4.png


To exclude files based on file name, select Enable filters and configure either inclusions or exclusions. Specify one or more filters, which can include the following wildcard characters:


Wildcard Character

Description

*

Matches any string in place of the asterisk.

[  ]

Matches any characters contained in the brackets, or a range of characters separated by a dash.

?

Matches any character in place of the question mark.

 

Be aware that these filters apply globally to all antivirus scans.


OneFS can be configured to automatically scan files as they are accessed by users from the WebUI by navigating to Data Protection > Antivirus > Settings. In the On-Access Scans area, specify whether you want files to be scanned as they are accessed. 


On-access

Description

Open

To require that all files be scanned before they are opened by a user, select Enable scan of files on open, and then specify whether you want to allow access to files that cannot be scanned by selecting or clearing Enable file access when scanning fails.

Close

To scan files after they are closed, select Enable scan of files on close.


Note that on-access scans operate independently of antivirus policies.


For example, the following syntax will disable scanning on file open:


# isi antivirus settings modify --scan-on-open no


The amount of time OneFS retains antivirus reports before automatically deleting them can be configured via the WebUI by navigating to Data Protection > Antivirus > Settings > Reports and specifying a retention period.

To add an ICAP server from the WebUI, navigate to Data Protection > Antivirus > ICAP Servers, select Add an ICAP Server, enter its IP address,  and click Enable.


Or, from the CLI:


# isi antivirus servers create --enabled <url>


An antivirus policy that causes specific files to be scanned for viruses each time the policy is run can be crafted from the WebUI by navigating to Data Protection > Antivirus > Policies and creating an Antivirus Policy. Name the policy, specify the directory(s) that you want to scan in the Paths field, set the preferred recursion depth (full or number of subdirectories), and configure a schedule if desired. Note that scheduled policies can also be run manually at any time.


Option

Description

Run the policy only manually

Click ‘Manual’.

Run the policy according to a schedule

  1. Click ‘Scheduled’.
  2. Specify how often you want the policy to run.

 

Individual files can also be manually scanned for viruses. For example, the following CLI syntax will initiate a scan of the /ifs/data/infected file:

 

# isi antivirus scan /ifs/data/infected

Result: Succeeded

Report ID: R:5e1d083c:6f86


To quarantine a file to prevent it from being accessed by users, from the WebUI, browse to Data Protection > Antivirus > Detected Threats and select More > Quarantine File in the appropriate row of the Antivirus Threat Reports table. Or from the CLI:


# isi antivirus quarantine /ifs/data/infected


The quarantine status of a file can be inspected as follows:

# isi antivirus status /ifs/data/infected

File: /ifs/data/infected

  Last Scan: Never

Quarantined: Yes

 

It can also easily be un-quarantined, or released:

# isi antivirus release /ifs/data/infected

# isi antivirus status /ifs/data/infected

File: /ifs/data/infected

  Last Scan: Never

Quarantined: No

 

If a threat is detected in a file, and the file is irreparable and no longer needed, you can manually remove the file. For example, the following command deletes the /ifs/data/infected file:


# rm /ifs/data/infected


When sizing the ICAP servers for a cluster, the number of ICAP servers deployed per Isilon node (ICAP/node) is often used as the primary metric. With multiple ICAP servers per node, OneFS distributes files to the ICAP servers in an equally weighted, round-robin manner and does not consider the processing power of the ICAP servers when allocating files. Because of this, try to keep the configuration and resource allocation (CPU, memory, network) of each ICAP server relatively equal to avoid scanning bottlenecks.

 

  • If ICAP servers are virtual machines, their resources should be dedicated (CPU, memory, network) and the OS optimized to minimize swapping and latency.
  • Network latency is a significant factor to keep in mind when planning a cluster ICAP solution. Where possible, ensure that network routing is symmetric (e.g. switch stacks, hops, delay, static routes, sbr, etc) and keep latency to a minimum.
  • The majority of infected files tend to be <10MB in size, so reducing the file size to be scanned is also advisable. Select and setup a routine for updating the file types and sizes that would be scanned or skipped.
  • Scan only the data that is necessary, and ensure the cluster is running a current OneFS version.
  • For clusters with heavy workloads or high rate of change, consider scheduling scans during low periods instead of on access/close.
  • Round-robin scanning task allocation is per node, rather than across cluster. This can potentially lead to variable congestion on individual ICAP servers, depending on how clients connect and load a cluster.
  • If the cluster is running SyncIQ replication and has a heavy workload, it is also good to stagger the activity.
  • Suggest creating a separate RBAC account for AntiVirus operations.


The following guidelines are a useful starting point for sizing ICAP server sizing:


ICAP Attribute

Details

ICAP servers

  • Policy scan: Minimum of two ICAP servers for a small cluster, increasing as cluster grows.
  • On-access scan: At least one dedicated ICAP server per node.

ICAP threads

Test different thread numbers to determine the best value for your environment. For example:

  • McAfee: 50 to 100
  • Symantec: ~20

Network bandwidth

Suggested network connectivity bandwidth for ICAP servers, depending on the average file size:

  • <1 MB average file size: 1Gbps for ICAP servers
  • >1 MB average file size: 10Gbps for ICAP servers

CPU Load

In general, the scanning workload for ICAP servers is CPU intensive. If ICAP server CPU utilization >95%, either increase CPU of the ICAP servers or raise the ICAP servers per cluster ratio.

 

The number of ICAP server threads is one of the primary ICAP server-side tunables, and recommendations vary widely across vendors and products. However, the  ‘too_busy’ status and 'failed to scan' ratio are useful in gauging whether a cluster’s ICAP server(s) are too busy to handle further requests.


Firstly, OneFS reports the status of ICAP servers connected to isi_avscan_d, and this can be dumped to a logfile and viewed using the following command:


# kill -USR2 `ps -auxw | grep -i avscan | grep -v grep | awk '{print $2}'`


All of the isi_avscan_d daemon’s state information is logged to the file /var/log/isi_avscan_d.log. The following CLI command can be used to parse the ICAP server status from this file. For example:


# cat /var/log/isi_avscan_d.log | grep “too_busy”

2020-01-08T23:15:22Z <3.6> tme-sandbox-3 isi_avscan_d[71792]: [0x80070ba00]    too_busy: true


If the ‘too_busy’ field is set to ‘true’, as above, this typically indicates that an ICAP server is overloaded, suggesting that there are insufficient ICAP servers for the workload. In this case, the recommendation is to add more ICAP servers until the too_busy state is reported as ‘false’ for all ICAP servers. Conversely, be aware that having an ICAP server to cluster node ratio that is too high can also lead to performance issues. This becomes more apparent on large clusters with a high rate of change.


Secondly, the ‘failed to scan’ ratio can be calculated from the ‘failed’ and ‘scanned’ stats available via the following sysctl command:


# sysctl efs.bam.av.stats | egrep -i 'failed|scanned'


The formula for determining the ‘failed to scan’ ratio is:


(‘Failed’ number / ‘Scanned’ number) x 100 = Failed to scan %


If this percentage is much above zero, consider adding additional ICAP servers, or increasing bandwidth to existing servers if they’re network-bound.