The Benefits of the Job Engine

NOTE: This topic is part of the Uptime Information Hub.


The Benefits of the Job Engine


System administrators work hard to ensure that their EMC Isilon cluster runs smoothly for their users. Performing routine maintenance is a critical process for achieving optimal cluster performance. But cluster maintenance can be time consuming. Fortunately, the Isilon OneFS®operating system makes cluster maintenance efficient by using the Job Engine. The Job Engine is the core of the Isilon work distribution system. It manages maintenance jobs that run in the background for optimal performance.


Performing cluster maintenance in most traditioonal network attached storage (NAS) solutions requires a system administrator to start a diagnostic tool or test on each set of drives within a storage head in a cluster, which can potentially impact workflows while diagnostics are running. The OneFS Job Engine offers several advantages over the traditional NAS maintenance approach, including the following:

  • Minimal impact on cluster performance while running maintenance jobs
  • Easy scheduling of cluster maintenance jobs
  • The ability to run multiple jobs at the same time in OneFS 7.1


This article offers an overview of the benefits of Job Engine features.

Minimizing Impact on Cluster Performance


The Job Engine runs jobs on hard drives across the entire cluster. It divides large storage management and protection jobs into smaller tasks and work items, and then allocates them to worker threads on each node. The Job Engine takes advantage of parallel process execution wherever possible. This means that it adaptively manages maintenance jobs based on available cluster resources.


Each job is assigned a default priority and impact policy. These default priorities and impact policies control the impact of jobs on cluster performance as the job is running. Priority takes effect when two or more jobs are queued to run, and then determines when the job can run. The impact policy determines how many resources the job can consume. Most jobs are set to low impact by default. This means the job will consume a minimum amount of cluster resources.


The default settings for priority and impact policy can be configured for each job; however, Isilon recommends using the assigned defaults to minimize impact on cluster performance.

Easy Job Scheduling


The Job Engine manages over 20 cluster maintenance jobs that address file system protection, data replication, backup tasks, drive rebuilds, antivirus scanning, and more. Some jobs run on a regular schedule and a few jobs, such as FlexProtect, are automatically triggered by an event. Jobs can be manually started at any time. This mix of flexibility and automation ensures that cluster maintenance jobs can be completed when they are needed most.


If you need to start a job before it is scheduled to run, or restart a job that has failed, you can easily start, stop, pause, or resume a job from either the OneFS web administration interface or the OneFS command-line interface. For more information, see the OneFS administration guides on the EMC Online Support site. Or watch the video, How to manually start a job in OneFS 6.0 – 7.0.

Running Multiple Jobs at the Same Time


In OneFS 7.0 and earlier, only one Job Engine job can run at a time. If, for example, a drive fails while a SnapshotDelete job running, a FlexProtect job will automatically start to rebuild data on another drive. Because the FlexProtect job has a higher priority, the SnapshotDelete job will be stopped and queued. You will have to wait for the FlexProtect job to complete before restarting the SnapshotDelete job.


In OneFS 7.1, up to three jobs can run at the same time. In the same example, the SnapshotDelete job can continue to run as the FlexProtect job starts. This saves time and enables routine cluster maintenance to stay on schedule.


Some classes of jobs are not allowed to run at the same time if they belong to the same exclusion set, meaning that they perform similar functions such as restriping or marking files. To learn more about exclusion sets or Job Engine enhancements in OneFS 7.1, see the OneFS administration guides or the EMC Isilon OneFS Job Engine white paper.