We are pleased to announce the certification of Dell EMC Isilon version OneFS 8.2 with HDP 3.1 and CDH 6.3.1 via the Cloudera QATS program! See the Announcement.
Since the beginning of our respective partnerships with Dell EMC as legacy companies in 2015, both Cloudera and Hortonworks delivered differentiated consolidated data lake solutions to our joint install base of over a hundred customers running Hortonworks Data Platform (HDP) and Cloudera Distribution of Hadoop (CDH) in a shared storage configuration with Dell EMC Isilon.
Dell EMC Isilon scale-out Network Attached Storage (NAS) has the ability to run HDFS natively and incorporates critical components of the HDFS software stack such as the name-node and data-node inside the OneFS software. For Hadoop analytics, Isilon’s architecture minimizes bottlenecks, rapidly serves petabyte scale data sets and optimizes performance.
Cloudera’s new streamlined Quality Assurance Test Suite (QATS) certification process is designed to validate HDP and CDH on a variety of Cloud, Storage & Compute Platforms. The validation and certification of Dell EMC’s Isilon storage solution is enabling us to deepen and expand our partnership.
What is the Cloudera QATS Program?
The QATS program is Cloudera's’ highest certification level, with rigorous testing across the full breadth of HDP and CDH services. In this case it focused on testing all the services running with HDP 3.1 and CDH 6.3.1, and it validated the features and functions of the HDP and CDH cluster.
QATS is a product integration certification program designed to rigorously test Software, File System, Next-Gen Hardware and Containers with HDP and CDH. With dedicated Cloudera engineering resources to continuously and thoroughly test each release of HDP, QATS ensures that solutions are validated for a comprehensive suite of use cases and deliver high performance, under rigorous loads.
QATS functional testing was performed on the latest HDP 3.1 and CDH 6.3.1 versions with Isilon 8.2. In addition, Stress, High Availability, Reliability, Performance, Operational Readiness, and Integration Testing were executed to ensure all components accommodate system demands.
This new certification enables Cloudera and Dell EMC to provide best-in-class support to our customers. Our teams will continue to collaborate at various levels – across engineering R&D as well as our go-to-market plans – to ensure continuous product compatibility and high-quality customer service. And we will continue to run QATS tests against future product versions including the on premise version of CDP when it is made available by Cloudera.
QATS Test Coverage
The QATS certification covers an extensive range of tests including:
- Functional testing of all components including MR, YARN, Hive, Hive with LLAP, HBase, and Spark with Kerberos security
- Integration tests to ensure that all services are configured, integrated, and are working well with each other
- Stress testing of different components that are used frequently
- Performance testing and validation
- Concurrency and multi-tenancy testing of Hive, Spark, HBase, and other highly utilized services
- Operational readiness testing to ensure alerts, monitoring, auditing, and more
What does this mean for our joint customers?
QATS certification on Isilon provides existing customers with better support and reduced risk. Clusters that use CDH/HDP in conjunction with Cloudera Certified Technologies operate with lower risk and lower total cost of ownership (TCO). Cloudera Certified Technologies have been tested and validated to use supported APIs and to comply with Cloudera development guidelines for integration with Hadoop.
Certification includes the following components:
- Hive LLAP
Specific Features within HDFS that are not support with Isilon HDFS:
- HDFS Extended ACLs
- HDFS Cache (Isilon has internal caching within OneFS)
- Erasure Coding - HDP getErasureCodingPolicy no supported. Note: OneFS has Erasure Coding natively in OneFS
- fsck NameNode commands
- HDFS NFS Support
- Trash Behaviour
- Distcp with preserve checksum (-pc) option. Skip CRC checks when using Isilon, Isilon maintains it's own integrity checking internally in OneFS.
- HDFS Snapshot
- HDFS ShortcircuitReads
- HDFS Quota (configure Quotas in OneFS instead).
Other features not support with Isilon:
- Ranger Audit & TAG based Authorization Policies
Dell EMC Isilon 8.2 has now been certified with HDP 3.1 and CDH 6.3.1. The solution will be jointly supported by Cloudera and Dell EMC. We have a joint support process in place that involves triaging any issue that occurs with our solution, regardless of where it is discovered and directing issues to the appropriate teams, either in Cloudera or Dell EMC.
Cloudera and Dell EMC will also develop a joint reference architecture and solutions around Hadoop Tiered Storage. Hadoop Tiered Storage will enable customers to use existing Direct Attached Storage (DAS) clusters for hot data and Isilon for cold data within the same logical Hadoop cluster to simultaneously deliver extreme performance and economic scaling.
Note: Certification of Dell ECS as the S3 object store using QATS for both CDH and HDP is underway.