ECS: Bucket Management - Overview and troubleshooting

           

   Article Number:     538941                                   Article Version: 3     Article Type:    Break Fix 
   

 


Product:

 

ECS Appliance,ECS Software

 

Issue:

 

 

This article describes buckets, bucket management and basic troubleshooting tips.   
   
     
                                                           

 

 

Resolution:

 

 

   

      Bucket Management:   

   

     
      What is a bucket?   

    Buckets are containers for objects created in a namespace and sometimes considered a logical container for sub-tenants. In S3, containers are called buckets and this term has been adopted by ECS. In Atmos, the equivalent of a bucket is a subtenant; in Swift, the equivalent of a bucket is a container, and for CAS, a bucket is a CAS pool. Buckets are global resources in ECS. Each bucket is created in a namespace and each namespace is created in a Replication Group.   
    Buckets are global resources that can span multiple sites. Bucket creation involves assigning it to a namespace and a Replication Group. The bucket level is where ownership and file or CAS access is enabled.   
   
    Buckets can be accessed via different tools same time, i.e. access the same bucket with GeoDrive and S3Browser.   

     
      What bucket options are supported?   

Multi-protocol support:   

      S3, Dell EMC Atmos, Swift, Fileaccess via NFSv3, Fileaccess to CIFS via "GeoDrive" Application     
     
      ACL:     
      S3 allows to set the following permissions:     
          

                                                                                                                                                                                                                                                             
ReadAllows user to list the objects in the bucket.
Read ACLAllows user to read the bucket ACL.
WriteAllows user to create or update any object in the bucket.
Write ACLAllows user to write the ACL for the bucket.
ExecuteSets the execute permission when accessed as a file system. This permission has no effect when the object is accessed using the ECS object protocols.
Full ControlAllows user to Read, Write, Read ACL, and Write ACL.           
               Note:  Non-owners can Read, Write, Read ACL, and Write ACL if the permission has been granted or can only list the objects.
Privileged WriteAllows user to perform writes to a bucket or object when the user does not have normal write permission. Required for CAS buckets.
DeleteAllows user to delete buckets and objects. Required for CAS buckets.
NoneUser has no privileges on the bucket.
   

      Multiple users can be created with different permissions. Refer to the ECS HELP in ECS UI -  Administration Guide - "Home" - "Buckets" -  "Working with buckets in the ECS Portal"     
      For CAS refer to ECS: Set up bucket ACLs for CAS     
     
      CAS:     
      This must be set when creating the bucket. CAS buckets cant be accessed with other protocols and the CAS option must be set when creating a bucket.     
      Refer to: ECS: CAS User & Bucket Administration Best Practices     
     
      Filesystem:     
      Required for NFSv3. Required on bucket creation, cant be altered later.     
     
      Quota:     
      Soft and Hard Quota can be assigned on either, bucket or namespace level.     
      ECS: Storage Efficiency     
     
      Retention:     
      Retention can be set on bucket or namespace level. Retention applies to CAS and S3 buckets.     
      Refer also to: ECS: CAS retention in ECS     
     
      Versioning:     
      S3 allows you to create versions of objects when they are updated or deleted.     
      Refer to the Data Access Guide for details     
     
      Lifecycle:     
      Lifecycle policies can be uploaded via s3curl or using the S3browser.     
      Refer to the Data Access Guide for details     
     
      Access during outage (ADO):     
      ECS allows you to configure buckets to be accessible if one of your VDC fails.     
          

   

      How to create a bucket?   

    ECS offers several ways to create a bucket, but the workflow must be followed in the right order.   
         
  1.         Upload an ECS license.     
  2.      
  3.         Select a set of nodes to create at least one storage pool.     
  4.      
  5.         Create a VDC.     
  6.      
  7.         Create at least one replication group.     
  8.      
  9.         Create at least one namespace. A namespace is the equivalent of a tenant.     
  10.      
  11.         Create one or more buckets.     
  12.    
    This can be done via UI or Rest API.   
    As the exact commands may vary between different releases, refer to the appropriate ECS documentation for your current release.   
   
    UI:   
    Log In to the ECS system using your browser and select the HELP. From there open the Administration Guide. Find the appropriate steps in "Getting started" - "Initial configuration".   
   
    Rest API:   
    List the ECS REST API references   
        

      Considerations:   

    Before creating a buckets remember that some options can not be changed later:   
         
  •         Replication Group     
  •      
  •         Server-side Encryption     
  •      
  •         File System     
  •      
  •         CAS     
  •      
  •         Metadata Search     
  •    
   
      Make sure the buckets are configured properly, else a migration might be required.     
     
      The following rules apply to the naming of S3 buckets in ECS:   
   
         
  •         Must be between one and 255 characters in length. (S3 requires bucket names to be 1–255 characters long)     
  •      
  •         Can include dot (.), hyphen (-), and underscore (_) characters and alphanumeric characters ([a-zA-Z0-9])     
  •      
  •         Can start with a hyphen (-) or alphanumeric character.     
  •      
  •         Cannot start with a dot (.)     
  •      
  •         Cannot contain a double dot (..)     
  •      
  •         Cannot end with a dot (.)     
  •      
  •         Must not be formatted as IPv4 address.     
  •      
  •         Bucket names must be unique within a namespace     
  •      
  •         Namespace and bucket names should be DNS compatible since they can appear in a DNS record.     
  •    
    Bucket best practices:   
         
  •         Use buckets for specific environment, workflow, or uses. For instance: dev, test, finance, operations, etc.     
  •      
  •         In multi-site deployments, create buckets at the VDC site closest to the application accessing and updating the objects. There is overhead involved with checking the latest copy if the ownership of object is at a remote site.     
  •      
  •         For best performance, recommended to have less than 1000 buckets in a single namespace     
  •    
   
    Additional informations can be found in the Administration guides and whitepapers:   
    From ECS help: Log In to the ECS system using your browser and select the help (questionmark top right side).   
    ECS Overview and Architecture   
    ECS Admistration Guide   
    ECS Data Access Guide   
   
        

      How to test buckets?   

    There are various ways to test S3 connectivity, these are a few examples for accessing them. Simplest method to access a S3 bucket via S3browser.   
    Other methods are:                                                                                                                           
WinSCP:  ECS: Connecting ECS buckets with WINSCP
s3test.sh script: ECS: How to test S3 create bucket, upload object, read object, delete object, delete bucket with the s3test.sh script
S3curl:    ECS: How to perform basic s3 operations on ECS 3.x using the s3curl.pl script.
S3Browser:  ECS : How to connect ECS with S3Browser?
        

      What are the bucket limitiations?   

    Cloud Scale - ECS is an object storage platform for both traditional and next-gen workloads. ECS’s software-defined layered architecture promotes limitless scalability.   
    Feature highlights are:   
    - Globally distributed object infrastructure   
    - Exabyte+ scale without limits on storage pool, cluster or federated environment capacity   
    - No limits exist on the number of objects in a system, namespace or bucket   
    - Efficient at both small and large file workloads with no limits to object size   
   
    Therefore buckets are just limited by the maximum capacity configured to the system the bucket is configured on.   
    Buckets are multi-protocol enabled, except CAS buckets. CAS buckets cannot be shared.   
   
    Behavior During Site Outage Temporary site outage (TSO):   
    Client access and API-operation availability at the namespace, bucket and object levels during a TSO is governed the following ADO options set at the namespace and bucket level:   
    • Off (default) - Strong consistency is maintained during a temporary outage.   
    • On - Eventually consistent access is allowed during a temporary site outage.   
    Data consistency during a TSO is implemented at the bucket level. Configuration is set at the namespace level, which sets the default ADO setting in place for ADO during new bucket creation. and can be overridden at new bucket creation; meaning TSO can be configured for some buckets and not for others.   
        

      How to delete a bucket?   

    To delete a bucket, make sure the bucket is entirely empty. This not only includes current objects, also, if configured, versions and delete markers need to be deleted before you can remove that bucket from.   
    If it is not possible to delete the bucket manually, a lifecycle policy can be set on the bucket to delete it.   
    Refer to:   
    538501 - ECS: How to delete a S3 bucket with many objects in it using lifecycle policies   
   
        

      Troubleshooting:   

    Check the application for an error code. Further information can be found here. ECS: S3 error codes with possible causes   
    On capacity issues check that the ECS still has free capacity and no quota was breached. Once a quota is reached.   
    CREATE QUOTA KBA FOR QUOTA FULL.   
    ECS: How to update Bucket Quota by management API   
   
   
    If a bucket is unavailable during a single VDC outage:   
    ECS: The importance of the ADO (Access During Outage) parameter   
    By default, ADO is not enabled, and strong consistency is maintained. All client API requests where authoritative namespace, bucket or object data is required but temporarily unavailable will fail. Object operations of read, create, update and delete as well as list buckets not owned by an online site, will fail. Also, operations of create and edit of bucket, user and namespace will also fail. The initial site owner of bucket, namespace and an object, is the site where the resource was first created. During a TSO, certain operations may fail if the site owner of resource is not accessible.   
    Highlights of operations permitted or not permitted during a temporary site outage include:   
    • Creation, deletion, and update of buckets, namespaces, object users, authentication providers, RGs and NFS user and group mappings are not allowed from any site.   
    • Listing buckets within a namespace is allowed if the namespace owner site is available.   
    Bring the VDC online again or change the ADO parameter to enable access.   
   
   
    ACL may be set incorrectly. Make sure proper user and permissions are set if getting permission denied or insufficient permission responses from the ECS system.   
    ECS: Blank Custom Group ACL - Bucket created using create_subtenant.sh - Atmos   
    ECS: Bucket and Object ACL for a non-owner object user in S3 Browser   
   
    S3Browser does not list the bucket, even the user has the permissions assigned:   
    ECS: Bucket and Object ACL for a non-owner object user in S3Browser   
   
   
                                                                

 

 

Notes: