Data Domain: Retention Lock (RL) frequently asked questions (FAQ)

           

   Article Number:     492081                                   Article Version: 3     Article Type:    Break Fix 
   

 


Product:

 

Data Domain,Data Domain Retention Lock

 

Issue:

 

 

This article provides a concise overview of Data Domain Retention Lock functionality and answers to related 'frequently asked questions'   
     
                                                           

 

 

Resolution:

 

 

What is retention lock?   
   
    Retention lock is functionality which can be used on Data Domain Restorers (DDRs) to prevent modification and/or deletion of a certain set of files for a pre-determined period of time (i.e. retention locked files are read only until their retention period expires).   
   
    What are the different versions/flavours of retention lock?   
   
    Retention lock functionality is available in two different flavours:   

         
  •         Governance: The less strict of the two retention lock flavours (i.e. locks against files can be reverted if necessary)     
  •      
  •         Compliance: The stricter of the two flavours which adheres to a number of common regulatory standards (i.e. locks against files cannot be reverted, the DDR must be configured with a 'security officer' user who must authenticate certain commands, and there are various restrictions on other functionality to prevent locked data from being removed/locks being reverted early)     
  •    
    Note that:   
         
  •         Compliance mode is only available in DDOS 5.3 (and later)     
  •      
  •         Each flavour of retention lock requires a separate license key     
  •      
  •         Retention lock functionality is enabled on a per mtree basis     
  •      
  •         As a result a single system can use both governance and compliance mode against separate mtrees however it would need to have separate governance and compliance licenses installed     
  •      
  •         Retention lock functionality should not be used against mtrees used to store Avamar data (as this can prevent Avamar from functioning as expected)     
  •    
Which data access protocols are supported with retention lock?   
         
  •         The NFS/CIFS/DDBoost protocols are fully supported against mtrees using retention lock governance or compliance mode     
  •      
  •         The VTL protocol is only supported against mtrees using retention lock governance mode - refer to the Data Domain Administration Guide to determine how to unlock retention locked tapes such that they can be written to     
  •    
    Note that prior to DDOS 5.5.2.0 retention lock could not be enabled for an mtree which is used as a DDBoost logical storage unit (LSU) - any attempt to do so would fail with the following error:   
   
    Retention-lock governance or compliance mode cannot be set on mtree which is exported as a ddboost storage unit.   
   
    As of DDOS 5.5.2.0 (and later), however, this is supported so the above error should no longer be seen (and retention lock should be enabled against the mtree as expected). Note that some versions of the DDOS 5.5 and 5.6 administration guides incorrectly state that this functionality is still not supported.   
   
    Which regulatory standards are met by retention lock compliance mode?   
   
    The list of regulatory standards met by retention lock compliance mode include the following:   
         
  •         SEC 17a-4(f)     
  •      
  •         CFTC Rule 1.31b     
  •      
  •         FDA 21 CFR Part 11     
  •      
  •         Sarbanes-Oxley Act     
  •      
  •         IRS 98025 and 97-22     
  •      
  •         ISO Standard 15489-1     
  •      
  •         MoREQ2010     
  •    
    For full details of certification information please contact your contracted support provider.   
   
    How is retention lock governance enabled?   
         
  •         A retention lock governance license is added to the DDR     
  •      
  •         Retention lock governance mode is enabled against any required mtree:     
  •    
   

      # mtree retention-lock enable mode governance mtree [mtree]   

How is retention lock compliance enabled?   
         
  •         A retention lock compliance license is added to the DDR     
  •      
  •         A user with role of 'security' should be created (assuming such a user does not already exist):     
  •    
   

      (ADMIN USER) # user add [username] role security   

   
         
  •         ​The user with role of 'security' should log into the DDR and enable security user authorisations:     
  •    
   

      (SECURITY USER): # authorization policy set security-officer enabled   

   
         
  •         The system should be configured for retention lock compliance mode - note that once the following command completes the system reboots automatically:     
  •    
   

      (ADMIN USER) # system retention-lock compliance configure   

   
         
  •         Once the system has rebooted retention lock compliance mode should be enabled on the system:     
  •    
   

      (ADMIN USER) # system retention-lock compliance enable   

   
         
  •         Retention lock compliance mode is enabled against any required mtree:     
  •    
   

      (ADMIN USER) # mtree retention-lock enable mode compliance mtree [mtree]   

   

      How is it possible to determine which mtrees have retention lock enabled?     
     
      Mtrees with retention lock enabled will be indicated in the output of 'mtree list', for example:     
     
      sysadmin@dd630# mtree list       
        Name                                Pre-Comp (GiB)   Status    Tenant-Unit       
        ---------------------------------   --------------   -------   -----------       
        ...       
        /data/col1/rich-retention-lock                 0.0   RW/RLGE   -       
        /data/col1/rl_test                             0.0   RW/RLGD   -       
        /data/col1/rl_test_comp                        0.0   RW/RLCE   -       
        /data/col1/test                                3.1   RW/RLGE   -       
        ...       
        ---------------------------------   --------------   -------   -----------       
        ...       
         RLGE : Retention-Lock Governance Enabled       
         RLGD : Retention-Lock Governance Disabled       
         RLCE : Retention-Lock Compliance Enabled
     
     
      How are file retention/lock periods set?     
     
      Once retention lock is enabled against an mtree a minimum and maximum retention period must be set. These periods dictate the minimum and maximum time a file within the mtree can be locked for. For example:     
     
      # mtree retention-lock set min-retention-period [period] mtree [mtree]       
        # mtree retention-lock set max-retention-period [period] mtree [mtree]
     
     
      Periods can be given in various units as follows:     
     
      1min       
        1hr       
        1day       
        1mo       
        1year
     
     
      Note that:   

   
         
  •         A minimum retention period cannot be less than 12 hours     
  •      
  •         A maximum retention period cannot be greater than 70 years     
  •      
  •         The minimum retention period must be less than the maximum retention period     
  •      
  •         Retention periods for each mtree are set in the same way regardless of the flavour of retention lock being used     
  •    
   

      How can existing retention periods be displayed?     
     
      This can be done via the following two commands:     
     
      # mtree retention-lock show min-retention-period mtree [mtree]       
        # mtree retention-lock show max-retention-period mtree [mtree]
     
     
      For example:     
     
      sysadmin@dd630# mtree retention-lock show min-retention-period mtree /data/col1/rl_test       
        Retention-lock min-retention-period of mtree /data/col1/rl_test is: 720 minutes.       
        sysadmin@dd630# mtree retention-lock show max-retention-period mtree /data/col1/rl_test       
        Retention-lock max-retention-period of mtree /data/col1/rl_test is: 30 days.
     
     
      How are files within an mtree with retention lock enabled actually locked?     
     
      Note that:   

   
         
  •         When retention lock is enabled against an mtree existing files within the mtree are *not* automatically locked (i.e. all pre-existing files remain read/write)     
  •      
  •         When a new file is written to an mtree with retention lock enabled the file is *not* automatically retention locked (i.e. the new file will remain read/write)     
  •    
   

      To retention lock a specific file the atime of the file must be modified to match the date/time until which the file should be retention locked (i.e. the date/time until which it should remain read only). Until the atime is modified in this way the file will *not* be retention locked (and can be modified/removed).     
     
      A files atime can be changed from an NFS/CIFS client using the 'touch' command, i.e.:     
     
      # touch –a –t [expiry time] [file to be locked]     
     
      For example to set atime of /data/col1/rl_test/testfile to 07:05 on 8th June:     
     
      # touch -a -t 06080705 /data/col1/rl_test/testfile     
     
      The period from current time to future atime must be within the minimum and maximum retention periods for the mtree otherwise an error will be generated when modifying the files atime:     
     
      # touch -a -t 08080705 /data/col1/rl_test/testfile       
        touch: setting times of `/data/col1/rl_test/testfile': Permission denied
     
     
      A corresponding message will also be shown in DDFS log files:     
     
      06/07 13:44:57.197 (tid 0x2b28ee5258c0): Attempt to set atime of /data/col1/rl_test/testfile to larger than maximum retention period of mtree.     
     
      CIFS (Windows) clients do not, by default, include a touch command/utility however a number of such utilities are freely available for download from various 3rd party web sites.     
     
      Note that files cannot be retention locked until they have been completely written to the DDR (i.e. it is not possible to create an empty file, retention lock that file, then write data to the file).     
     
      Which backup applications support automatically retention locking files after writing them to a DDR?     
     
      Certain archiving applications (such as EMC Source One or Symantec Enterprise Vault) can be configured to automatically modify the atime of files in this manner after writing data to a DDR. Refer to the 'EMC Data Domain Archive Application Compatibility Guide' document for a full list of such applications and to application specific documentation to determine how to enable this functionality.     
     
      Many backup applications (such as EMC Networker) do *not* support such functionality. As a result custom scripts must be developed to manually set retention periods of files written by such applications. In this scenario it should be ensured that custom scripts set files atimes such that they are unlocked prior to the backup application attempting to delete the file. Failure to do this can result in the backup application attempting to delete locked files (which fails) causing the file to be left on the DDR indefinitely consuming disk space.     
     
      What can/can't be done against a locked file?   

   
         
  •         Retention locked files are read only and therefore cannot be modified or deleted     
  •      
  •         Once a files retention period expires it is 'unlocked' - when in an unlocked state the file still cannot be modified however it can be deleted. Note that the DDR does not automatically delete a file when its retention period expires (subsequent deletion has to be performed from a client system/backup application)     
  •      
  •         Once set the retention period for a specific file cannot be reduced (i.e. the files atime cannot be brought forwards)     
  •      
  •         Retention periods can, however, be increased (atime can be increased up to the maximum retention period for the mtree)     
  •      
  •         Ownership and permission settings for a file can continue to be modified whilst the file is locked     
  •    
    Note that it is only possible to rename or delete a directory in a retention lock enabled mtree if that directory contains no files. If the directory contains files (even if these files are not currently retention locked) the rename/delete of the directory will fail.   

      Is it possible to completely remove the retention lock against a file or set of files?     
     
      It is possible to 'revert' (i.e. remove) the retention lock against files in mtrees using governance mode - this is done with the following command:     
     
      # mtree retention-lock revert [path]     
     
      Once the retention lock against a file is removed it can be modified/deleted as normal. Note that if this command is run against a directory then all files within that directory and all subdirectories will have their retention locks removed.     
     
      It is not possible to revert a retention lock against files in mtrees using compliance mode - if this is attempted a corresponding error will be displayed:     
     
      # mtree retention-lock revert /data/col1/rl_test_comp/testfile       
        This operation is not allowed. Mtree is in retention-lock compliance mode.
     
     
      What happens if a retention locked file is attempted to be modified or removed?     
     
      Any attempt to modify or delete a retention locked file will cause a corresponding 'permission denied' error to be displayed:     
     
      # echo " test2" >> /data/col1/rl_test/testfile       
        bash: testfile: Permission denied       
        # rm testfile       
        rm: remove write-protected regular file `testfile'? y       
        rm: cannot remove `testfile': Permission denied
     
     
      DDFS logs will indicate that the operation has failed due to the file being retention locked:     
     
      06/07 07:06:59.756 (tid 0x2b5a77605d50): Atime of retention-lock file /data/col1/rl_test/testfile is not expired.       
        06/07 07:07:42.504 (tid 0x2b5a79111390): Atime of retention-lock file /data/col1/rl_test/testfile is not expired.
     
     
      Is it possible to list all files which are currently retention locked?     
     
      Yes - this can be performed via the 'mtree retention-lock report generate retention-details' command, i.e.:     
     
        mtree retention-lock report generate retention-details       
                        mtrees {<mtree-list> | all}       
                        [format {text | tsv | csv}]       
                        [output-file <file-name>]       
                                               Report detailed information of       
                                               retention-lock files.
     
     
      For example to list details of all locked files in the /data/col1/rl_test mtree the following would be performed:     
     
      sysadmin@dd890# mtree retention-lock report generate retention-details mtrees /data/col1/jftest       
        Report generated on: Fri Jul  1 14:19:31 2016       
       
        Report for mtree: /data/col1/jftest       
        File Path        Mode        Size(Bytes)        Expiration Date       
        /data/col1/jftest/file1 governance 10521456 Sat Jul  2 22:35:48 2016       
        /data/col1/jftest/testdir/file2 governance 10521680 Sat Jul  2 22:35:42 2016       
        /data/col1/jftest/file3 governance 10521820 Sun Jul 10 22:36:09 2016       
        Total files: 3
     
     
      Note that this command is only available on DDOS 5.5 (and later).   

   
      Is it possible to completely disable retention lock for an mtree (after it has been enabled)?     
     
      Yes - for mtrees using governance mode this is performed via the 'mtree retention-lock disable' command, i.e.:     
     
      # mtree retention-lock disable mtree [mtree]     
     
      For example:     
     
      sysadmin@dd630# mtree retention-lock disable mtree /data/col1/rl_test       
        Retention-lock feature is disabled (previously enabled) for mtree /data/col1/rl_test.
     
     
      Once disabled mtree list will indicate that retention lock was used against the mtree but has since been disabled, i.e.:     
     
      sysadmin@dd630# mtree list       
        Name                                Pre-Comp (GiB)   Status    Tenant-Unit       
        ---------------------------------   --------------   -------   -----------       
        ...       
        /data/col1/rl_test                             0.0   RW/RLGD   -       
        ...
     
     
      Note that once retention lock is disabled against an mtree:   
   
         
  •         New files written to the mtree cannot be retention locked     
  •      
  •         Files which are already locked stay locked for their previously defined retention period (i.e. all locks are not automatically reverted when retention lock is disabled against an mtree)     
  •      
  •         Existing locks against files in an mtree where retention lock has been disabled cannot be reverted (so all necessary reversions must be done prior to retention lock being disabled):     
  •    
   
      sysadmin@dd630# mtree retention-lock revert /data/col1/rl_test/testfile       
        **** Retention-lock feature is disabled (previously enabled) for mtree which contains the path /data/col1/rl_test/testfile.
   
   
         
  •         Retention lock cannot be disabled for an mtree using compliance mode:     
  •    
   
      sysadmin@dd630# mtree retention-lock disable mtree /data/col1/rl_test_comp       
        **** Operation is not allowed because the system is a retention-lock compliance system
   
   
     
      Can replication still be used against mtrees which have retention lock enabled?     
     
      Yes - retention locked mtrees/files can be replicated using various replication topologies:   
   
         
  •         Directory replication - only supported for files using governance mode - does not replicate minimum and maximum retention periods to the destination system     
  •      
  •         Mtree replication - can be used for either governance or compliance mode data and does replicate minimum and maximum retention periods to the destination system     
  •      
  •         Collection replication - can be used for either governance or compliance mode data and does replicate minimum and maximum retention periods to the destination system     
  •    
   
      Note that for retention locks to be preserved on destination systems:   
   
         
  •         Both the source and destination systems must have corresponding retention lock licenses installed     
  •      
  •         Mtree replication contexts must have 'Replication propagate-retention-lock' set to Enabled     
  •      
  •         For files replicated by directory replication the corresponding mtrees minimum and maximum retention periods must be manually set on the destination system     
  •    
   
      Are there any other restrictions when replicating retention locked files?     
     
      Yes - resync's of replication contexts (i.e. trying to re-establish a previously configured but broken context) where retention locked data is used can fail. Note that:   
   
         
  •         If mtree replication is used and the destination mtree contains retention locked files which do not exist on the source a resync will fail     
  •      
  •         If directory replication is used and the destination has retention lock enabled but the source does not then a resync will fail     
  •    
   
      Note that when using mtree replication a resync will succeed in the following scenarios as long as the destination mtree does not contain retention locked files not present on the source DDR:   
   
         
  •         The source mtree does not have retention lock enabled but the destination does     
  •      
  •         The destination mtree does not have retention lock enabled but the source does     
  •    
   
      It is also not possible to enable retention lock compliance mode on an mtree which is already a member of an mtree replication context. In this scenario:   
   
         
  •         The mtree replication context should be broken on both source and destination systems:     
  •    
   
      # replication break mtree://[destination system]/data/col1/[mtree]   
   
         
  •         A new mtree replication context should be created on both source and destination systems:     
  •    
   
      # replication add source mtree://[source system]/data/col1/[mtree] destination mtree://[destination system/data/col1/[mtree]   
   
         
  •         Retention lock compliance mode should be enabled on the source system:     
  •    
   
      # mtree retention-lock enable mode compliance mtree [mtree]   
   
         
  •         The newly created replication context should be resync'd on the source system:     
  •    
   
      # replication resync mtree://[destination system/data/col1/[mtree]   
   
     
      Can retention locked files be fastcopied?     
     
      Yes files which are retention locked can be fastcopied as normal. Note that if the destination mtree holding the fastcopy is retention lock enabled then the retention lock of the file will be preserved against the fastcopy. If the destination mtree is not retention lock enabled then the fascopy will not be retention locked.     
     
      Are there any other restrictions in system functionality when using retention lock?     
     
      The following commands are disallowed on systems using retention lock compliance mode:     
     
      # user reset       
        # filesys destroy       
        # filesys archive unit del [archive unit name]
     
     
      Such systems also cannot be booted to single user mode for recovery by technical support without use of a USB key and physical access to the system     
     
      The following commands are disallowed against mtrees using retention lock compliance mode:     
     
      # mtree delete [mtree]       
        # mtree retention-lock reset [min-retention-period period | max-retention-period period] mtree [mtree]       
        # mtree retention-lock disable mtree [mtree]       
        # mtree retention-lock revert
     
     
      Note that mtrees using retention lock governance mode (or where this mode was once enabled) can only be deleted once the mtree contains no files - if there are any files remaining in the mtree an error will be returned     
     
      Is the system clock important on retention lock enabled systems?     
     
      Yes - retention lock compliance enabled systems enable an internal 'security clock' to prevent malicious tampering with the system clock (which could allow retention locked files to be deleted early). The times of the security and system clocks are regularly compared and if there is an accumulated 2 week skew between the two in a single calendar year the Data Domain File System (DDFS) will be automatically disabled to prevent access to data on the DDR. This can also happen if the system clock is suddenly modified and time is changed by more than 2 weeks.     
     
      In this scenario DDFS can be re-enabled by:   
   
         
  •         Logging into the DDR     
  •      
  •         Checking that the system clock is set correctly     
  •      
  •         Enabling the file system: # filesys enable     
  •      
  •         When prompted enter details of a user with role of security to allow the security clock to be reset and DDFS enable     
  •    
What steps can be taken if a DDR using retention locked files fills to capacity?   
   
    Assuming there is no 'cleanable' space on the DDR (i.e. clean has been run but the system remains filled to capacity) its contents should be reviewed to determine whether:   
         
  •         There are any files which are not retention locked which can be removed     
  •      
  •         There are any files locked via governance mode which can have their locks reverted and removed     
  •    
    Once this has been done clean should be run again to physically free space on the system. Alternatively if no physical data can be deleted from the system additional physical storage should be added to the DDR and the file system expanded (assuming expansion is supported by the current DDR/configuration).   
   
    If the only files on the system are locked using compliance mode it is not possible to revert locks and remove these files. As a result space will not be able to be freed unless:   
         
  •         The retention period is reached for some/all of the files - after this point they can be deleted and clean run (as described above)     
  •      
  •         The system is completely reinstalled from USB key (which will cause loss of all data on the DDR)     
  •      
  •         Additional physical storage is added to the system (as described above)     
  •    
    Note that it is entirely possible to completely fill a DDR with files locked via compliance mode. In this scenario there is nothing an administrator/technical support can do to free space (i.e. there is no low level functionality to remove/revert compliance mode locks and delete corresponding files early).