NetWorker, Avamar, SHA-1 Certificates, and You
As you may be aware, the major browser vendors are gradually (or not so gradually) sunsetting support for SSL certificates signed using the SHA-1 hashing algorithm. This is coming very, very soon. As you may also be aware, Dell EMC released a technical advisory because certain NetWorker and Avamar components use certificates signed using SHA-1:
ETA 493820: Avamar, NetWorker: Browser support for SHA-1 Certificates expiring January 1, 2017 may cause incompatibility with Avamar and NetWorker Virtual Edition Browser UI functionality
If you use Avamar or NetWorker, you should review this ETA since this may impact operations in your environment
Note: While I do work for Dell EMC, this post is not an official Dell EMC document. The ETA document is the official Dell EMC response to this issue. Any information provided here is provided as-is by me personally and should only be used at your own risk.
Frequently Asked Questions
Q: Will this impact my backups?
Q: What types of Avamar nodes are affected by this issue? Does this issue affect both physical Avamar nodes and Avamar Virtual Edition?
A: This issue affects physical Avamar nodes of all supported hardware configurations as well as Avamar Virtual Edition. The issue affects services running on the Utility Node (or Single Node for Single Node Servers).
Q: Is the remote access hardware (DRAC, RMM, etc.) on physical Avamar Servers affected by this issue?
A: For Gen4 and Gen4S, yes. You can use http instead of https as a workaround. The tools and procedures for replacing these certificates need to be tidied up before they can be made available for customer use. This effort is ongoing. The SSL certificate used for the remote access interface on Gen4T hardware uses a stronger signature algorithm, so Gen4T is not affected by this issue.
Q: What types of NetWorker servers are affected by this issue? Does this issue affect physical NetWorker servers, NetWorker servers installed in virtual environments, or the NetWorker Virtual Edition (NVE) appliance?
A: This issue affects only the NetWorker Virtual Edition (NVE) appliance. The issue does not affect physical NetWorker servers or virtual NetWorker servers where the NetWorker software has been installed on a "beige box" Windows or Linux virtual machine.
Q: Is the NetWorker Virtual Backup Appliance (VBA) affected?
Q: Is any other NetWorker software affected?
A: No. All other NetWorker certificates use SHA-256 or SHA-512 signatures.
Q: Exactly what problem(s) will this issue cause?
A: Starting with Chrome 56, the Chrome browser can no longer access certain browser-based interfaces on the Avamar Server, AVE, NVE, or VBA. For other browsers, an additional warning message will be displayed indicating that the server's certificate is using a weak signature algorithm. If you browse to https://myserver.example.com to access the "Documents and Downloads" page, you will receive a certificate error. Depending on the URL used to access certain features, there may also be issues accessing Avamar Installer, DTLT, or other browser-based services.
Q: What Avamar software services are affected?
A: The Apache Web Server and any interfaces that use it are affected. That means only browser-based services like the Documents and Downloads page, DTLT, Avamar Client Manager, Proxy Deployment Manager, etc. will be affected. The Avamar Extended Retention (AER) GUI is affected.
Q: Are the Avamar Administrator Server (MCS) or Avamar Administrator GUI (MC-GUI) affected?
A: No. While the MCS does use a SHA-1 certificate, these services do not use a browser engine and are therefore unaffected by this issue. The procedure for replacing this certificate is in development.
Q: How do I fix it?
A: The affected certificates will need to be replaced. See KB 493774 for the instructions to replace the Apache Web Server certificate and KB 467848 for the instructions to replace the AER GUI certificate. Ideally, these certificates should be replaced with certificates signed by an internal or external certificate authority (CA) but there are also instructions in the KB for regenerating the certificate as a self-signed certificate using SHA-256.
Q: One of the commands in the KB failed.
Q: I completed the procedure in KB 493774. Why does my certificate still say it's using SHA-1?
A: Make sure you're looking at the signature algorithm, not the certificate fingerprints. It's the signature algorithm that matters.
Q: Can I contact support about this?
A: Yes but please don't. The instructions for checking and replacing the certificates are fairly straightforward and we're expecting this issue will probably put a strain on the support team as it is.
Q: Can I use SHA-512 instead?
A: Go for it. In the openssl command for signing the certificate, replace the -sha256 flag with -sha512 instead.
Q: Does this issue affect the security of the system?
A: Not yet. While SHA-1 has been broken (see the "Shattered" attack), it would still take significant time (days or months) and resources (potentially six figure dollar amounts) to generate a collision. It's time to walk to the exits in an orderly fashion (by replacing any SHA-1 certificates still around with more secure SHA-2 or SHA-3 certificates), not time to panic.
Entirely apart from the weakness of SHA-1 certificates, if you're still using the self-signed certificates that ship with these systems, worrying about the SHA-1 signatures is akin to worrying that the picture on your fake ID might be fake. Self-signed certificates provide no assurance of identity. If you really want to make sure your system is secure, you must use certificates signed by a certificate authority.
Q: I heard Avamar uses SHA-1 internally. Is that true? Isn't that a security risk?
A: The Avamar de-dupe and storage mechanisms use SHA-1 hashes internally. However, these hashes are not used in a security-sensitive context. Avamar also has a number of mechanisms in place to detect SHA-1 collisions and help protect the integrity of the data even in the event of a collision. You're much more likely to be affected by disk errors than SHA-1 collisions in the de-dupe engine.
Q: The Webkit svn repository fell over when somebody checked the "Shattered" PDFs into it. Would backing up these two PDFs cause an issue on the Avamar server?
A: No. Avamar uses sub-file hashing when backing up and restoring files. The Avamar architecture is resilient against this type of issue since we don't use the SHA-1 hash of the whole file to uniquely identify it. Since there's sometimes a difference between theory and practice, I also specifically tested this scenario in the lab. The two PDFs backed up and restored without issue and there was no impact to the Avamar server.
Q: I have another question.