I dont think this tool allows to apply tree quota on an existing directory.
the release notes says once enabled:
Online quota update tool
Once a tree, user, or group quota is enabled on a directory, in some
instances, differences may occur between a reported quota usage and the