5 Replies Latest reply: Jan 9, 2018 11:30 AM by Peter Serocka RSS

Isilon 36000x / directory full

ismail123

In model 36000x the / directory is full and its showing more than 100%, I have completely checked the files inside it, all are config files no user files are there. I do see an KB article stating "On older Isilon IQ X and NL-series nodes, the / directory has a maximum size of 500 MB and typically shows 95% or greater used capacity even when there are not any extra user files or firmware packages installed. Newer nodes like the X200 and S200 have / directories with a maximum size of 1 GB and typically show 49% used capacity.  " but the issue here is the node is getting crashed sometime due to this.

 

 

Is there any workaround to fix this temporarily or is there a way to increase the / directory space?

 

Experts please advice...

 

 

 

ISILON-2# isi_for_array -s 'df -h /'

ISILON-1: Filesystem Size    Used   Avail Capacity  Mounted on

ISILON-1: /dev/imdd0a 496M    483M    -27M   106%    /

ISILON-2: Filesystem Size    Used   Avail Capacity  Mounted on

ISILON-2: /dev/imdd0a 496M    481M    -24M   105%    /

ISILON-3: Filesystem Size    Used   Avail Capacity  Mounted on

ISILON-3: /dev/imdd0a 496M    480M    -24M   105%    /

ISILON-4: Filesystem Size    Used   Avail Capacity  Mounted on

ISILON-4: /dev/imdd0a 496M    477M    -20M   104%    /

ISILON-5: Filesystem Size    Used   Avail Capacity  Mounted on

ISILON-5: /dev/imdd0a 496M    479M    -22M   105%    /

ISILON-6: Filesystem Size    Used   Avail Capacity  Mounted on

ISILON-6: /dev/imdd0a 496M    473M    -16M   104%    /

ISILON-7: Filesystem Size    Used   Avail Capacity  Mounted on

ISILON-7: /dev/imdd0a 496M    479M    -22M   105%    /

 

 

Regards,

Ismail

  • 1. Re: Isilon 36000x / directory full
    Chris Klosterman

    Obviously these nodes are no longer under active support, right?  So what I would suggest is to look and see what if anything you can clean up.

     

    isilon-1# cd /var/

    isilon-1# du -sm * | sort -n -k1 | tail -5

    1 yp

    10 backups

    16 db

    43 crash

    86 log

    Crash can of course be cleaned up, especially if there are old core dumps in it that you no longer need.  Older log files get rolled over in /var/log and compressed with .gz.  Perhaps delete those. 

     

    Look in /root and /tmp as well, sometimes people put upgrade or patch files in one of those locations and forget to clean up after themselves.

     

    Yes seeing it over 100% is definitely an area for concern.  I just hope this cluster is no longer is production (again assuming that the nodes are no longer under active maintenance).  If they are, then certainly open an SR, and have support help you do this.

     

    ~Chris

  • 2. Re: Isilon 36000x / directory full
    ismail123

    Hi Chris,

     

    Thanks for your reply.

    The thing is I saw too many.gz old log files and I moved them to ifs folders but then also the partition shows full.The /var/log/ has used only 33M then which folder is holding other space?

     

    It says /dev/imdd0a this folder is holding more data. Is there any other folders might have caused more space inside / directory?

    I checked /var/crash also that doesn't have more data.

     

    Please advise me where to check for unwanted files.. I checked for patch files and user files nothing found.

     

     

    ISILON-2# df -h

    Filesystem     Size    Used   Avail Capacity  Mounted on

    /dev/imdd0a    496M    481M    -24M   105%    /

    devfs          1.0K    1.0K      0B   100%    /dev

    /dev/imdd1a    496M     33M    424M     7%    /var

    /dev/imdd2a    496M    297K    456M     0%    /var/crash

    OneFS          227T    182T     43T    81%    /ifs

     

    Regards,

    Ismail

  • 3. Re: Isilon 36000x / directory full
    Peter Serocka

    Yes, one would focus only on the "/" filesystem mounted on /dev/imdd0a

     

    Check the how many MBs are used for regular, visible files:

     

    du -shx /

    (the "x" option makes du report only for "/" in this case, i.e. omitting /var and /var/crash).

     

    Does this yield a number significantly *lower" than the usage reported by "df -h /"  ? 

    That df number was 481 MB in your earlier result.

     

    If the du number is much lower than the df number, it usually means tha

    files have been deleted but are still "open" by some process(es). Unless

    we have a clue which process(es) are responsible and then could be restarted,

    the simplest and most certain way to a resolution is rebooting each node in question.

     

    If the du number is about the same as the df number,

    we need to inspect the file tre under "/" (excluding /var/ and /var/crash)

    in more detail. Use

     

    du -xm  -d 3 / |  sort -n

     

    The output will be all in MBs, so that the sorting makes sense.

    Note the usage numbers for higher level directories are inclusive,

    so usage is reported multiple times (usage for /usr/local is also in /usr, which again is in /).

     

    Increase the reporting depth from 3 to 4 or higher for more insights.

    Of course it would be good to have for reference one node or cluster

    where "/" isn't filled up...

     

    hth

    -- Peter

  • 4. Re: Isilon 36000x / directory full
    ismail123

    Hi Peter,

     

    Thanks for your reply.

     

    You are correct, I found /usr/local holding around 259MB of files, but not sure which are config files and which are unwanted files.

    Below is the o/p could you please help me on this to reduce some space.

     

    Apologies for the legnthy o/p. could'nt able to attach.

     

    ISILON-2# du -xm  -d 3 / |  sort -n

    1       /boot/defaults

    1       /boot/firmware

    1       /boot/kernel

    1       /boot/modules

    1       /boot/zfs

    1       /compadmin

    1       /dev

    1       /etc/X11

    1       /etc/bluetooth

    1       /etc/defaults

    1       /etc/ganglia

    1       /etc/gnats

    1       /etc/gss

    1       /etc/ifs

    1       /etc/ifs/hw

    1       /etc/ifs/licensing

    1       /etc/ifs/log

    1       /etc/ifs/patchmgr

    1       /etc/ifs/scripts

    1       /etc/isdn

    1       /etc/lcd

    1       /etc/lcd/InvertedSmall

    1       /etc/lcd/SmallFilled

    1       /etc/local

    1       /etc/mail

    1       /etc/mcp

    1       /etc/mcp/defaults

    1       /etc/mcp/override

    1       /etc/mcp/scripts

    1       /etc/mcp/sync

    1       /etc/mcp/sys

    1       /etc/mcp/templates

    1       /etc/mtree

    1       /etc/ntp

    1       /etc/openldap

    1       /etc/pam.d

    1       /etc/periodic

    1       /etc/periodic/daily

    1       /etc/periodic/monthly

    1       /etc/periodic/security

    1       /etc/periodic/weekly

    1       /etc/ppp

    1       /etc/rc.d

    1       /etc/rc.d_disabled

    1       /etc/security

    1       /etc/skel

    1       /etc/ssh

    1       /etc/ssl

    1       /etc/xinetd.d

    1       /etc/zfs

    1       /ifs

    1       /keystore

    1       /lib/geom

    1       /libexec

    1       /media

    1       /mnt

    1       /opt

    1       /proc

    1       /rescue

    1       /root

    1       /root/.ccache

    1       /root/.makekit

    1       /root/.snmp

    1       /root/.ssh

    1       /tmp

    1       /tmp/.ICE-unix

    1       /tmp/.X11-unix

    1       /tmp/.XIM-unix

    1       /tmp/.font-unix

    1       /tmp/ufp

    1       /tmp/ufp/isi_audit_d

    1       /tmp/ufp/isi_cbind_d

    1       /tmp/ufp/isi_cpool_d

    1       /tmp/ufp/isi_cpool_io_d

    1       /tmp/ufp/isi_dmilog

    1       /tmp/ufp/isi_dnsiq_d

    1       /tmp/ufp/isi_drive_d

    1       /tmp/ufp/isi_drivemon

    1       /tmp/ufp/isi_fsa_tool

    1       /tmp/ufp/isi_job_d

    1       /tmp/ufp/isi_km_d

    1       /tmp/ufp/isi_migr_pworker

    1       /tmp/ufp/isi_migr_sched

    1       /tmp/ufp/isi_migr_sworker

    1       /tmp/ufp/isi_netgroup_d

    1       /tmp/ufp/isi_opensm

    1       /tmp/ufp/isi_papi_d

    1       /tmp/ufp/isi_quota_report_d

    1       /tmp/ufp/isi_sasphymon

    1       /tmp/ufp/isi_snapshot_d

    1       /tmp/ufp/isi_stats_d

    1       /tmp/ufp/isi_stats_hist_d

    1       /tmp/ufp/lsass

    1       /tmp/ufp/lwio

    1       /tmp/ufp/lwreg

    1       /tmp/ufp/mountd

    1       /tmp/ufp/netlogon

    1       /tmp/ufp/srvsvc

    1       /usr/X11R6

    1       /usr/X11R6/share

    1       /usr/bin/isi_audit

    1       /usr/bin/isi_celog

    1       /usr/bin/remotesupport

    1       /usr/lib/aout

    1       /usr/lib/compat

    1       /usr/lib/dtrace

    1       /usr/lib/engines

    1       /usr/lib/gss

    1       /usr/lib/krb5

    1       /usr/libdata

    1       /usr/libdata/TSMConvertor

    1       /usr/libdata/gcc

    1       /usr/libdata/ldscripts

    1       /usr/libdata/lint

    1       /usr/libdata/patchmgr

    1       /usr/libexec/isilon

    1       /usr/libexec/lpr

    1       /usr/libexec/sendmail

    1       /usr/libexec/sm.bin

    1       /usr/likewise/bin

    1       /usr/likewise/data

    1       /usr/likewise/libexec

    1       /usr/likewise/sbin

    1       /usr/likewise/share

    1       /usr/local/etc

    1       /usr/local/info

    1       /usr/local/isi_check_cluster

    1       /usr/local/libdata

    1       /usr/local/libexec

    1       /usr/share/calendar

    1       /usr/share/et

    1       /usr/share/gconfig

    1       /usr/share/isdn

    1       /usr/share/isi_xsd

    1       /usr/share/man1

    1       /usr/share/man5

    1       /usr/share/man8

    1       /usr/share/me

    1       /usr/share/mk

    1       /usr/share/nls

    1       /usr/share/openssl

    1       /usr/share/security

    1       /usr/share/sendmail

    1       /usr/share/skel

    1       /usr/share/snmp

    1       /usr/share/ssl

    1       /usr/share/tabset

    1       /usr/share/tmac

    1       /usr/share/vi

    1       /usr/share/zoneinfo

    1       /usr/update

    1       /usr/update/mnt

    1       /usr/update/patch

    1       /usr/update/root

    1       /usr/update/var

    1       /var

    2       /etc/gconfig

    2       /usr/share/doc

    2       /usr/share/examples

    2       /usr/share/locale

    2       /usr/share/papi

    2       /usr/share/syscons

    3       /bin

    3       /sbin

    3       /usr/bin/isi_hwtools

    3       /usr/share/groff_font

    3       /usr/share/info

    4       /usr/local/man

    4       /usr/share/dict

    4       /usr/share/misc

    5       /etc

    5       /usr/local/SupportIQ

    5       /usr/local/apache2

    9       /usr/share/man

    11      /lib

    11      /usr/local/sbin

    12      /usr/local/www

    14      /usr/likewise/lib

    14      /usr/local/connectemc

    14      /usr/sbin

    15      /usr/likewise

    19      /usr/local/bin

    24      /boot/kernel.amd64

    24      /usr/libexec

    29      /boot

    31      /usr/share

    33      /usr/local/share

    40      /usr/bin

    45      /usr/lib

    67      /usr/local/aspera

    91      /usr/local/lib

    259     /usr/local

    425     /usr

    475     /

     

     

     

     

    ISILON-2# cd /usr/lib

    ISILON-2# ls

    Scrt1.o                         libiconv.la                     libisi_file_filter.so.1         libisi_pq.so.1                  libpanelw.so

    aout                            libiconv.so                     libisi_filter.so                libisi_preconfig_transport.so   libpanelw.so.4

    charset.alias                   libiconv.so.7                   libisi_filter.so.1              libisi_preconfig_transport.so.1 libpcap.so

    compat                          libifs.so                       libisi_flexnet.so               libisi_priv.so                  libpmc.so

    crt1.o                          libifs.so.1                     libisi_flexnet.so.1             libisi_priv.so.1                libpmc.so.4

    crtbegin.o                      libipsec.so                     libisi_flt_audit.so             libisi_priv_db.so               libproc.so

    crtbeginS.o                     libipx.so                       libisi_flt_audit.so.1           libisi_priv_db.so.1             libproc.so.1

    crtbeginT.o                     libisc.so                       libisi_fsa.so                   libisi_pstore_gcfg.so           libpthread.a

    crtend.o                        libisc.so.37                    libisi_fsa.so.1                 libisi_pstore_gcfg.so.1         libpthread.so

    crtendS.o                       libisccc.so                     libisi_gconfig.so               libisi_qpl.so                   libpython.so

    crti.o                          libisccc.so.30                  libisi_gconfig.so.1             libisi_qpl.so.1                 libradius.so

    crtn.o                          libisccfg.so                    libisi_gconfig_c_client.so      libisi_quota.so                 libradius.so.3

    dtrace                          libisccfg.so.30                 libisi_gconfig_c_client.so.1    libisi_quota.so.1               libreadline.so

    engines                         libisi_acl.so                   libisi_gconfig_client.so        libisi_remote_support.so        librpcsvc.so

    gcov                            libisi_acl.so.1                 libisi_gconfig_client.so.1      libisi_remote_support.so.1      librpcsvc.so.4

    gcrt1.o                         libisi_archive.so               libisi_gconfig_server.so        libisi_rest_client.so           librt.so

    gss                             libisi_archive.so.2             libisi_gconfig_server.so.1      libisi_rest_client.so.1         librt.so.1

    kgzldr.o                        libisi_audit.so                 libisi_gmp.so                   libisi_rest_server.so           libsbuf.so

    krb5                            libisi_audit.so.1               libisi_gmp.so.1                 libisi_rest_server.so.1         libsmb.so

    kt_isi_pstore.so                libisi_auth.so                  libisi_hal.so                   libisi_rnv.so                   libsmb.so.3

    kt_isi_pstore.so.1              libisi_auth.so.1                libisi_hal.so.1                 libisi_rnv.so.1                 libssl.so

    libalias.so                     libisi_auth_config.so           libisi_hdfs.so                  libisi_sbtree.so                libssl.so.5

    libarchive.so                   libisi_auth_config.so.1         libisi_hdfs.so.1                libisi_sbtree.so.1              libssl.so.6

    libarchive.so.4                 libisi_auth_cpp.so              libisi_hdfs_gcfg.so             libisi_services_gcfg.so         libssp.so

    libavl.so                       libisi_auth_cpp.so.1            libisi_hdfs_gcfg.so.1           libisi_services_gcfg.so.1       libstdc++.so

    libbegemot.so                   libisi_auth_upgrade.so          libisi_hdfs_protobuf.so         libisi_shareinfo.so             libstdc++.so.6

    libbind.so                      libisi_auth_upgrade.so.1        libisi_hdfs_protobuf.so.1       libisi_shareinfo.so.1           libtacplus.so

    libbind.so.5                    libisi_avscan.so                libisi_http_config.so           libisi_shareinfo_cpp.so         libtacplus.so.3

    libbind9.so                     libisi_avscan.so.1              libisi_http_config.so.1         libisi_shareinfo_cpp.so.1       libtermcap.a

    libbind9.so.31                  libisi_bedrs_base.so            libisi_hw.so                    libisi_smb.so                   libtermcap.so

    libboost_date_time.so           libisi_bedrs_base.so.1          libisi_hw.so.1                  libisi_smb.so.1                 libtermcapw.a

    libboost_date_time.so.4         libisi_bedrs_config.so          libisi_idi.so                   libisi_smb_upgrade.so           libtermcapw.so

    libboost_filesystem.so          libisi_bedrs_config.so.1        libisi_idi.so.1                 libisi_smb_upgrade.so.1         libtermlib.a

    libboost_filesystem.so.4        libisi_bedrs_config_impl.so     libisi_idmap.so                 libisi_snapshot.so              libtermlib.so

    libboost_iostreams.so           libisi_bedrs_config_impl.so.1   libisi_idmap.so.1               libisi_snapshot.so.1            libtermlibw.a

    libboost_iostreams.so.4         libisi_bedrs_env.so             libisi_ilog.so                  libisi_sra_gcfg.so              libtermlibw.so

    libboost_program_options.so     libisi_bedrs_env.so.1           libisi_ilog.so.1                libisi_sra_gcfg.so.1            libthr.so

    libboost_program_options.so.4   libisi_bedrs_env_impl.so        libisi_iscsi.so                 libisi_sstore.so                libthread_db.so

    libboost_random.so              libisi_bedrs_env_impl.so.1      libisi_iscsi.so.1               libisi_sstore.so.1              libthread_db.so.3

    libboost_regex.so               libisi_bedrs_logging.so         libisi_isns_client.so           libisi_stats.so                 libtinfo.a

    libboost_regex.so.4             libisi_bedrs_logging.so.1       libisi_isns_client.so.1         libisi_stats.so.1               libtinfo.so

    libboost_serialization.so       libisi_bedrs_logging_impl.so    libisi_job.so                   libisi_sysctl_gcfg.so           libtinfow.a

    libboost_serialization.so.4     libisi_bedrs_logging_impl.so.1  libisi_job.so.1                 libisi_sysctl_gcfg.so.1         libtinfow.so

    libboost_system.so              libisi_bedrs_mli.so             libisi_join.so                  libisi_testutil.so              libufs.so

    libboost_system.so.4            libisi_bedrs_mli.so.1           libisi_join.so.1                libisi_testutil.so.1            libugidfw.so

    libboost_thread.so              libisi_bedrs_platform.so        libisi_journal.so               libisi_thr_dbg.so               libugidfw.so.3

    libboost_thread.so.4            libisi_bedrs_platform.so.1      libisi_journal.so.1             libisi_thr_dbg.so.1             libumem.so

    libbsdxml.so                    libisi_bedrs_platform_impl.so   libisi_json.so                  libisi_thread.so                libusbhid.so

    libbsm.so                       libisi_bedrs_platform_impl.so.1 libisi_json.so.1                libisi_thread.so.1              libusbhid.so.3

    libbsm.so.2                     libisi_bedrs_status.so          libisi_km_server.so             libisi_ufp.so                   libutil.so

    libbsnmp.so                     libisi_bedrs_status.so.1        libisi_km_server.so.1           libisi_ufp.so.1                 libuutil.so

    libbz2.so                       libisi_bedrs_task.so            libisi_kmcfg.so                 libisi_uktr.so                  libvmwarerpc.so

    libbz2.so.3                     libisi_bedrs_task.so.1          libisi_kmcfg.so.1               libisi_uktr.so.1                libvmwarerpc.so.1

    libc.so                         libisi_blockmask.so             libisi_kmcommonlib.so           libisi_upgrade_api.so           libwrap.so

    libcalendar.so                  libisi_blockmask.so.1           libisi_kmcommonlib.so.1         libisi_upgrade_api.so.1         libwrap.so.5

    libcalendar.so.4                libisi_burst_sdk.so             libisi_kmctrllib.so             libisi_util.so                  libxml2.so

    libcam.so                       libisi_burst_sdk.so.2           libisi_kmctrllib.so.1           libisi_util.so.1                libxml2.so.6

    libcharset.la                   libisi_cbind.so                 libisi_kmdatalib.so             libisi_util_cpp.so              libypclnt.so

    libcharset.so                   libisi_cbind.so.1               libisi_kmdatalib.so.1           libisi_util_cpp.so.1            libypclnt.so.3

    libcharset.so.1                 libisi_celog.so                 libisi_kmrpclib.so              libisi_vasa_service.so          libz.so

    libcom_err.so                   libisi_celog.so.1               libisi_kmrpclib.so.1            libisi_vasa_service.so.1        pam_chroot.so

    libcom_err.so.3                 libisi_celog_query.so           libisi_krb5_config.so           libisi_vc_gcfg.so               pam_chroot.so.4

    libcom_err.so.4                 libisi_celog_query.so.1         libisi_krb5_config.so.1         libisi_vc_gcfg.so.1             pam_deny.so

    libcomplib.so                   libisi_changelist.so            libisi_l3_cache.so              libisi_version.so               pam_deny.so.4

    libcomplib.so.1                 libisi_changelist.so.1          libisi_l3_cache.so.1            libisi_version.so.1             pam_echo.so

    libcrypt.so                     libisi_cloud_api.so             libisi_licensing.so             libisi_vol_copy_config.so       pam_echo.so.4

    libcrypto.so                    libisi_cloud_api.so.1           libisi_licensing.so.2           libisi_vol_copy_config.so.1     pam_exec.so

    libctf.so                       libisi_cluster_config.so        libisi_line_edit.so             libisi_xml.so                   pam_exec.so.4

    libcurses.a                     libisi_cluster_config.so.1      libisi_line_edit.so.1           libisi_xml.so.1                 pam_ftpusers.so

    libcurses.so                    libisi_config.so                libisi_lnv.so                   libisi_zone.so                  pam_ftpusers.so.4

    libcursesw.a                    libisi_config.so.1              libisi_lnv.so.1                 libisi_zone.so.1                pam_group.so

    libcursesw.so                   libisi_cpool_bkup.so            libisi_logger.so                libk5crypto.so                  pam_group.so.4

    libdevinfo.so                   libisi_cpool_bkup.so.1          libisi_logger.so.1              libk5crypto.so.3                pam_guest.so

    libdevinfo.so.4                 libisi_cpool_cbm.so             libisi_lwconfig.so              libkadm5clnt.so                 pam_guest.so.4

    libdevstat.so                   libisi_cpool_cbm.so.1           libisi_lwconfig.so.1            libkadm5clnt.so.6               pam_lastlog.so

    libdialog.so                    libisi_cpool_config.so          libisi_lwconfig_local.so        libkadm5srv.so                  pam_lastlog.so.4

    libdialog.so.6                  libisi_cpool_config.so.1        libisi_lwconfig_local.so.1      libkadm5srv.so.6                pam_login_access.so

    libdmalloc.so                   libisi_cpool_crh_client.so      libisi_migrate.so               libkdb5.so                      pam_login_access.so.4

    libdmalloc.so.0                 libisi_cpool_crh_client.so.1    libisi_migrate.so.2             libkdb5.so.4                    pam_nologin.so

    libdmallocth.so                 libisi_cpool_crh_common.so      libisi_migrate_private.so       libkiconv.so                    pam_nologin.so.4

    libdmallocth.so.0               libisi_cpool_crh_common.so.1    libisi_migrate_private.so.2     libkrb5.so                      pam_opie.so

    libdmallocthcxx.so              libisi_cpool_d.so               libisi_mps.so                   libkrb5.so.3                    pam_opie.so.4

    libdmallocthcxx.so.1            libisi_cpool_d.so.1             libisi_mps.so.1                 libkrb5support.so               pam_opieaccess.so

    libdmallocxx.so                 libisi_cpool_d_api.so           libisi_ndmp.so                  libkrb5support.so.0             pam_opieaccess.so.4

    libdmallocxx.so.1               libisi_cpool_d_api.so.1         libisi_ndmp.so.1                libkvm.so                       pam_passwdqc.so

    libdns.so                       libisi_cpool_d_common.so        libisi_ndmpbrm2.so              liblwres.so                     pam_passwdqc.so.4

    libdns.so.39                    libisi_cpool_d_common.so.1      libisi_ndmpbrm2.so.1            liblwres.so.30                  pam_permit.so

    libdtrace.so                    libisi_cpool_d_tch.so           libisi_net.so                   libm.so                         pam_permit.so.4

    libdwarf.so                     libisi_cpool_d_tch.so.1         libisi_net.so.1                 libmagic.so                     pam_radius.so

    libdwarf.so.1                   libisi_cpool_km.so              libisi_newfs.so                 libmagic.so.3                   pam_radius.so.4

    libedit.so                      libisi_cpool_km.so.1            libisi_newfs.so.1               libmd.so                        pam_rhosts.so

    libelf.so                       libisi_cpool_rdr.so             libisi_nfs.so                   libmemstat.so                   pam_rhosts.so.4

    libelf.so.1                     libisi_cpool_rdr.so.1           libisi_nfs.so.1                 libmemstat.so.2                 pam_rootok.so

    libexecinfo.so                  libisi_cpool_security.so        libisi_nfs_config.so            libmenu.so                      pam_rootok.so.4

    libexecinfo.so.1                libisi_cpool_security.so.1      libisi_nfs_config.so.1          libmenu.so.4                    pam_securetty.so

    libfetch.so                     libisi_daemon.so                libisi_node_config_gcfg.so      libmenuw.so                     pam_securetty.so.4

    libfetch.so.5                   libisi_daemon.so.1              libisi_node_config_gcfg.so.1    libmenuw.so.4                   pam_self.so

    libform.so                      libisi_date.so                  libisi_node_status_gcfg.so      libmilter.so                    pam_self.so.4

    libform.so.4                    libisi_date.so.1                libisi_node_status_gcfg.so.1    libmilter.so.4                  pam_tacplus.so

    libformw.so                     libisi_dda.so                   libisi_ntoken.so                libmp.so                        pam_tacplus.so.4

    libformw.so.4                   libisi_dda.so.1                 libisi_ntoken.so.1              libmp.so.6                      pam_unix.so

    libftpio.so                     libisi_dedupe_gcfg.so           libisi_ntp_config.so            libncp.so                       pam_unix.so.4

    libftpio.so.7                   libisi_dedupe_gcfg.so.1         libisi_ntp_config.so.1          libncp.so.3                     snmp_atm.so

    libgcc.a                        libisi_diskpool.so              libisi_object.so                libncurses.so                   snmp_atm.so.5

    libgcc_s.so                     libisi_diskpool.so.1            libisi_object.so.1              libncursesw.so                  snmp_bridge.so

    libgeom.so                      libisi_dns.so                   libisi_object_api.so            libnetgraph.so                  snmp_bridge.so.5

    libgnuregex.so                  libisi_dns.so.1                 libisi_object_api.so.1          libnetgraph.so.3                snmp_hostres.so

    libgnuregex.so.4                libisi_domain.so                libisi_openssl.so               libngatm.so                     snmp_hostres.so.5

    libgomp.so                      libisi_domain.so.1              libisi_openssl.so.1             libngatm.so.3                   snmp_mibII.so

    libgomp.so.1                    libisi_ecs.so                   libisi_persona.so               libnvpair.so                    snmp_mibII.so.5

    libgpib.so                      libisi_ecs.so.1                 libisi_persona.so.1             libobjc.so                      snmp_netgraph.so

    libgpib.so.2                    libisi_emctar.so                libisi_pkg.so                   libobjc.so.3                    snmp_netgraph.so.5

    libgssapi_krb5.so               libisi_emctar.so.1              libisi_pkg.so.1                 libopie.so                      snmp_pf.so

    libgssapi_krb5.so.2             libisi_event.so                 libisi_platform_api.so          libopie.so.5                    snmp_pf.so.5

    libgssrpc.so                    libisi_event.so.1               libisi_platform_api.so.1        libpam.so

    libgssrpc.so.4                  libisi_event_r.so               libisi_pools_fsa_gcfg.so        libpam.so.4

    libhistory.so                   libisi_event_r.so.1             libisi_pools_fsa_gcfg.so.1      libpanel.so

    libhistory.so.7                 libisi_file_filter.so           libisi_pq.so                    libpanel.so.4

  • 5. Re: Isilon 36000x / directory full
    Peter Serocka

    There is nothing unusual as far as I see. The KB article you quoted has some more instructions, like

    removing firmware packages.  I assume you walked though all the steps.

     

    What else could be done?

     

    Compare the "du" outputs for nodes ISILON-1 and ISILON-6,

    the totals already differ by some percent -- maybe the directory

    with relevant amount of extra data can be identified.

     

    Even better if we there are historic du runs archived:

    Look under /ifs/data/Isilon_Support/pkg

    for old logs previously sent to Isilon; I can help extracting

    the du outputs later.

     

    Finally, if the cluster doesn't use Aspera, and if you are brave of heart,

    consider backing up and removing

    the   /usr/local/aspera  sub-tree.

    It containes pre-installed third party software, and a OneFS 7.2.1 virtual node

    which I tested runs fine without it...

     

    -- Peter