
-
1. Re: Isilon 36000x / directory full
Chris Klosterman Jan 2, 2018 12:42 PM (in response to ismail123)Obviously these nodes are no longer under active support, right? So what I would suggest is to look and see what if anything you can clean up.
isilon-1# cd /var/
isilon-1# du -sm * | sort -n -k1 | tail -5
1 yp
10 backups
16 db
43 crash
86 log
Crash can of course be cleaned up, especially if there are old core dumps in it that you no longer need. Older log files get rolled over in /var/log and compressed with .gz. Perhaps delete those.
Look in /root and /tmp as well, sometimes people put upgrade or patch files in one of those locations and forget to clean up after themselves.
Yes seeing it over 100% is definitely an area for concern. I just hope this cluster is no longer is production (again assuming that the nodes are no longer under active maintenance). If they are, then certainly open an SR, and have support help you do this.
~Chris
-
2. Re: Isilon 36000x / directory full
ismail123 Jan 4, 2018 10:24 PM (in response to Chris Klosterman)Hi Chris,
Thanks for your reply.
The thing is I saw too many.gz old log files and I moved them to ifs folders but then also the partition shows full.The /var/log/ has used only 33M then which folder is holding other space?
It says /dev/imdd0a this folder is holding more data. Is there any other folders might have caused more space inside / directory?
I checked /var/crash also that doesn't have more data.
Please advise me where to check for unwanted files.. I checked for patch files and user files nothing found.
ISILON-2# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/imdd0a 496M 481M -24M 105% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/imdd1a 496M 33M 424M 7% /var
/dev/imdd2a 496M 297K 456M 0% /var/crash
OneFS 227T 182T 43T 81% /ifs
Regards,
Ismail
-
3. Re: Isilon 36000x / directory full
Peter Serocka Jan 5, 2018 7:50 AM (in response to ismail123)Yes, one would focus only on the "/" filesystem mounted on /dev/imdd0a
Check the how many MBs are used for regular, visible files:
du -shx /
(the "x" option makes du report only for "/" in this case, i.e. omitting /var and /var/crash).
Does this yield a number significantly *lower" than the usage reported by "df -h /" ?
That df number was 481 MB in your earlier result.
If the du number is much lower than the df number, it usually means tha
files have been deleted but are still "open" by some process(es). Unless
we have a clue which process(es) are responsible and then could be restarted,
the simplest and most certain way to a resolution is rebooting each node in question.
If the du number is about the same as the df number,
we need to inspect the file tre under "/" (excluding /var/ and /var/crash)
in more detail. Use
du -xm -d 3 / | sort -n
The output will be all in MBs, so that the sorting makes sense.
Note the usage numbers for higher level directories are inclusive,
so usage is reported multiple times (usage for /usr/local is also in /usr, which again is in /).
Increase the reporting depth from 3 to 4 or higher for more insights.
Of course it would be good to have for reference one node or cluster
where "/" isn't filled up...
hth
-- Peter
-
4. Re: Isilon 36000x / directory full
ismail123 Jan 8, 2018 11:20 AM (in response to Peter Serocka)Hi Peter,
Thanks for your reply.
You are correct, I found /usr/local holding around 259MB of files, but not sure which are config files and which are unwanted files.
Below is the o/p could you please help me on this to reduce some space.
Apologies for the legnthy o/p. could'nt able to attach.
ISILON-2# du -xm -d 3 / | sort -n
1 /boot/defaults
1 /boot/firmware
1 /boot/kernel
1 /boot/modules
1 /boot/zfs
1 /compadmin
1 /dev
1 /etc/X11
1 /etc/bluetooth
1 /etc/defaults
1 /etc/ganglia
1 /etc/gnats
1 /etc/gss
1 /etc/ifs
1 /etc/ifs/hw
1 /etc/ifs/licensing
1 /etc/ifs/log
1 /etc/ifs/patchmgr
1 /etc/ifs/scripts
1 /etc/isdn
1 /etc/lcd
1 /etc/lcd/InvertedSmall
1 /etc/lcd/SmallFilled
1 /etc/local
1 /etc/mail
1 /etc/mcp
1 /etc/mcp/defaults
1 /etc/mcp/override
1 /etc/mcp/scripts
1 /etc/mcp/sync
1 /etc/mcp/sys
1 /etc/mcp/templates
1 /etc/mtree
1 /etc/ntp
1 /etc/openldap
1 /etc/pam.d
1 /etc/periodic
1 /etc/periodic/daily
1 /etc/periodic/monthly
1 /etc/periodic/security
1 /etc/periodic/weekly
1 /etc/ppp
1 /etc/rc.d
1 /etc/rc.d_disabled
1 /etc/security
1 /etc/skel
1 /etc/ssh
1 /etc/ssl
1 /etc/xinetd.d
1 /etc/zfs
1 /ifs
1 /keystore
1 /lib/geom
1 /libexec
1 /media
1 /mnt
1 /opt
1 /proc
1 /rescue
1 /root
1 /root/.ccache
1 /root/.makekit
1 /root/.snmp
1 /root/.ssh
1 /tmp
1 /tmp/.ICE-unix
1 /tmp/.X11-unix
1 /tmp/.XIM-unix
1 /tmp/.font-unix
1 /tmp/ufp
1 /tmp/ufp/isi_audit_d
1 /tmp/ufp/isi_cbind_d
1 /tmp/ufp/isi_cpool_d
1 /tmp/ufp/isi_cpool_io_d
1 /tmp/ufp/isi_dmilog
1 /tmp/ufp/isi_dnsiq_d
1 /tmp/ufp/isi_drive_d
1 /tmp/ufp/isi_drivemon
1 /tmp/ufp/isi_fsa_tool
1 /tmp/ufp/isi_job_d
1 /tmp/ufp/isi_km_d
1 /tmp/ufp/isi_migr_pworker
1 /tmp/ufp/isi_migr_sched
1 /tmp/ufp/isi_migr_sworker
1 /tmp/ufp/isi_netgroup_d
1 /tmp/ufp/isi_opensm
1 /tmp/ufp/isi_papi_d
1 /tmp/ufp/isi_quota_report_d
1 /tmp/ufp/isi_sasphymon
1 /tmp/ufp/isi_snapshot_d
1 /tmp/ufp/isi_stats_d
1 /tmp/ufp/isi_stats_hist_d
1 /tmp/ufp/lsass
1 /tmp/ufp/lwio
1 /tmp/ufp/lwreg
1 /tmp/ufp/mountd
1 /tmp/ufp/netlogon
1 /tmp/ufp/srvsvc
1 /usr/X11R6
1 /usr/X11R6/share
1 /usr/bin/isi_audit
1 /usr/bin/isi_celog
1 /usr/bin/remotesupport
1 /usr/lib/aout
1 /usr/lib/compat
1 /usr/lib/dtrace
1 /usr/lib/engines
1 /usr/lib/gss
1 /usr/lib/krb5
1 /usr/libdata
1 /usr/libdata/TSMConvertor
1 /usr/libdata/gcc
1 /usr/libdata/ldscripts
1 /usr/libdata/lint
1 /usr/libdata/patchmgr
1 /usr/libexec/isilon
1 /usr/libexec/lpr
1 /usr/libexec/sendmail
1 /usr/libexec/sm.bin
1 /usr/likewise/bin
1 /usr/likewise/data
1 /usr/likewise/libexec
1 /usr/likewise/sbin
1 /usr/likewise/share
1 /usr/local/etc
1 /usr/local/info
1 /usr/local/isi_check_cluster
1 /usr/local/libdata
1 /usr/local/libexec
1 /usr/share/calendar
1 /usr/share/et
1 /usr/share/gconfig
1 /usr/share/isdn
1 /usr/share/isi_xsd
1 /usr/share/man1
1 /usr/share/man5
1 /usr/share/man8
1 /usr/share/me
1 /usr/share/mk
1 /usr/share/nls
1 /usr/share/openssl
1 /usr/share/security
1 /usr/share/sendmail
1 /usr/share/skel
1 /usr/share/snmp
1 /usr/share/ssl
1 /usr/share/tabset
1 /usr/share/tmac
1 /usr/share/vi
1 /usr/share/zoneinfo
1 /usr/update
1 /usr/update/mnt
1 /usr/update/patch
1 /usr/update/root
1 /usr/update/var
1 /var
2 /etc/gconfig
2 /usr/share/doc
2 /usr/share/examples
2 /usr/share/locale
2 /usr/share/papi
2 /usr/share/syscons
3 /bin
3 /sbin
3 /usr/bin/isi_hwtools
3 /usr/share/groff_font
3 /usr/share/info
4 /usr/local/man
4 /usr/share/dict
4 /usr/share/misc
5 /etc
5 /usr/local/SupportIQ
5 /usr/local/apache2
9 /usr/share/man
11 /lib
11 /usr/local/sbin
12 /usr/local/www
14 /usr/likewise/lib
14 /usr/local/connectemc
14 /usr/sbin
15 /usr/likewise
19 /usr/local/bin
24 /boot/kernel.amd64
24 /usr/libexec
29 /boot
31 /usr/share
33 /usr/local/share
40 /usr/bin
45 /usr/lib
67 /usr/local/aspera
91 /usr/local/lib
259 /usr/local
425 /usr
475 /
ISILON-2# cd /usr/lib
ISILON-2# ls
Scrt1.o libiconv.la libisi_file_filter.so.1 libisi_pq.so.1 libpanelw.so
aout libiconv.so libisi_filter.so libisi_preconfig_transport.so libpanelw.so.4
charset.alias libiconv.so.7 libisi_filter.so.1 libisi_preconfig_transport.so.1 libpcap.so
compat libifs.so libisi_flexnet.so libisi_priv.so libpmc.so
crt1.o libifs.so.1 libisi_flexnet.so.1 libisi_priv.so.1 libpmc.so.4
crtbegin.o libipsec.so libisi_flt_audit.so libisi_priv_db.so libproc.so
crtbeginS.o libipx.so libisi_flt_audit.so.1 libisi_priv_db.so.1 libproc.so.1
crtbeginT.o libisc.so libisi_fsa.so libisi_pstore_gcfg.so libpthread.a
crtend.o libisc.so.37 libisi_fsa.so.1 libisi_pstore_gcfg.so.1 libpthread.so
crtendS.o libisccc.so libisi_gconfig.so libisi_qpl.so libpython.so
crti.o libisccc.so.30 libisi_gconfig.so.1 libisi_qpl.so.1 libradius.so
crtn.o libisccfg.so libisi_gconfig_c_client.so libisi_quota.so libradius.so.3
dtrace libisccfg.so.30 libisi_gconfig_c_client.so.1 libisi_quota.so.1 libreadline.so
engines libisi_acl.so libisi_gconfig_client.so libisi_remote_support.so librpcsvc.so
gcov libisi_acl.so.1 libisi_gconfig_client.so.1 libisi_remote_support.so.1 librpcsvc.so.4
gcrt1.o libisi_archive.so libisi_gconfig_server.so libisi_rest_client.so librt.so
gss libisi_archive.so.2 libisi_gconfig_server.so.1 libisi_rest_client.so.1 librt.so.1
kgzldr.o libisi_audit.so libisi_gmp.so libisi_rest_server.so libsbuf.so
krb5 libisi_audit.so.1 libisi_gmp.so.1 libisi_rest_server.so.1 libsmb.so
kt_isi_pstore.so libisi_auth.so libisi_hal.so libisi_rnv.so libsmb.so.3
kt_isi_pstore.so.1 libisi_auth.so.1 libisi_hal.so.1 libisi_rnv.so.1 libssl.so
libalias.so libisi_auth_config.so libisi_hdfs.so libisi_sbtree.so libssl.so.5
libarchive.so libisi_auth_config.so.1 libisi_hdfs.so.1 libisi_sbtree.so.1 libssl.so.6
libarchive.so.4 libisi_auth_cpp.so libisi_hdfs_gcfg.so libisi_services_gcfg.so libssp.so
libavl.so libisi_auth_cpp.so.1 libisi_hdfs_gcfg.so.1 libisi_services_gcfg.so.1 libstdc++.so
libbegemot.so libisi_auth_upgrade.so libisi_hdfs_protobuf.so libisi_shareinfo.so libstdc++.so.6
libbind.so libisi_auth_upgrade.so.1 libisi_hdfs_protobuf.so.1 libisi_shareinfo.so.1 libtacplus.so
libbind.so.5 libisi_avscan.so libisi_http_config.so libisi_shareinfo_cpp.so libtacplus.so.3
libbind9.so libisi_avscan.so.1 libisi_http_config.so.1 libisi_shareinfo_cpp.so.1 libtermcap.a
libbind9.so.31 libisi_bedrs_base.so libisi_hw.so libisi_smb.so libtermcap.so
libboost_date_time.so libisi_bedrs_base.so.1 libisi_hw.so.1 libisi_smb.so.1 libtermcapw.a
libboost_date_time.so.4 libisi_bedrs_config.so libisi_idi.so libisi_smb_upgrade.so libtermcapw.so
libboost_filesystem.so libisi_bedrs_config.so.1 libisi_idi.so.1 libisi_smb_upgrade.so.1 libtermlib.a
libboost_filesystem.so.4 libisi_bedrs_config_impl.so libisi_idmap.so libisi_snapshot.so libtermlib.so
libboost_iostreams.so libisi_bedrs_config_impl.so.1 libisi_idmap.so.1 libisi_snapshot.so.1 libtermlibw.a
libboost_iostreams.so.4 libisi_bedrs_env.so libisi_ilog.so libisi_sra_gcfg.so libtermlibw.so
libboost_program_options.so libisi_bedrs_env.so.1 libisi_ilog.so.1 libisi_sra_gcfg.so.1 libthr.so
libboost_program_options.so.4 libisi_bedrs_env_impl.so libisi_iscsi.so libisi_sstore.so libthread_db.so
libboost_random.so libisi_bedrs_env_impl.so.1 libisi_iscsi.so.1 libisi_sstore.so.1 libthread_db.so.3
libboost_regex.so libisi_bedrs_logging.so libisi_isns_client.so libisi_stats.so libtinfo.a
libboost_regex.so.4 libisi_bedrs_logging.so.1 libisi_isns_client.so.1 libisi_stats.so.1 libtinfo.so
libboost_serialization.so libisi_bedrs_logging_impl.so libisi_job.so libisi_sysctl_gcfg.so libtinfow.a
libboost_serialization.so.4 libisi_bedrs_logging_impl.so.1 libisi_job.so.1 libisi_sysctl_gcfg.so.1 libtinfow.so
libboost_system.so libisi_bedrs_mli.so libisi_join.so libisi_testutil.so libufs.so
libboost_system.so.4 libisi_bedrs_mli.so.1 libisi_join.so.1 libisi_testutil.so.1 libugidfw.so
libboost_thread.so libisi_bedrs_platform.so libisi_journal.so libisi_thr_dbg.so libugidfw.so.3
libboost_thread.so.4 libisi_bedrs_platform.so.1 libisi_journal.so.1 libisi_thr_dbg.so.1 libumem.so
libbsdxml.so libisi_bedrs_platform_impl.so libisi_json.so libisi_thread.so libusbhid.so
libbsm.so libisi_bedrs_platform_impl.so.1 libisi_json.so.1 libisi_thread.so.1 libusbhid.so.3
libbsm.so.2 libisi_bedrs_status.so libisi_km_server.so libisi_ufp.so libutil.so
libbsnmp.so libisi_bedrs_status.so.1 libisi_km_server.so.1 libisi_ufp.so.1 libuutil.so
libbz2.so libisi_bedrs_task.so libisi_kmcfg.so libisi_uktr.so libvmwarerpc.so
libbz2.so.3 libisi_bedrs_task.so.1 libisi_kmcfg.so.1 libisi_uktr.so.1 libvmwarerpc.so.1
libc.so libisi_blockmask.so libisi_kmcommonlib.so libisi_upgrade_api.so libwrap.so
libcalendar.so libisi_blockmask.so.1 libisi_kmcommonlib.so.1 libisi_upgrade_api.so.1 libwrap.so.5
libcalendar.so.4 libisi_burst_sdk.so libisi_kmctrllib.so libisi_util.so libxml2.so
libcam.so libisi_burst_sdk.so.2 libisi_kmctrllib.so.1 libisi_util.so.1 libxml2.so.6
libcharset.la libisi_cbind.so libisi_kmdatalib.so libisi_util_cpp.so libypclnt.so
libcharset.so libisi_cbind.so.1 libisi_kmdatalib.so.1 libisi_util_cpp.so.1 libypclnt.so.3
libcharset.so.1 libisi_celog.so libisi_kmrpclib.so libisi_vasa_service.so libz.so
libcom_err.so libisi_celog.so.1 libisi_kmrpclib.so.1 libisi_vasa_service.so.1 pam_chroot.so
libcom_err.so.3 libisi_celog_query.so libisi_krb5_config.so libisi_vc_gcfg.so pam_chroot.so.4
libcom_err.so.4 libisi_celog_query.so.1 libisi_krb5_config.so.1 libisi_vc_gcfg.so.1 pam_deny.so
libcomplib.so libisi_changelist.so libisi_l3_cache.so libisi_version.so pam_deny.so.4
libcomplib.so.1 libisi_changelist.so.1 libisi_l3_cache.so.1 libisi_version.so.1 pam_echo.so
libcrypt.so libisi_cloud_api.so libisi_licensing.so libisi_vol_copy_config.so pam_echo.so.4
libcrypto.so libisi_cloud_api.so.1 libisi_licensing.so.2 libisi_vol_copy_config.so.1 pam_exec.so
libctf.so libisi_cluster_config.so libisi_line_edit.so libisi_xml.so pam_exec.so.4
libcurses.a libisi_cluster_config.so.1 libisi_line_edit.so.1 libisi_xml.so.1 pam_ftpusers.so
libcurses.so libisi_config.so libisi_lnv.so libisi_zone.so pam_ftpusers.so.4
libcursesw.a libisi_config.so.1 libisi_lnv.so.1 libisi_zone.so.1 pam_group.so
libcursesw.so libisi_cpool_bkup.so libisi_logger.so libk5crypto.so pam_group.so.4
libdevinfo.so libisi_cpool_bkup.so.1 libisi_logger.so.1 libk5crypto.so.3 pam_guest.so
libdevinfo.so.4 libisi_cpool_cbm.so libisi_lwconfig.so libkadm5clnt.so pam_guest.so.4
libdevstat.so libisi_cpool_cbm.so.1 libisi_lwconfig.so.1 libkadm5clnt.so.6 pam_lastlog.so
libdialog.so libisi_cpool_config.so libisi_lwconfig_local.so libkadm5srv.so pam_lastlog.so.4
libdialog.so.6 libisi_cpool_config.so.1 libisi_lwconfig_local.so.1 libkadm5srv.so.6 pam_login_access.so
libdmalloc.so libisi_cpool_crh_client.so libisi_migrate.so libkdb5.so pam_login_access.so.4
libdmalloc.so.0 libisi_cpool_crh_client.so.1 libisi_migrate.so.2 libkdb5.so.4 pam_nologin.so
libdmallocth.so libisi_cpool_crh_common.so libisi_migrate_private.so libkiconv.so pam_nologin.so.4
libdmallocth.so.0 libisi_cpool_crh_common.so.1 libisi_migrate_private.so.2 libkrb5.so pam_opie.so
libdmallocthcxx.so libisi_cpool_d.so libisi_mps.so libkrb5.so.3 pam_opie.so.4
libdmallocthcxx.so.1 libisi_cpool_d.so.1 libisi_mps.so.1 libkrb5support.so pam_opieaccess.so
libdmallocxx.so libisi_cpool_d_api.so libisi_ndmp.so libkrb5support.so.0 pam_opieaccess.so.4
libdmallocxx.so.1 libisi_cpool_d_api.so.1 libisi_ndmp.so.1 libkvm.so pam_passwdqc.so
libdns.so libisi_cpool_d_common.so libisi_ndmpbrm2.so liblwres.so pam_passwdqc.so.4
libdns.so.39 libisi_cpool_d_common.so.1 libisi_ndmpbrm2.so.1 liblwres.so.30 pam_permit.so
libdtrace.so libisi_cpool_d_tch.so libisi_net.so libm.so pam_permit.so.4
libdwarf.so libisi_cpool_d_tch.so.1 libisi_net.so.1 libmagic.so pam_radius.so
libdwarf.so.1 libisi_cpool_km.so libisi_newfs.so libmagic.so.3 pam_radius.so.4
libedit.so libisi_cpool_km.so.1 libisi_newfs.so.1 libmd.so pam_rhosts.so
libelf.so libisi_cpool_rdr.so libisi_nfs.so libmemstat.so pam_rhosts.so.4
libelf.so.1 libisi_cpool_rdr.so.1 libisi_nfs.so.1 libmemstat.so.2 pam_rootok.so
libexecinfo.so libisi_cpool_security.so libisi_nfs_config.so libmenu.so pam_rootok.so.4
libexecinfo.so.1 libisi_cpool_security.so.1 libisi_nfs_config.so.1 libmenu.so.4 pam_securetty.so
libfetch.so libisi_daemon.so libisi_node_config_gcfg.so libmenuw.so pam_securetty.so.4
libfetch.so.5 libisi_daemon.so.1 libisi_node_config_gcfg.so.1 libmenuw.so.4 pam_self.so
libform.so libisi_date.so libisi_node_status_gcfg.so libmilter.so pam_self.so.4
libform.so.4 libisi_date.so.1 libisi_node_status_gcfg.so.1 libmilter.so.4 pam_tacplus.so
libformw.so libisi_dda.so libisi_ntoken.so libmp.so pam_tacplus.so.4
libformw.so.4 libisi_dda.so.1 libisi_ntoken.so.1 libmp.so.6 pam_unix.so
libftpio.so libisi_dedupe_gcfg.so libisi_ntp_config.so libncp.so pam_unix.so.4
libftpio.so.7 libisi_dedupe_gcfg.so.1 libisi_ntp_config.so.1 libncp.so.3 snmp_atm.so
libgcc.a libisi_diskpool.so libisi_object.so libncurses.so snmp_atm.so.5
libgcc_s.so libisi_diskpool.so.1 libisi_object.so.1 libncursesw.so snmp_bridge.so
libgeom.so libisi_dns.so libisi_object_api.so libnetgraph.so snmp_bridge.so.5
libgnuregex.so libisi_dns.so.1 libisi_object_api.so.1 libnetgraph.so.3 snmp_hostres.so
libgnuregex.so.4 libisi_domain.so libisi_openssl.so libngatm.so snmp_hostres.so.5
libgomp.so libisi_domain.so.1 libisi_openssl.so.1 libngatm.so.3 snmp_mibII.so
libgomp.so.1 libisi_ecs.so libisi_persona.so libnvpair.so snmp_mibII.so.5
libgpib.so libisi_ecs.so.1 libisi_persona.so.1 libobjc.so snmp_netgraph.so
libgpib.so.2 libisi_emctar.so libisi_pkg.so libobjc.so.3 snmp_netgraph.so.5
libgssapi_krb5.so libisi_emctar.so.1 libisi_pkg.so.1 libopie.so snmp_pf.so
libgssapi_krb5.so.2 libisi_event.so libisi_platform_api.so libopie.so.5 snmp_pf.so.5
libgssrpc.so libisi_event.so.1 libisi_platform_api.so.1 libpam.so
libgssrpc.so.4 libisi_event_r.so libisi_pools_fsa_gcfg.so libpam.so.4
libhistory.so libisi_event_r.so.1 libisi_pools_fsa_gcfg.so.1 libpanel.so
libhistory.so.7 libisi_file_filter.so libisi_pq.so libpanel.so.4
-
5. Re: Isilon 36000x / directory full
Peter Serocka Jan 9, 2018 11:30 AM (in response to ismail123)There is nothing unusual as far as I see. The KB article you quoted has some more instructions, like
removing firmware packages. I assume you walked though all the steps.
What else could be done?
Compare the "du" outputs for nodes ISILON-1 and ISILON-6,
the totals already differ by some percent -- maybe the directory
with relevant amount of extra data can be identified.
Even better if we there are historic du runs archived:
Look under /ifs/data/Isilon_Support/pkg
for old logs previously sent to Isilon; I can help extracting
the du outputs later.
Finally, if the cluster doesn't use Aspera, and if you are brave of heart,
consider backing up and removing
the /usr/local/aspera sub-tree.
It containes pre-installed third party software, and a OneFS 7.2.1 virtual node
which I tested runs fine without it...
-- Peter