tengel/papyri
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
papyri/md/nfs_debugging.md
tengel e8fb7b288e initial import
2024-03-20 11:40:22 -05:00

8 KiB
Raw Permalink Blame History

NFS Debugging

Contents

Userspace Tools

Using rpcdebug is the easiest way to manipulate the kernel interfaces in place of echoing bitmasks to /proc.

Option Description
-c Clear the given debug flags
-s Set the given debug flags
-m module Specify which module's flags to set or clear.
-v Increase the verbosity of rpcdebug's output
-h Print a help message and exit
-vh Print the available debug flags

For the -m option, the available modules are:

Module Description
nfsd The NFS server
nfs The NFS client
nlm The Network Lock Manager, in either an NFS client or server
rpc The Remote Procedure Call module, either NFS client or server

Examples:

rpcdebug -m rpc -s all   # sets all debug flags for RPC
rpcdebug -m rpc -c all   # clears all debug flags for RPC

rpcdebug -m nfsd -s all   # sets all debug flags for NFS Server
rpcdebug -m nfsd -c all   # clears all debug flags for NFS Server

Kernel Interfaces

A bitmask of the debug flags can be echoed into the interface to enable output to syslog; 0 is the default:

/proc/sys/sunrpc/nfsd_debug
/proc/sys/sunrpc/nfs_debug
/proc/sys/sunrpc/nlm_debug
/proc/sys/sunrpc/rpc_debug

Sysctl controls are registered for these interfaces, so they can be used instead of echo:

sysctl -w sunrpc.rpc_debug=1023
sysctl -w sunrpc.rpc_debug=0

sysctl -w sunrpc.nfsd_debug=1023
sysctl -w sunrpc.nfsd_debug=0

At runtime the server holds information that can be examined:

grep . /proc/net/rpc/*/content
cat /proc/fs/nfs/exports
cat /proc/net/rpc/nfsd
ls -l /proc/fs/nfsd

A rundown of /proc/net/rpc/nfsd (the userspace tool nfsstat pretty-prints this info):

/proc/net/rpc/nfsd

* rc (reply cache): <hits> <misses> <nocache>
- hits: client it's retransmitting
- misses: a operation that requires caching
- nocache: a operation that no requires caching

* fh (filehandle): <stale> <total-lookups> <anonlookups> <dir-not-in-cache> <nodir-not-in-cache>
- stale: file handle errors
- total-lookups, anonlookups, dir-not-in-cache, nodir-not-in-cache
  . always seem to be zeros

* io (input/output): <bytes-read> <bytes-written>
- bytes-read: bytes read directly from disk
- bytes-written: bytes written to disk

* th (threads): <threads> <fullcnt> <10%-20%> <20%-30%> ... <90%-100%> <100%>
- threads: number of nfsd threads
- fullcnt: number of times that the last 10% of threads are busy
- 10%-20%, 20%-30% ... 90%-100%: 10 numbers representing 10-20%, 20-30% to 100%
  . Counts the number of times a given interval are busy

* ra (read-ahead): <cache-size> <10%> <20%> ... <100%> <not-found>
- cache-size: always the double of number threads
- 10%, 20% ... 100%: how deep it found what was looking for
- not-found: not found in the read-ahead cache

* net: <netcnt> <netudpcnt> <nettcpcnt> <nettcpconn>
- netcnt: counts every read
- netudpcnt: counts every UDP packet it receives
- nettcpcnt: counts every time it receives data from a TCP connection
- nettcpconn: count every TCP connection it receives

* rpc: <rpccnt> <rpcbadfmt+rpcbadauth+rpcbadclnt> <rpcbadfmt> <rpcbadauth> <rpcbadclnt>
- rpccnt: counts all rpc operations
- rpcbadfmt: counts if while processing a RPC it encounters the following errors:
  . err_bad_dir, err_bad_rpc, err_bad_prog, err_bad_vers, err_bad_proc, err_bad
- rpcbadauth: bad authentication
  . does not count if you try to mount from a machine that it's not in your exports file
- rpcbadclnt: unused

* procN (N = vers): <vs_nproc> <null> <getattr> <setattr> <lookup> <access> <readlink> <read> <write> <create> <mkdir> <symlink> <mknod> <remove> <rmdir> <rename> <link> <readdir> <readdirplus> <fsstat> <fsinfo> <pathconf> <commit>
- vs_nproc: number of procedures for NFS version
  . v2: nfsproc.c, 18
  . v3: nfs3proc.c, 22
  - v4, nfs4proc.c, 2
- statistics: generated from NFS operations at runtime

* proc4ops: <ops> <x..y>
- ops: the definition of LAST_NFS4_OP, OP_RELEASE_LOCKOWNER = 39, plus 1 (so 40); defined in nfs4.h
- x..y: the array of nfs_opcount up to LAST_NFS4_OP (nfsdstats.nfs4_opcount[i])

NFSD debug flags

/usr/include/linux/nfsd/debug.h (kernel 3.13.5)

/*
 * knfsd debug flags
 */
#define NFSDDBG_SOCK            0x0001
#define NFSDDBG_FH              0x0002
#define NFSDDBG_EXPORT          0x0004
#define NFSDDBG_SVC             0x0008
#define NFSDDBG_PROC            0x0010
#define NFSDDBG_FILEOP          0x0020
#define NFSDDBG_AUTH            0x0040
#define NFSDDBG_REPCACHE        0x0080
#define NFSDDBG_XDR             0x0100
#define NFSDDBG_LOCKD           0x0200
#define NFSDDBG_ALL             0x7FFF
#define NFSDDBG_NOCHANGE        0xFFFF

NFS debug flags

/usr/include/linux/nfs_fs.h (kernel 3.13.5)

/*
 * NFS debug flags
 */
#define NFSDBG_VFS              0x0001
#define NFSDBG_DIRCACHE         0x0002
#define NFSDBG_LOOKUPCACHE      0x0004
#define NFSDBG_PAGECACHE        0x0008
#define NFSDBG_PROC             0x0010
#define NFSDBG_XDR              0x0020
#define NFSDBG_FILE             0x0040
#define NFSDBG_ROOT             0x0080
#define NFSDBG_CALLBACK         0x0100
#define NFSDBG_CLIENT           0x0200
#define NFSDBG_MOUNT            0x0400
#define NFSDBG_FSCACHE          0x0800
#define NFSDBG_PNFS             0x1000
#define NFSDBG_PNFS_LD          0x2000
#define NFSDBG_STATE            0x4000
#define NFSDBG_ALL              0xFFFF

NLM debug flags

/usr/include/linux/lockd/debug.h (kernel 3.13.5)

/*
 * Debug flags
 */
#define NLMDBG_SVC              0x0001
#define NLMDBG_CLIENT           0x0002
#define NLMDBG_CLNTLOCK         0x0004
#define NLMDBG_SVCLOCK          0x0008
#define NLMDBG_MONITOR          0x0010
#define NLMDBG_CLNTSUBS         0x0020
#define NLMDBG_SVCSUBS          0x0040
#define NLMDBG_HOSTCACHE        0x0080
#define NLMDBG_XDR              0x0100
#define NLMDBG_ALL              0x7fff

RPC debug flags

/usr/include/linux/sunrpc/debug.h (kernel 3.13.5)

/*
 * RPC debug facilities
 */
#define RPCDBG_XPRT             0x0001
#define RPCDBG_CALL             0x0002
#define RPCDBG_DEBUG            0x0004
#define RPCDBG_NFS              0x0008
#define RPCDBG_AUTH             0x0010
#define RPCDBG_BIND             0x0020
#define RPCDBG_SCHED            0x0040
#define RPCDBG_TRANS            0x0080
#define RPCDBG_SVCXPRT          0x0100
#define RPCDBG_SVCDSP           0x0200
#define RPCDBG_MISC             0x0400
#define RPCDBG_CACHE            0x0800
#define RPCDBG_ALL              0x7fff

General Notes

  • While the number of threads can be increased at runtime via an echo to /proc/fs/nfsd/threads, the cache size (double the threads, see the ra line of /proc/net/rpc/nfsd) is not dynamic. The NFS daemon must be restarted with the new thread size during initialization (/etc/sysconfig/nfs on RHEL and CentOS) in order for the thread cache to properly adjust.

References

Bugs