tengel/papyri
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
papyri/md/nfs_debugging.md
tengel e8fb7b288e initial import
2024-03-20 11:40:22 -05:00

248 lines
8 KiB
Markdown
Raw Permalink Blame History

# NFS Debugging
## Contents
- [Userspace Tools](#userspace-tools)
- [Kernel Interfaces](#kernel-interfaces)
- [NFSD debug flags](#nfsd-debug-flags)
- [NFS debug flags](#nfs-debug-flags)
- [NLM debug flags](#nlm-debug-flags)
- [RPC debug flags](#rpc-debug-flags)
- [General Notes](#general-notes)
- [References](#references)
- [Bugs](#bugs)
## Userspace Tools
Using `rpcdebug` is the easiest way to manipulate the kernel interfaces in place of echoing bitmasks to `/proc`.
| **Option** | **Description** |
| ----------- | --------------------------------------------- |
| `-c` | Clear the given debug flags |
| `-s` | Set the given debug flags |
| `-m module` | Specify which module's flags to set or clear. |
| `-v` | Increase the verbosity of rpcdebug's output |
| `-h` | Print a help message and exit |
| `-vh` | Print the available debug flags |
For the `-m` option, the available modules are:
| **Module** | **Description** |
| ---------- | ------------------------------------------------------------- |
| nfsd | The NFS server |
| nfs | The NFS client |
| nlm | The Network Lock Manager, in either an NFS client or server |
| rpc | The Remote Procedure Call module, either NFS client or server |
Examples:
```
rpcdebug -m rpc -s all # sets all debug flags for RPC
rpcdebug -m rpc -c all # clears all debug flags for RPC
rpcdebug -m nfsd -s all # sets all debug flags for NFS Server
rpcdebug -m nfsd -c all # clears all debug flags for NFS Server
```
## Kernel Interfaces
A bitmask of the debug flags can be echoed into the interface to enable output to syslog; 0 is the default:
```
/proc/sys/sunrpc/nfsd_debug
/proc/sys/sunrpc/nfs_debug
/proc/sys/sunrpc/nlm_debug
/proc/sys/sunrpc/rpc_debug
```
Sysctl controls are registered for these interfaces, so they can be used instead of echo:
```
sysctl -w sunrpc.rpc_debug=1023
sysctl -w sunrpc.rpc_debug=0
sysctl -w sunrpc.nfsd_debug=1023
sysctl -w sunrpc.nfsd_debug=0
```
At runtime the server holds information that can be examined:
```
grep . /proc/net/rpc/*/content
cat /proc/fs/nfs/exports
cat /proc/net/rpc/nfsd
ls -l /proc/fs/nfsd
```
A rundown of `/proc/net/rpc/nfsd` (the userspace tool `nfsstat` pretty-prints this info):
```
/proc/net/rpc/nfsd
* rc (reply cache): <hits> <misses> <nocache>
- hits: client it's retransmitting
- misses: a operation that requires caching
- nocache: a operation that no requires caching
* fh (filehandle): <stale> <total-lookups> <anonlookups> <dir-not-in-cache> <nodir-not-in-cache>
- stale: file handle errors
- total-lookups, anonlookups, dir-not-in-cache, nodir-not-in-cache
. always seem to be zeros
* io (input/output): <bytes-read> <bytes-written>
- bytes-read: bytes read directly from disk
- bytes-written: bytes written to disk
* th (threads): <threads> <fullcnt> <10%-20%> <20%-30%> ... <90%-100%> <100%>
- threads: number of nfsd threads
- fullcnt: number of times that the last 10% of threads are busy
- 10%-20%, 20%-30% ... 90%-100%: 10 numbers representing 10-20%, 20-30% to 100%
. Counts the number of times a given interval are busy
* ra (read-ahead): <cache-size> <10%> <20%> ... <100%> <not-found>
- cache-size: always the double of number threads
- 10%, 20% ... 100%: how deep it found what was looking for
- not-found: not found in the read-ahead cache
* net: <netcnt> <netudpcnt> <nettcpcnt> <nettcpconn>
- netcnt: counts every read
- netudpcnt: counts every UDP packet it receives
- nettcpcnt: counts every time it receives data from a TCP connection
- nettcpconn: count every TCP connection it receives
* rpc: <rpccnt> <rpcbadfmt+rpcbadauth+rpcbadclnt> <rpcbadfmt> <rpcbadauth> <rpcbadclnt>
- rpccnt: counts all rpc operations
- rpcbadfmt: counts if while processing a RPC it encounters the following errors:
. err_bad_dir, err_bad_rpc, err_bad_prog, err_bad_vers, err_bad_proc, err_bad
- rpcbadauth: bad authentication
. does not count if you try to mount from a machine that it's not in your exports file
- rpcbadclnt: unused
* procN (N = vers): <vs_nproc> <null> <getattr> <setattr> <lookup> <access> <readlink> <read> <write> <create> <mkdir> <symlink> <mknod> <remove> <rmdir> <rename> <link> <readdir> <readdirplus> <fsstat> <fsinfo> <pathconf> <commit>
- vs_nproc: number of procedures for NFS version
. v2: nfsproc.c, 18
. v3: nfs3proc.c, 22
- v4, nfs4proc.c, 2
- statistics: generated from NFS operations at runtime
* proc4ops: <ops> <x..y>
- ops: the definition of LAST_NFS4_OP, OP_RELEASE_LOCKOWNER = 39, plus 1 (so 40); defined in nfs4.h
- x..y: the array of nfs_opcount up to LAST_NFS4_OP (nfsdstats.nfs4_opcount[i])
```
## NFSD debug flags
```
/usr/include/linux/nfsd/debug.h (kernel 3.13.5)
/*
* knfsd debug flags
*/
#define NFSDDBG_SOCK 0x0001
#define NFSDDBG_FH 0x0002
#define NFSDDBG_EXPORT 0x0004
#define NFSDDBG_SVC 0x0008
#define NFSDDBG_PROC 0x0010
#define NFSDDBG_FILEOP 0x0020
#define NFSDDBG_AUTH 0x0040
#define NFSDDBG_REPCACHE 0x0080
#define NFSDDBG_XDR 0x0100
#define NFSDDBG_LOCKD 0x0200
#define NFSDDBG_ALL 0x7FFF
#define NFSDDBG_NOCHANGE 0xFFFF
```
## NFS debug flags
```
/usr/include/linux/nfs_fs.h (kernel 3.13.5)
/*
* NFS debug flags
*/
#define NFSDBG_VFS 0x0001
#define NFSDBG_DIRCACHE 0x0002
#define NFSDBG_LOOKUPCACHE 0x0004
#define NFSDBG_PAGECACHE 0x0008
#define NFSDBG_PROC 0x0010
#define NFSDBG_XDR 0x0020
#define NFSDBG_FILE 0x0040
#define NFSDBG_ROOT 0x0080
#define NFSDBG_CALLBACK 0x0100
#define NFSDBG_CLIENT 0x0200
#define NFSDBG_MOUNT 0x0400
#define NFSDBG_FSCACHE 0x0800
#define NFSDBG_PNFS 0x1000
#define NFSDBG_PNFS_LD 0x2000
#define NFSDBG_STATE 0x4000
#define NFSDBG_ALL 0xFFFF
```
## NLM debug flags
```
/usr/include/linux/lockd/debug.h (kernel 3.13.5)
/*
* Debug flags
*/
#define NLMDBG_SVC 0x0001
#define NLMDBG_CLIENT 0x0002
#define NLMDBG_CLNTLOCK 0x0004
#define NLMDBG_SVCLOCK 0x0008
#define NLMDBG_MONITOR 0x0010
#define NLMDBG_CLNTSUBS 0x0020
#define NLMDBG_SVCSUBS 0x0040
#define NLMDBG_HOSTCACHE 0x0080
#define NLMDBG_XDR 0x0100
#define NLMDBG_ALL 0x7fff
```
## RPC debug flags
```
/usr/include/linux/sunrpc/debug.h (kernel 3.13.5)
/*
* RPC debug facilities
*/
#define RPCDBG_XPRT 0x0001
#define RPCDBG_CALL 0x0002
#define RPCDBG_DEBUG 0x0004
#define RPCDBG_NFS 0x0008
#define RPCDBG_AUTH 0x0010
#define RPCDBG_BIND 0x0020
#define RPCDBG_SCHED 0x0040
#define RPCDBG_TRANS 0x0080
#define RPCDBG_SVCXPRT 0x0100
#define RPCDBG_SVCDSP 0x0200
#define RPCDBG_MISC 0x0400
#define RPCDBG_CACHE 0x0800
#define RPCDBG_ALL 0x7fff
```
## General Notes
- While the number of threads can be increased at runtime via an echo to `/proc/fs/nfsd/threads`, the cache size (double the threads, see the `ra` line of /proc/net/rpc/nfsd) is not dynamic. The NFS daemon must be restarted with the new thread size during initialization (`/etc/sysconfig/nfs` on RHEL and CentOS) in order for the thread cache to properly adjust.
## References
- <https://github.com/torvalds/linux/tree/master/include/linux>
- <http://linux.die.net/man/8/rpcdebug>
- <http://utcc.utoronto.ca/~cks/space/blog/linux/NFSClientDebuggingBits>
- <http://www.novell.com/support/kb/doc.php?id=7011571>
- <http://stromberg.dnsalias.org/~strombrg/NFS-troubleshooting-2.html>
- <http://www.opensubscriber.com/message/nfs@lists.sourceforge.net/7833588.html>
## Bugs
- <https://bugzilla.redhat.com/show_bug.cgi?id=465412>
Reference in a new issue View git blame Copy permalink