# NFS Debugging

## Contents

  - [Userspace Tools](#userspace-tools)
  - [Kernel Interfaces](#kernel-interfaces)
  - [NFSD debug flags](#nfsd-debug-flags)
  - [NFS debug flags](#nfs-debug-flags)
  - [NLM debug flags](#nlm-debug-flags)
  - [RPC debug flags](#rpc-debug-flags)
  - [General Notes](#general-notes)
  - [References](#references)
  - [Bugs](#bugs)


## Userspace Tools

Using `rpcdebug` is the easiest way to manipulate the kernel interfaces in place of echoing bitmasks to `/proc`.

| **Option**  | **Description**                               |
| ----------- | --------------------------------------------- |
| `-c`        | Clear the given debug flags                   |
| `-s`        | Set the given debug flags                     |
| `-m module` | Specify which module's flags to set or clear. |
| `-v`        | Increase the verbosity of rpcdebug's output   |
| `-h`        | Print a help message and exit                 |
| `-vh`       | Print the available debug flags               |

For the `-m` option, the available modules are:

| **Module** | **Description**                                               |
| ---------- | ------------------------------------------------------------- |
| nfsd       | The NFS server                                                |
| nfs        | The NFS client                                                |
| nlm        | The Network Lock Manager, in either an NFS client or server   |
| rpc        | The Remote Procedure Call module, either NFS client or server |

Examples:

```
rpcdebug -m rpc -s all   # sets all debug flags for RPC
rpcdebug -m rpc -c all   # clears all debug flags for RPC

rpcdebug -m nfsd -s all   # sets all debug flags for NFS Server
rpcdebug -m nfsd -c all   # clears all debug flags for NFS Server
```


## Kernel Interfaces

A bitmask of the debug flags can be echoed into the interface to enable output to syslog; 0 is the default:

```
/proc/sys/sunrpc/nfsd_debug
/proc/sys/sunrpc/nfs_debug
/proc/sys/sunrpc/nlm_debug
/proc/sys/sunrpc/rpc_debug
```

Sysctl controls are registered for these interfaces, so they can be used instead of echo:

```
sysctl -w sunrpc.rpc_debug=1023
sysctl -w sunrpc.rpc_debug=0

sysctl -w sunrpc.nfsd_debug=1023
sysctl -w sunrpc.nfsd_debug=0
```

At runtime the server holds information that can be examined:

```
grep . /proc/net/rpc/*/content
cat /proc/fs/nfs/exports
cat /proc/net/rpc/nfsd
ls -l /proc/fs/nfsd
```

A rundown of `/proc/net/rpc/nfsd` (the userspace tool `nfsstat` pretty-prints this info):

```
/proc/net/rpc/nfsd

* rc (reply cache): <hits> <misses> <nocache>
- hits: client it's retransmitting
- misses: a operation that requires caching
- nocache: a operation that no requires caching

* fh (filehandle): <stale> <total-lookups> <anonlookups> <dir-not-in-cache> <nodir-not-in-cache>
- stale: file handle errors
- total-lookups, anonlookups, dir-not-in-cache, nodir-not-in-cache
  . always seem to be zeros

* io (input/output): <bytes-read> <bytes-written>
- bytes-read: bytes read directly from disk
- bytes-written: bytes written to disk

* th (threads): <threads> <fullcnt> <10%-20%> <20%-30%> ... <90%-100%> <100%>
- threads: number of nfsd threads
- fullcnt: number of times that the last 10% of threads are busy
- 10%-20%, 20%-30% ... 90%-100%: 10 numbers representing 10-20%, 20-30% to 100%
  . Counts the number of times a given interval are busy

* ra (read-ahead): <cache-size> <10%> <20%> ... <100%> <not-found>
- cache-size: always the double of number threads
- 10%, 20% ... 100%: how deep it found what was looking for
- not-found: not found in the read-ahead cache

* net: <netcnt> <netudpcnt> <nettcpcnt> <nettcpconn>
- netcnt: counts every read
- netudpcnt: counts every UDP packet it receives
- nettcpcnt: counts every time it receives data from a TCP connection
- nettcpconn: count every TCP connection it receives

* rpc: <rpccnt> <rpcbadfmt+rpcbadauth+rpcbadclnt> <rpcbadfmt> <rpcbadauth> <rpcbadclnt>
- rpccnt: counts all rpc operations
- rpcbadfmt: counts if while processing a RPC it encounters the following errors:
  . err_bad_dir, err_bad_rpc, err_bad_prog, err_bad_vers, err_bad_proc, err_bad
- rpcbadauth: bad authentication
  . does not count if you try to mount from a machine that it's not in your exports file
- rpcbadclnt: unused

* procN (N = vers): <vs_nproc> <null> <getattr> <setattr> <lookup> <access> <readlink> <read> <write> <create> <mkdir> <symlink> <mknod> <remove> <rmdir> <rename> <link> <readdir> <readdirplus> <fsstat> <fsinfo> <pathconf> <commit>
- vs_nproc: number of procedures for NFS version
  . v2: nfsproc.c, 18
  . v3: nfs3proc.c, 22
  - v4, nfs4proc.c, 2
- statistics: generated from NFS operations at runtime

* proc4ops: <ops> <x..y>
- ops: the definition of LAST_NFS4_OP, OP_RELEASE_LOCKOWNER = 39, plus 1 (so 40); defined in nfs4.h
- x..y: the array of nfs_opcount up to LAST_NFS4_OP (nfsdstats.nfs4_opcount[i])
```


## NFSD debug flags

```
/usr/include/linux/nfsd/debug.h (kernel 3.13.5)

/*
 * knfsd debug flags
 */
#define NFSDDBG_SOCK            0x0001
#define NFSDDBG_FH              0x0002
#define NFSDDBG_EXPORT          0x0004
#define NFSDDBG_SVC             0x0008
#define NFSDDBG_PROC            0x0010
#define NFSDDBG_FILEOP          0x0020
#define NFSDDBG_AUTH            0x0040
#define NFSDDBG_REPCACHE        0x0080
#define NFSDDBG_XDR             0x0100
#define NFSDDBG_LOCKD           0x0200
#define NFSDDBG_ALL             0x7FFF
#define NFSDDBG_NOCHANGE        0xFFFF
```


## NFS debug flags

```
/usr/include/linux/nfs_fs.h (kernel 3.13.5)

/*
 * NFS debug flags
 */
#define NFSDBG_VFS              0x0001
#define NFSDBG_DIRCACHE         0x0002
#define NFSDBG_LOOKUPCACHE      0x0004
#define NFSDBG_PAGECACHE        0x0008
#define NFSDBG_PROC             0x0010
#define NFSDBG_XDR              0x0020
#define NFSDBG_FILE             0x0040
#define NFSDBG_ROOT             0x0080
#define NFSDBG_CALLBACK         0x0100
#define NFSDBG_CLIENT           0x0200
#define NFSDBG_MOUNT            0x0400
#define NFSDBG_FSCACHE          0x0800
#define NFSDBG_PNFS             0x1000
#define NFSDBG_PNFS_LD          0x2000
#define NFSDBG_STATE            0x4000
#define NFSDBG_ALL              0xFFFF
```


## NLM debug flags

```
/usr/include/linux/lockd/debug.h (kernel 3.13.5)

/*
 * Debug flags
 */
#define NLMDBG_SVC              0x0001
#define NLMDBG_CLIENT           0x0002
#define NLMDBG_CLNTLOCK         0x0004
#define NLMDBG_SVCLOCK          0x0008
#define NLMDBG_MONITOR          0x0010
#define NLMDBG_CLNTSUBS         0x0020
#define NLMDBG_SVCSUBS          0x0040
#define NLMDBG_HOSTCACHE        0x0080
#define NLMDBG_XDR              0x0100
#define NLMDBG_ALL              0x7fff
```


## RPC debug flags

```
/usr/include/linux/sunrpc/debug.h (kernel 3.13.5)

/*
 * RPC debug facilities
 */
#define RPCDBG_XPRT             0x0001
#define RPCDBG_CALL             0x0002
#define RPCDBG_DEBUG            0x0004
#define RPCDBG_NFS              0x0008
#define RPCDBG_AUTH             0x0010
#define RPCDBG_BIND             0x0020
#define RPCDBG_SCHED            0x0040
#define RPCDBG_TRANS            0x0080
#define RPCDBG_SVCXPRT          0x0100
#define RPCDBG_SVCDSP           0x0200
#define RPCDBG_MISC             0x0400
#define RPCDBG_CACHE            0x0800
#define RPCDBG_ALL              0x7fff
```


## General Notes

  - While the number of threads can be increased at runtime via an echo to `/proc/fs/nfsd/threads`, the cache size (double the threads, see the `ra` line of /proc/net/rpc/nfsd) is not dynamic. The NFS daemon must be restarted with the new thread size during initialization (`/etc/sysconfig/nfs` on RHEL and CentOS) in order for the thread cache to properly adjust.


## References

  - <https://github.com/torvalds/linux/tree/master/include/linux>
  - <http://linux.die.net/man/8/rpcdebug>
  - <http://utcc.utoronto.ca/~cks/space/blog/linux/NFSClientDebuggingBits>
  - <http://www.novell.com/support/kb/doc.php?id=7011571>
  - <http://stromberg.dnsalias.org/~strombrg/NFS-troubleshooting-2.html>
  - <http://www.opensubscriber.com/message/nfs@lists.sourceforge.net/7833588.html>


## Bugs

  - <https://bugzilla.redhat.com/show_bug.cgi?id=465412>