tengel/papyri
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

initial import

Browse source
This commit is contained in:
tengel 2024-03-20 11:40:22 -05:00
parent e3e0eb7656
commit e8fb7b288e
43 changed files with 14946 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

49
INDEX.md Normal file
View file

@ -0,0 +1,49 @@
# Index
- [Active Directory with Winbind](md/active_directory_with_winbind.md)
- [Arch UEFI Installation](md/arch_uefi_installation.md)
- [CIFS Client Setup](md/cifs_client_setup.md)
- [Compose Key Sequences](md/compose_key_sequences.md)
- [Conversion Table](md/conversion_table.md)
- [Debian BOINC Client](md/debian_boinc_client.md)
- [Debian Server Setup](md/debian_server_setup.md)
- [Debian Tor Relay](md/debian_tor_relay.md)
- [Device Mapper Mechanics](md/device_mapper_mechanics.md)
- [Device Mapper Multipath](md/device_mapper_multipath.md)
- [DRBD Build Steps](md/drbd_build_steps.md)
- [Fonts and Linux](md/fonts_and_linux.md)
- [GlusterFS Build Steps](md/glusterfs_build_steps.md)
- [Grub 2 Info](md/grub_2_info.md)
- [Jumbo Frames](md/jumbo_frames.md)
- [Kernel Module Weak Updates](md/kernel_module_weak_updates.md)
- [Linux Partitioning](md/linux_partitioning.md)
- [Linux x86 Storage](md/linux_x86_storage.md)
- [LVM Mechanics](md/lvm_mechanics.md)
- [LVM Snapshot Merging](md/lvm_snapshot_merging.md)
- [MongoDB Basics](md/mongodb_basics.md)
- [Network Quick Reference](md/network_quick_reference.md)
- [NFS Debugging](md/nfs_debugging.md)
- [NFS Setup](md/nfs_setup.md)
- [Oracle Environment](md/oracle_environment.md)
- [RAID Penalties](md/raid_penalties.md)
- [Reducing the root LV](md/reducing_the_root_lv.md)
- [RHCS Mechanics](md/rhcs_mechanics.md)
- [RHEL7 Networking](md/rhel7_networking.md)
- [SCSI Sense Data](md/scsi_sense_data.md)
- [SMTP Relay](md/smtp_relay.md)
- [Stunnel Setup](md/stunnel_setup.md)
- [System Backup](md/system_backup.md)
- [systemd Mechanics](md/systemd_mechanics.md)
- [Tomcat Configuration](md/tomcat_configuration.md)
- [Tomcat Logging](md/tomcat_logging.md)
- [Tomcat Mechanics](md/tomcat_mechanics.md)
- [Tomcat Packaging](md/tomcat_packaging.md)
- [Tuning nf conntrack](md/tuning_nf_conntrack.md)
- [Understanding Swap Use](md/understanding_swap_use.md)
- [Vsftpd Setup](md/vsftpd_setup.md)
- [XFS Info](md/xfs_info.md)
# Licenses
* SPDX-License-Identifier: CC-BY-SA-4.0
* SPDX-License-Identifier: MIT

313
md/active_directory_with_winbind.md Normal file
View file

@ -0,0 +1,313 @@
# Active Directory with Winbind
## Contents
- [Prerequisites](#prerequisites)
- [AD Setup Information](#ad-setup-information)
- [Implementation](#implementation)
- [Install RPMs](#install-rpms)
- [DNS Configuration](#dns-configuration)
- [Configure Kerberos](#configure-kerberos)
- [Get a Kerberos ticket](#get-a-kerberos-ticket)
- [List the ticket provided](#list-the-ticket-provided)
- [Destroy the ticket](#destroy-the-ticket)
- [Samba Configuration](#samba-configuration)
- [Join the domain](#join-the-domain)
- [Configure winbind authentication](#configure-winbind-authentication)
- [PAM Configuration](#pam-configuration)
- [RHE5 and RHEL6](#rhel5-and-rhel6)
- [RHEL6 Only](#rhel6-only)
- [Parent Home Directory](#parent-home-directory)
- [Testing](#testing)
- [Cached Logins](#cached-logins)
- [User crontabs](#user-crontabs)
- [References](#references)
## Prerequisites
### AD Setup Information
Needed information:
- NETBIOS name of one or more domain controllers
- DNS IPs of same servers that resolve lookups
- Admin level user already in the AD
Examples used in this article:
- AD1.DOMAIN.LOCAL, AD2.DOMAIN.LOCAL
- 192.168.100.10, 192.168.100.20
- 'admin'
## Implementation
### Install RPMs
Standard YUM install:
```
RHEL6:
# yum install samba-winbind samba-winbind-clients krb5-workstation krb5-libs
RHEL5:
# yum install samba3x-winbind samba3x-client krb5-workstation krb5-libs
```
Notes:
- krb5-workstation adds `/usr/kerberos/bin` to your `$PATH`, you may need to log in again so that `kinit` and other apps are now found
- RHEL5 `winbind` is not Windows 2008R2+ friendly. Use `winbind3x` (samba3x) RPMs instead
### DNS Configuration
```
/etc/resolv.conf
nameserver 192.168.100.10
nameserver 192.168.100.20
search DOMAIN.LOCAL
```
### Configure Kerberos
```
/etc/krb5.conf
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = DOMAIN.LOCAL
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
forwardable = yes
[realms]
DOMAIN.LOCAL = {
kdc = AD1.DOMAIN.LOCAL:88
kdc = AD2.DOMAIN.LOCAL:88
admin_server = AD1.DOMAIN.LOCAL:749
admin_server = AD2.DOMAIN.LOCAL:749
}
[domain_realm]
.DOMAIN.local = DOMAIN.LOCAL
DOMAIN.local = DOMAIN.LOCAL
[appdefaults]
pam = {
debug = false
ticket_lifetime = 36000
renew_lifetime = 36000
forwardable = true
krb4_convert = false
}
```
#### Get a Kerberos ticket
```
# kinit admin@DOMAIN.LOCAL
```
#### List the ticket provided
```
# klist
```
#### Destroy the ticket
```
# kdestroy
```
### Samba Configuration
```
/etc/samba/smb.conf
[global]
workgroup = DOMAIN
interfaces = 127.0.0.1 eth0
bind interfaces only = true
security = ads
passdb backend = tdbsam
template shell = /bin/bash
template homedir = /home/%D/%U
realm = DOMAIN.LOCAL
password server = AD1.DOMAIN.LOCAL, AD2.DOMAIN.LOCAL
winbind use default domain = yes
winbind enum users = yes
winbind enum groups = yes
winbind refresh tickets = yes
idmap uid = 16777216-33554431
idmap gid = 16777216-33554431
printing = cups
printcap name = cups
load printers = no
```
If required you can assign a name to the server. This is useful since NT has a limit of 15 chars to the servers. Just add:
```
netbios name = MYNTNAME
```
#### Join the domain
```
# net ads join -U admin
```
Example session:
```
# net ads join -U admin
Enter admin's password:
Using short domain name -- DOMAIN
Joined 'MYSERVER' to realm 'domain.local'
[2012/03/04 06:06:06.123456, 0] libads/kerberos.c:333(ads_kinit_password)
kerberos_kinit_password MYSERVER$@DOMAIN.LOCAL failed: Client not found in Kerberos database
DNS update failed!
```
> This error message is expected, the server joined the domain, but the AD DNS was not updated for your server.
#### Configure winbind authentication
```
# authconfig-tui
```
1. Select **Use Winbind** under the User Information section
2. Select **Use MD5 Passwords** under the Authentication section
3. Select **Use Shadow Passwords** under the Authentication section
4. Select **Use Winbind Authentication** under the Authentication section
5. Select **Local Authentication is sufficient** under the Authentication section
6. Click **Next**
7. Click **OK** (*not* Join Domain\!)
### PAM Configuration
The system may need to be updated to make two configuration changes; it's possible one or both of these are already taken care of however. The first change is to update the existing line for `pam_winbind.so` and add extra config; the second is to add/update the `pam_mkhomedir.so` line to have the user's home directory create itself.
> Whenever editing PAM config files, ALWAYS test logins in a second terminal before you log out of the editing session. Breaking a PAM config file can cause _root_ to be locked out and require single-user mode to rescue.
#### RHEL5 and RHEL6
This is a pseudo diff of the changes to be made; examine the existing file and apply only the needed values as shown.
```
/etc/pam.d/system-auth
< auth sufficient pam_winbind.so use_first_pass
---
> auth sufficient pam_winbind.so krb5_auth krb5_ccache_type=FILE use_first_pass
> session required pam_mkhomedir.so skel=/etc/skel umask=0022
```
#### RHEL6 Only
**Configure**
In RHEL5/CentOS5 all the various other PAM configuration files _sub-include_ system-auth; in RHEL6 this was split out into two different files; some sub-include `system-auth` (like sudo), some sub-include `password-auth` (like sshd). Changing both files is required.
```
/etc/pam.d/password-auth
make the exact same changes as outlined above
```
### Parent Home Directory
All DOMAIN homedirs will be created below this dir by pam\_mkhomedir.so (via smb.conf `template homedir` variable):
```
# mkdir /home/DOMAIN
# chcon --reference=/home /home/DOMAIN
```
## Testing
Test the basics:
```
# wbinfo -u
# wbinfo -g
# ssh DOMAIN\\admin@localhost
```
> Due to an interesting conflict between the presence of local user 'admin' in `/etc/passwd` (with /home/admin defined) and the attempt to use /home/DOMAIN/admin during a DOMAIN login you can get curious permission denied results. It's best to test DOMAIN logins with a username _other_ than one that exists in /etc/passwd on the local machine to avoid the DOMAIN login conflict with pam\_mkhomedir.
## Cached Logins
The pam\_winbind.so module supports cached logins - this can be handy if the Active Directory server(s) become unavailable, you'll still be able to log into Linux. It is very useful to include a cache time otherwise the cache seems _not_ to be updated regardless of the default value(300 secs). This parameter specifies the number of seconds the winbindd will cache user and group information before querying an AD server again.
In the same global section as defined above, add a new directive as shown:
```
/etc/samba/smb.conf
[global]
...
winbind offline logon = yes
winbind cache time = 600
...
```
It's possible that this file may not exist; create it if needed:
```
/etc/security/pam_winbind.conf
[global]
cached_login = yes
```
Perform a standard Winbind restart and test things out:
```
# service winbind restart
# smbcontrol winbind offline
# wbinfo --online-status
BUILTIN : online
MYSERVER : online
DOMAIN : offline
# ssh DOMAIN\\username@localhost
Domain Controller unreachable, using cached credentials instead. Network resources may be unavailable
...
# smbcontrol winbind online
```
Some items - such as groups - don't get added to the cache until there is a successful login when things are in online mode; this may affect tools like sudo or sshd if they are configured to allow/restrict access based on group level membership. Your exact situation will determine any further tweaks needed to fully support offline access in an emergency. This can be fixed with the use of winbind cache time as noted above. If required, the cache can be deleted by removing the `/var/lib/samba/*.tdb` files.
## User crontabs
There is a problem with the vixie-cron (RHEL5) and cronie \<= 1.4.7 (RHEL6) packages and crontabs which belong to remote network users; when CROND starts up at boot it cannot "see" these remote users when scanning the `/var/spool/cron/` crontabs as networking is not online yet; as a consequence it places each unmatched crontab in an "Orphan" list and never checks again. Restarting CROND after Winbind/LDAP/NIS/etc. are up will work correctly, so one possible solution if this is needed is to place a 'service crond restart' in `/etc/rc.d/rc.local` if you must use this type of crontab.
This issue was fixed in 1.4.8 release of cronie; if it is not yet available any RPM upgrade will have to be manually rebuilt from the Koji system (Fedora packaging) to obtain a newer release.
- Git commit: <https://git.fedorahosted.org/cgit/cronie.git/commit/?id=8b407876f276f96914111bd9954f21f627db7b11>
- Koji package: <http://koji.fedoraproject.org/koji/packageinfo?packageID=5724>
The cronie package in RHEL6 replaces the vixie-cron and anacron packages from RHEL5. Compiling (rebuilding) cronie for RHEL5 and doing a manual package swap may work but is untested. It would be best to create a local user account to run the crontabs instead of using a remote network user until (and if) Red Hat releases packages which address this issue.
## References
- <https://wiki.samba.org/index.php/PAM_Offline_Authentication>

120
md/arch_uefi_installation.md Normal file
View file

@ -0,0 +1,120 @@
# Arch UEFI Installation
## Contents
- [Overview](#overview)
- [Process](#process)
## Overview
A concise example of how to use a UEFI system with GPT disk partitioning with [Arch](https://www.archlinux.org); the intent is to demonstrate how the EFI partition works, how it's mounted, and how [GRUB](http://www.gnu.org/software/grub/) is configured.
This example was designed using a [Virtualbox](https://www.virtualbox.org/) host with the guest VM in EFI mode `VM -> Settings -> System -> Enable EFI`, then booting the standard [Arch ISO](https://www.archlinux.org/download/) and using UEFI mode install.
## Process
First, partition your device in GPT format (`gdisk` or `parted`) like so:
> Note that gdisk and parted display the UEFI (`/dev/sda1` below) in different ways -- gdisk shows it as an ESP (EFI System Partition), parted likes to show it instead with flags "boot,esp" -- this is all the same, it's type ef00 under the hood on a GPT disk.
```
/dev/sda1 size:200M type:ef00 "EFI System Partition" or "boot,esp"
/dev/sda2 size:500M type:8300 "Linux Filesystem" or "Linux"
/dev/sda3 size:rest type:8e00 "Linux LVM"
```
Now create `/dev/sda3` as LVM:
```
pvcreate /dev/sda3
vgcreate vglocal00 /dev/sda3
lvcreate -L 1G -n swap00 vglocal00
lvcreate -l 100%FREE -n root00 vglocal00
```
Make your swap, ext4 and vfat for EFI:
```
mkswap /dev/vglocal00/swap00
mkfs.vfat /dev/sda1
mkfs.ext4 /dev/sda2
mkfs.ext4 /dev/vglocal00/root00
```
Mount everything - notice how `/dev/sda1` is a /boot/efi VFAT (aka FAT32) partition type:
```
swapon /dev/vglocal00/swap00
mount /dev/vglocal00/root00 /mnt
mkdir /mnt/boot
mount /dev/sda2 /mnt/boot
mkdir /mnt/boot/efi
mount /dev/sda1 /mnt/boot/efi
```
Pacstrap the core packages and chroot into the mount:
```
pacstrap /mnt base
genfstab -p /mnt >> /mnt/etc/fstab
arch-chroot /mnt
```
Prep the system with all the usual things - adjust to your locale as desired:
```
export LANG="en_US.UTF-8"
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
locale-gen
cat << EOF > /etc/locale.conf
LANG="en_US.UTF-8"
LC_COLLATE="C"
EOF
cat << EOF > /etc/vconsole.conf
KEYMAP="us"
FONT="eurlatgr"
EOF
ln -s /usr/share/zoneinfo/America/Chicago /etc/localtime
hwclock --systohc --utc
echo "toolbox" > /etc/hostname
hostname "toolbox"
```
Install grub, kernel headers, os-prober (so grub can see Windows, etc.) and the UEFI tools:
```
pacman -Sy --noconfirm
pacman -S --noconfirm grub linux-headers os-prober intel-ucode dosfstools efibootmgr
```
Add the mkinitcpio hook for LVM:
```
sed -i.bak -r 's/^HOOKS=(.*)block(.*)/HOOKS=\1block lvm2\2/g' /etc/mkinitcpio.conf
mkinitcpio -p linux
```
Install grub2 in UEFI mode and add the hack for some BIOSes which expect the boot bits in a specific place:
```
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=arch_grub --recheck --debug
grub-mkconfig -o /boot/grub/grub.cfg
mkdir /boot/efi/EFI/boot
cp /boot/efi/EFI/arch_grub/grubx64.efi /boot/efi/EFI/boot/bootx64.efi
```
Finally, set the password for root, back your way out and reboot:
```
passwd root
exit
umount -R /mnt
reboot
```
The guest VM is now a UEFI + GPT based Linux system; from here the process can be expanded to work with real live devices and systems, custom partitions and so forth.

546
md/cifs_client_setup.md Normal file
View file

@ -0,0 +1,546 @@
# CIFS Client Setup
## Contents
- [Overview](#overview)
- [Conventions Used](#conventions-used)
- [Software Installation](#software-installation)
- [Share Testing](#share-testing)
- [Client Configuration](#client-configuration)
- [ACL Testing](#acl-testing)
- [Kerberos Authentication](#kerberos-authentication)
- [Linux Packages](#linux-packages)
- [Domain Controller](#domain-controller)
- [Test the Domain](#test-the-domain)
- [Generate the Keytab](#generate-the-keytab)
- [Client KRB5 Setup](#client-krb5-setup)
- [Connect and Test](#connect-and-test)
- [CIFS Debugging](#cifs-debugging)
- [Packaging Bugs](#packaging-bugs)
- [Troubleshooting](#troubleshooting)
- [RHEL5 Kerberos Fails](#rhel5-kerberos-fails)
- [Kerberos Upcall Fails](#kerberos-upcall-fails)
- [Additional Reading](#additional-reading)
## Overview
Common Internet File System (**CIFS**) is the Windows analog to Network File System (**NFS**); to [quote Microsoft](http://technet.microsoft.com/en-us/library/cc939973.aspx):
> The _Common Internet File System_ (CIFS) is the standard way that computer users share files across corporate intranets and the Internet. An enhanced version of the Microsoft open, cross-platform Server Message Block (SMB) protocol, CIFS is a native file-sharing protocol in Windows 2000. CIFS defines a series of commands used to pass information between networked computers.
Linux users are used to the [Samba project](http://www.samba.org/) and it's suite of utilities which provide the client side connection to CIFS shares on the Windows server platform, allowing a Linux server to mount the remote Windows share for use.
## Conventions Used
Setting up a Windows server is not covered herein; this document is
using these conventions:
- **Release**: Windows 2012 R2 Server
- **Domain**: CIFSGROUP
- **Server**: CIFSSERVER, 192.168.5.5
- **CIFS User**: cifsuser / p@ssw0rd
- **Test User**: testuser / p@ssw0rd, used for testing additional ACLs
- **CIFS Share**: c:\\cifsdata\\ as 'cifsdata', full ownership by user 'cifsuser'
The above values are used in all examples below. A dummy file `server-file.txt` has been created in `c:\cifsdata\` and the _cifsuser_ been given ownership; this will allow for initial testing.
## Software Installation
The filesystem kernel module `cifs.ko` is already provided by the kernel on all distros, however the userspace utilities need to be installed. This primarily consists of `mount.cifs`, `cifs.idmap`, `setcifsacl` and `getcifsacl`. While not absolutely required, installing the `smbclient` utility is also recommended for debugging and troubleshooting.
Install the required packages:
```
# RHEL5 / CentOS5
# Note: RHEL/CentOS 5 do not provide setcifsacl/getcifsacl
yum install samba3x-client
# RHEL6 / CentOS6 / RHEL7 / CentOS7
yum install cifs-utils samba-client
# Debian / Ubuntu
apt-get update; apt-get install cifs-utils smbclient
# Arch
pacman -Sy; pacman -S cifs-utils smbclient
# openSUSE
zypper install cifs-utils samba-client
```
Select distros need a special service enabled to mount network filesystems at boot:
```
# RHEL5 / CentOS5 / RHEL6 / CentOS6
chkconfig netfs on
# Debian 7 - ignore the name, it works for CIFS too
insserv mountnfs.sh
```
Distros using **systemd** and **upstart** require no special service to be enabled. Some distros (such as Debian) may automatically start and enable the _Winbind_ daemon which is not being used, disable it:
```
insserv -r winbind
service winbind stop
```
## Share Testing
Before setting up the CIFS mount with the kernel, use `smbclient` to test that the share can be listed:
```
# smbclient -W CIFSGROUP -U CIFSSERVER\\cifsuser -L //192.168.5.5
Enter CIFSSERVER\cifsuser's password:
Domain=[CIFSSERVER] OS=[Windows Server 2012 R2 Standard 9600] Server=[Windows Server 2012 R2 Standard 6.3]
Sharename Type Comment
--------- ---- -------
ADMIN$ Disk Remote Admin
C$ Disk Default share
cifsdata Disk
IPC$ IPC Remote IPC
```
If this is working correctly, completely connect to the share and ensure the `server-file.txt` can be seen:
```
# smbclient -W CIFSGROUP -U CIFSSERVER\\cifsuser //192.168.5.5/cifsdata
Enter CIFSSERVER\cifsuser's password:
Domain=[CIFSSERVER] OS=[Windows Server 2012 R2 Standard 9600] Server=[Windows Server 2012 R2 Standard 6.3]
smb: \> ls
. D 0 Sun Sep 21 14:54:42 2014
.. D 0 Sun Sep 21 14:54:42 2014
server-file.txt A 0 Sun Sep 21 14:54:25 2014
40957 blocks of size 1048576. 23832 blocks available
smb: \> exit
```
## Client Configuration
First, create a file to hold the _user_ and _password_ values - the _domain_ can also be specified, however experience has shown that the _domain_ setting in this file doesn't always work depending on which release of the software is present. We will specify the _domain_ in the `/etc/fstab` config instead.
```
/etc/cifspw
user=cifsuser
password=p@ssw0rd
```
Secure the file against prying eyes:
```
chmod 0600 /etc/cifspw
```
Next, add an entry to `/etc/fstab` for the mount - CIFS and POSIX permissions/attributes differ greatly, so notice that we are going to set a handful of values that maps the remote CIFS share to Linux friendly values.
> Remember that the user permissions actually used are those on the remote server used to mount the share (_cifsuser_ in these examples); these uid/gid mappings are for the Linux side only to provide POSIX friendly uid/gid usage as a client. Regardless of what is specified here, the remote end will always use the mount user from `/etc/cifspw`.
Options:
- **uid**: user to map
- **gid**: group to map
- **domain**: remote CIFS domain
- **credentials**: name of the file with user/pass
- **iocharset**: which character set to use locally
- **file\_mode**: default file permissions
- **dir\_mode**: default directory permissions
- **\_netdev**: on startup ensure the network is up first
- **soft**: if the CIFS share goes away, don't hang Linux
The user _nobody_ exists on all distros, however on Debian/Ubuntu the group is _nogroup_ whereas on all other distros the group is _nobody_. These are being used as a generic mapping example - they can be any local Linux user/group needed to accomplish the mission, for example _apache_ or _www-data_ if configuring a webserver that needs to write to the share.
```
/etc/fstab
# every distro except Debian/Ubuntu
//192.168.5.5/cifsdata /data cifs uid=nobody,gid=nobody,domain=CIFSGROUP,credentials=/etc/cifspw,iocharset=utf8,file_mode=0644,dir_mode=0755,_netdev,soft 0 0
# Debian/Ubuntu only
//192.168.5.5/cifsdata /data cifs uid=nobody,gid=nogroup,domain=CIFSGROUP,credentials=/etc/cifspw,iocharset=utf8,file_mode=0644,dir_mode=0755,_netdev,soft 0 0
```
Mount the share and test that basic create and delete privileges are working as expected, while examining the permissions and uid/gid mappings:
```
mkdir /data
mount /data
touch /data/test-file
ls -l /data/
-rw-r--r-- 0 nobody nobody 0 Sep 21 14:54 server-file.txt
-rw-r--r-- 1 nobody nobody 0 Sep 21 14:59 test-file
```
## ACL Testing
The `setcifsacl` and `getcifsacl` userspace tools allow setting/getting the ACLs present on the remote end. If the client is connected to the domain [via Winbind](active_directory_with_winbind.md) then the remote user name can be used and the `cifs.idmap` infrastructure will handle the details.
However, for these examples we're not connected via Winbind, so we need to obtain the [Security Identifier](http://en.wikipedia.org/wiki/Security_Identifier) (**SID**) of the user on the domain. By far the easiest way to accomplish this is by [downloading PSUtils](http://technet.microsoft.com/en-us/sysinternals/bb897417.aspx) and using `psgetsid.exe` from PowerShell.
Retrieve the SID of _testuser_:
```
PS C:\PSTools> .\PsGetsid.exe testuser
SID for CIFSSERVER\testuser:
S-1-5-21-762712803-3572108623-4099884218-1003
```
From the client, attempt to set an ACL for _testuser_ and check it; you can also use the standard Windows properties Security tab to verify it worked as intended:
```
setcifsacl -a "ACL:S-1-5-21-762712803-3572108623-4099884218-1003:ALLOWED/I/FULL" /data/test-file
getcifsacl /data/test-file
REVISION:0x1
CONTROL:0x8004
OWNER:S-1-5-21-762712803-3572108623-4099884218-1001
GROUP:S-1-5-21-762712803-3572108623-4099884218-513
ACL:S-1-5-21-762712803-3572108623-4099884218-500:ALLOWED/I/FULL
ACL:S-1-5-32-544:ALLOWED/I/FULL
ACL:S-1-5-21-762712803-3572108623-4099884218-1001:ALLOWED/I/FULL
ACL:S-1-5-18:ALLOWED/I/FULL
ACL:S-1-5-21-762712803-3572108623-4099884218-1003:ALLOWED/I/FULL
```
If the Linux client is connected with Winbind, the use of the remote domain\\username is possible, see the `setcifsacl(1)` man page for further information.
## Kerberos Authentication
If a Windows Active Directory domain is available - or you can create one - Kerberos (**KRB5**) authentication with a randomized password can be used in place of the credentials file (`/etc/cifspw`) for additional security. First, we need to redefine a convention:
- **Domain**: _CIFSDOMAIN_ (NetBIOS name), _cifsdomain.local_ (AD name)
Note that after this section is complete the usage of `smbclient` is different - since we will use a random password in the Kerberos principal, Kerberos authentication with smbclient must also be used:
```
smbclient -k -U cifsuser@CIFSDOMAIN.LOCAL //cifsserver/cifsdata
```
**The IP address cannot be used**, it has to be the name of the CIFS server as shown and described below, as the Kerberos infrastructure will now be using names, not IPs. The `smbclient` command can be used while the share is mounted, the two methods work together without issue.
### Linux Packages
Two additional packages are required along with the ones installed previously; they may or may not already be installed. Some distros may ask for KRB5 configuration post-install, just accept the defaults - we'll overwrite them later.
```
# RHEL / CentOS
yum install keyutils krb5-workstation
# Debian / Ubuntu
apt-get update; apt-get install keyutils krb5-user
# Arch
pacman -Sy; pacman -S keyutils krb5
# openSUSE
zypper install keyutils krb5-client
```
### Domain Controller
> If the server is already part of a domain, skip this step and use the existing domain. There is no going back from this section, back up your server first as required.
If an AD domain is not present, the server will need to have the Active Directory Domain Services (**AD DS**) services installed and the server itself promoted to Domain Controller. This is required to create the Key Distribution Center (**KDC**) on the domain, responsible for supplying session tickets and temporary session keys.
Assuming Windows 2012 R2:
1. Open _Server Manager_, go to _Local Server_ on the left, then scroll all the way to the bottom _Roles and Features_
2. Click the drop-down _Tasks_ on the right side of the _Roles and Features_ block, choose _Add Roles and Features_
3. Under _Installation Type_ in the Wizard choose the _Role-based_ option then Next
4. Under _Server Selection_ choose our server, CIFSSERVER then Next
5. Under _Server Roles_ choose **Active Directory Domain Services**, accept the popup, then Next
6. Under _Features_ and _AD DS_ accept the defaults then Next and start the install
Almost done, leave that dialog there even though it says you can close it and wait. Now that the bits are installed, you need to **Promote this server to a Domain Controller** -- the option is listed in blue text on the finish screen from the above steps, click it.
1. On the initial screen choose the last option _Add a Forest_ to start fresh
2. For _Root domain name_ enter the domain (_cifsdomain.local_ herein), then Next
3. The next screen asks for a Functional level and DNS, accept the defaults
4. Enter a password of your choosing for _Directory Services Restore Mode_, then Next
5. You will most likely get an error about DNS Delegation, click Next
6. The NetBIOS name (_CIFSDOMAIN_ herein) should automatically fill in; if not, enter it then Next
7. Accept all other defaults for _Paths_, click Next to review and Next to being the Prerequisites check
8. More warnings about DNS Delegation show up, ignore them
9. Click Install and go get coffee, this takes awhile
**The server will reboot automatically** when finished. Give it some time, even after the reboot it's doing things that cause the login to take quite awhile.
### Test the Domain
Identical to the above, first test the Domain for basic functionality -- this will ensure that something has not gone wrong if you had to promote this server to be a Domain Controller. Simply swap the old **CIFSGROUP** for **CIFSDOMAIN** in the `smbclient` command:
```
smbclient -W CIFSDOMAIN -U CIFSDOMAIN\\cifsuser //192.168.5.5/cifsdata
Enter CIFSDOMAIN\cifsuser's password:
Domain=[CIFSDOMAIN] OS=[Windows Server 2012 R2 Standard 9600] Server=[Windows Server 2012 R2 Standard 6.3]
smb: \> exit
```
If this is no longer working, correct it before continuing.
### Generate the Keytab
On the Windows server, the KRB5 keytab file needs to be generated to map the user to the principal at the same time. Open an Administrator PowerShell prompt, change to the shared directory and create it like so:
```
PS C:\Users\Administrator> cd C:\cifsdata
PS C:\cifsdata> ktpass.exe /princ cifsuser@CIFSDOMAIN.LOCAL /ptype KRB5_NT_PRINCIPAL /out krb5.keytab +rndPass /crypto AES256-SHA1 /mapuser CIFSDOMAIN\cifsuser
```
If this works successfully, a message should look like:
```
Targeting domain controller: CIFSSERVER.cifsdomain.local
Using legacy password setting method
Failed to set property 'servicePrincipalName' to 'cifsuser' on Dn 'CN=cifsuser,CN=Users,DC=cifsdomain,DC=local': 0x13.
WARNING: Unable to set SPN mapping data.
If cifsuser already has an SPN mapping installed for cifsuser, this is no cause for concern.
Key created.
Output keytab to krb5.keytab:
Keytab version: 0x502
keysize 75 cifsuser@CIFSDOMAIN.LOCAL ptype 1 (KRB5_NT_PRINCIPAL) vno 2 etype 0x12 (AES256-SHA1) keylength 32 (0x3053927eb10407491db5a4cd05849cca3ac96f7ce1bad32269e174efa50439f9)
```
Over on the Linux side, use `smbclient` to connect and download the file; this keytab file can be used on multiple clients, so make sure a backup copy is stored in a secure, non-public location.
> **SECURITY ALERT**: Do not leave this file laying around on the public share\! This file should be protected and accessible to only the Windows _Administrator_ or Linux _root_ users.
```
# smbclient -W CIFSDOMAIN -U CIFSDOMAIN\\cifsuser //192.168.5.5/cifsdata
smb: \> get krb5.keytab
smb: \> rm krb5.keytab
```
Move it into `/etc/` and secure it from prying eyes, then test that you can read the principal:
```
mv krb5.keytab /etc/krb5.keytab
chown root:root /etc/krb5.keytab
chmod 0600 /etc/krb5.keytab
# klist -ke
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
2 cifsuser@CIFSDOMAIN.LOCAL (aes256-cts-hmac-sha1-96)
```
> On RHEL5/CentOS5 use `/usr/kerberos/bin/klist` as it's not in `$PATH` until you log out and back in again.
### Client KRB5 Setup
The client needs a few files configured to utilize a transparent ticket via _upcall_ methods to the server. Some of these may already be present, or possibly configured in different files as there are several ways to do it. Debian and Ubuntu for instance configure these in `/etc/request-key.conf` by default, whereas RHEL/CentOS and openSUSE place them in separate files in `/etc/request-key.d/`:
```
/etc/request-key.d/cifs.idmap.conf
create cifs.idmap * * /usr/sbin/cifs.idmap %k
/etc/request-key.d/cifs.spnego.conf
create cifs.spnego * * /usr/sbin/cifs.upcall %k
/etc/request-key.d/dns_resolver.conf
create dns_resolver * * /usr/sbin/cifs.upcall %k
```
Next, we need to tell the client Kerberos libraries how to contact the upstream KRB5 infrastructure via the normal configuration in `/etc/krb5.conf` - this is only a basic template, adjust if other settings are already present:
```
/etc/krb5.conf
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = CIFSDOMAIN.LOCAL
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
[realms]
CIFSDOMAIN.LOCAL = {
kdc = cifsserver.cifsdomain.local
admin_server = cifsserver.cifsdomain.local
default_domain = cifsdomain.local
}
[domain_realm]
.cifsdomain.local = CIFSDOMAIN.LOCAL
cifsdomain.local = CIFSDOMAIN.LOCAL
```
Finally, add an entry to `/etc/hosts` to map the names to IP of the server and domain:
```
/etc/hosts
192.168.5.5 cifsserver cifsserver.cifsdomain.local cifsdomain.local
```
> Be careful using DNS (`/etc/resolv.conf`) instead - it's possible the Windows server - which is now a DNS server - will return public IPs for a DNS query. We are ensuring the client always uses the private network IP by using a local /etc/hosts configuration.
### Connect and Test
Using the identical style from the non-KRB5 mount, the options are changed slightly to indicate the security **krb5i** in `/etc/fstab`:
```
# every distro except Debian/Ubuntu
//cifsserver/cifsdata /data cifs user=cifsuser,sec=krb5i,uid=nobody,gid=nobody,iocharset=utf8,file_mode=0644,dir_mode=0755,_netdev,soft 0 0
# Debian/Ubuntu only
//cifsserver/cifsdata /data cifs user=cifsuser,sec=krb5i,uid=nobody,gid=nogroup,iocharset=utf8,file_mode=0644,dir_mode=0755,_netdev,soft 0 0
```
Connect to the share, test it out:
```
mount /data
touch /data/test-krb5.txt
ls -l /data/test-krb5.txt
rm -f /data/test-krb5.txt
```
At this point everything should be working as expected as with a non-Kerberos mount. The `smbclient` command must be adjusted to use Kerberos as well, no password prompt should now occur:
```
smbclient -k -U cifsuser@CIFSDOMAIN.LOCAL //cifsserver/cifsdata
```
See the debugging section below if something is now working tosatisfaction.
## CIFS Debugging
Various parts of the setup - particularly Kerberos - may not work as expected and require a bit of debugging. Fortunately a nice interface is present in the kernel module to activate on the fly.
First, enable the debug mode of the cifs module **after** it's been loaded into the kernel (which creates this interface). The default is **0** (no debugging), set it to **9** for max verbosity:
```
echo 9 > /proc/fs/cifs/cifsFYI
```
Next add an output destination for the debug info; for example, if using **rsyslog** on RHEL / CentOS 6:
```
echo '*.debug /var/log/cifs-debug.log' >> /etc/rsyslog.conf
touch /var/log/cifs-debug.log
service rsyslog restart
```
Exact configuration will depend on the distribution and client setup already in place, as the logger in use may be **syslog**, **rsyslog**, **syslog-ng**, **systemd-journald**, etc. Don't forget to disable the debugging interface when complete:
```
echo 0 > /proc/fs/cifs/cifsFYI
```
...otherwise performance will suffer as everything is being logged to a very high degree.
## Packaging Bugs
Some distributions may have slightly broken packages of _cifs-utils_ for which the symlink of idmap is missing. I've reported these two bugs on Ubuntu 14 and Arch, Red Hat already had a bug report and it was fixed:
- <https://bugs.launchpad.net/ubuntu/+source/cifs-utils/+bug/1372120>
- <https://bugs.archlinux.org/task/42052>
- <https://bugzilla.redhat.com/show_bug.cgi?id=985067>
In general, the fix is very easy until these packages are updated or if you cannot use the latest packages:
```
# Ubuntu 14, cifs-utils 2:6.0-1ubuntu2
mkdir /etc/cifs-utils
ln -s /usr/lib/x86_64-linux-gnu/cifs-utils/idmapwb.so /etc/cifs-utils/idmap-plugin
# Arch, cifs-utils 6.3-2
mkdir /etc/cifs-utils
ln -s /usr/lib/cifs-utils/idmapwb.so /etc/cifs-utils/idmap-plugin
```
For versions of cifs-utils less than 6.2 an error will occur if it's missing:
```
ERROR: unable to initialize idmapping plugin: /etc/cifs-utils/idmap-plugin: cannot open shared object file: No such file or directory
```
For cifs-utils 6.2 and above, it's a warning instead:
```
WARNING: unable to initialize idmapping plugin. Only "raw" SID strings will be accepted: /etc/cifs-utils/idmap-plugin: cannot open shared object file: No such file or directory
```
All the other distros have this symlink present and work as expected.
## Troubleshooting
### RHEL5 Kerberos Fails
When configuring Kerberos authentication, an error may present like so when the `mount` command is issued:
```
mount error(126): Required key not available
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
```
Enabling debugging reveals lines such as these in the log:
```
kernel: CIFS VFS: Send error in SessSetup = -126
kernel: fs/cifs/connect.c: CIFS VFS: leaving cifs_mount (xid = 2) rc = -126
kernel: CIFS VFS: cifs_mount failed w/return code = -126
```
Please refer to upstream [bugzilla\#574750](https://bugzilla.redhat.com/show_bug.cgi?id=574750) for full information; in a nutshell there's an issue with how the KRB5 credentials are obtained and used with the older code (RHEL5 doesn't use upstream `cifs-utils`). Adjust the `/etc/fstab` mount line to use `uid=0` per that bug report - be warned though this means only the root user can actually write to the share when it's a system level mount.
### Kerberos Upcall Fails
When configuring Kerberos authentication, an error may present like so when the `mount` command is issued:
```
mount error(38): Function not implemented
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
```
Enabling debugging reveals lines such as these in the log:
```
kernel: fs/cifs/sess.c: sess setup type 5
kernel: CIFS VFS: Kerberos negotiated but upcall support disabled!
kernel: CIFS VFS: Send error in SessSetup = -38
kernel: CIFS VFS: cifs_mount failed w/return code = -38
```
This indicates that `CONFIG_CIFS_UPCALL` is disabled in the kernel; it can be checked like so:
```
gzip -dc /proc/config.gz | grep CONFIG_CIFS_UPCALL
```
This feature is enabled on all major upstream kernels, however some providers (such as custom cloud images) may have their own kernel. This feature must be enabled for Kerberos to work with CIFS.
## Additional Reading
- <http://en.wikipedia.org/wiki/Server_Message_Block>
- <https://wiki.samba.org/index.php/LinuxCIFS_utils>
- <http://en.wikipedia.org/wiki/Security_Identifier>
- <http://technet.microsoft.com/en-us/library/cc733924%28v=ws.10%29.aspx>
- <https://access.redhat.com/solutions/262553>

443
md/compose_key_sequences.md Normal file
View file

@ -0,0 +1,443 @@
# Compose Key Sequences
## Contents
- [Overview](#overview)
- [Basic Configuration](#basic-configuration)
- [Advanced Configuration](#advanced-configuration)
- [Key Sequences Chart](#key-sequences-chart)
- [References](#references)
## Overview
A **compose key** must be defined for the input method (keyboard); each desktop has it's quirks on how it can be used - for instance in MATE if the _Right-Alt_ is defined you can only use the _Left-Shift_ in combination with it. As an example, to type `¡` the key sequence is _Right-Alt_ + _Left+Shift_ held down while typing `!!` -- using the Right-Shift with this sequence doesn't work. Each desktop will have it's own quirks.
## Basic Configuration
GNOME, KDE and MATE (and maybe others) have a graphical way to set the compose key:
- **GNOME** and **MATE**
1. Menu
2. System
3. Preferences
4. Keyboard
5. Layout (tab)
6. Options (button)
7. Compose key position
- **KDE4**
1. System Settings
2. Regional & Language
3. Keyboard Layout
4. Enable keyboard layouts
5. Advanced (tab)
6. Compose key position
- **KDE3**
1. System Settings
2. Regional & Language
3. Keyboard Layout
4. Xkb Options (tab)
5. Compose key position
## Advanced Configuration
Other desktops like XFCE rely on using more basic X11 configuration to set the compose key. There are several ways it might be implemented on a given desktop. The list of compose keys used below can be obtained via `grep compose /usr/share/X11/xkb/rules/xorg.lst` and generally looks something like:
| Option | Compose Key |
| ------------- | ----------- |
| compose:ralt | Right Alt |
| compose:lwin | Left Win |
| compose:rwin | Right Win |
| compose:menu | Menu |
| compose:lctrl | Left Ctrl |
| compose:rctrl | Right Ctrl |
| compose:caps | Caps Lock |
| compose:paus | Pause |
| compose:prsc | PrtSc |
| compose:sclk | Scroll Lock |
If `/etc/default/keyboard` file exists and/or is in use, set the value:
```
XKBOPTIONS="compose:ralt"
```
This will take affect after a new login; to change it right away in a shell use:
```
$ setxkbmap -option compose:ralt
```
Other locations that can be used where the `setxkbmap` command could added:
- ~/.xinit
- ~/.xsession
- ~/.config/autostart/somefile.desktop
An autostart file for desktops like XFCE generally looks like:
```
[Desktop Entry]
Encoding=UTF-8
Version=0.9.4
Type=Application
Name=Set compose key
Comment=Set compose key
Exec=/usr/bin/setxkbmap -option compose:ralt
StartupNotify=false
Terminal=false
Hidden=false
```
## Key Sequences Chart
| Unicode | Char | Compose | Comment |
| ------- | ---- | ------------------------------- | ------------------------------------------ |
| U00a0 | | `" "` | NO-BREAK SPACE |
| U00a1 | `¡` | `"!!"` | INVERTED EXCLAMATION MARK |
| U00a2 | `¢` | `"\|c" "c\|" "c/" "/c"` | CENT SIGN |
| U00a3 | `£` | `"L-" "-L"` | POUND SIGN |
| U00a4 | `¤` | `"ox" "xo"` | CURRENCY SIGN |
| U00a5 | `¥` | `"Y=" "=Y"` | YEN SIGN |
| U00a6 | `¦` | `"!^"` | BROKEN BAR |
| U00a7 | `§` | `"so" "os"` | SECTION SIGN |
| U00a9 | `©` | `"oc" "oC" "Oc" "OC"` | COPYRIGHT SIGN |
| U00aa | `ª` | `"^_a"` | FEMININE ORDINAL INDICATOR |
| U00ab | `«` | `"<<"` | LEFT-POINTING DOUBLE ANGLE QUOTATION MARK |
| U00ac | `¬` | `",-" "-,"` | NOT SIGN |
| U00ae | `®` | `"or" "oR" "Or" "OR"` | REGISTERED SIGN |
| U00b0 | `°` | `"oo"` | DEGREE SIGN |
| U00b1 | `±` | `"+-"` | PLUS-MINUS SIGN |
| U00b2 | `²` | `"^2"` | SUPERSCRIPT TWO |
| U00b3 | `³` | `"^3"` | SUPERSCRIPT THREE |
| U00b5 | `µ` | `"mu"` | MICRO SIGN |
| U00b6 | `¶` | `"p!" "P!" "PP"` | PILCROW SIGN |
| U00b7 | `·` | `".."` | MIDDLE DOT |
| U00b8 | `¸` | `", " " ,"` | CEDILLA |
| U00b9 | `¹` | `"^1"` | SUPERSCRIPT ONE |
| U00ba | `º` | `"^_o"` | MASCULINE ORDINAL INDICATOR |
| U00bb | `»` | `">>"` | RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK |
| U00bc | `¼` | `"14"` | VULGAR FRACTION ONE QUARTER |
| U00bd | `½` | `"12"` | VULGAR FRACTION ONE HALF |
| U00be | `¾` | `"34"` | VULGAR FRACTION THREE QUARTERS |
| U00bf | `¿` | `"??"` | INVERTED QUESTION MARK |
| U00c0 | `À` | ``"`A"`` | LATIN CAPITAL LETTER A WITH GRAVE |
| U00c1 | `Á` | `"'A"` | LATIN CAPITAL LETTER A WITH ACUTE |
| U00c2 | `Â` | `"^A"` | LATIN CAPITAL LETTER A WITH CIRCUMFLEX |
| U00c3 | `Ã` | `"~A"` | LATIN CAPITAL LETTER A WITH TILDE |
| U00c4 | `Ä` | `""A"` | LATIN CAPITAL LETTER A WITH DIAERESIS |
| U00c5 | `Å` | `"oA"` | LATIN CAPITAL LETTER A WITH RING ABOVE |
| U00c6 | `Æ` | `"AE"` | LATIN CAPITAL LETTER AE |
| U00c7 | `Ç` | `",C"` | LATIN CAPITAL LETTER C WITH CEDILLA |
| U00c8 | `È` | ``"`E"`` | LATIN CAPITAL LETTER E WITH GRAVE |
| U00c9 | `É` | `"'E"` | LATIN CAPITAL LETTER E WITH ACUTE |
| U00ca | `Ê` | `"^E"` | LATIN CAPITAL LETTER E WITH CIRCUMFLEX |
| U00cb | `Ë` | `""E"` | LATIN CAPITAL LETTER E WITH DIAERESIS |
| U00cc | `Ì` | ``"`I"`` | LATIN CAPITAL LETTER I WITH GRAVE |
| U00cd | `Í` | `"'I"` | LATIN CAPITAL LETTER I WITH ACUTE |
| U00ce | `Î` | `"^I"` | LATIN CAPITAL LETTER I WITH CIRCUMFLEX |
| U00cf | `Ï` | `""I"` | LATIN CAPITAL LETTER I WITH DIAERESIS |
| U00d0 | `Ð` | `"DH"` | LATIN CAPITAL LETTER ETH |
| U00d1 | `Ñ` | `"~N"` | LATIN CAPITAL LETTER N WITH TILDE |
| U00d2 | `Ò` | ``"`O"`` | LATIN CAPITAL LETTER O WITH GRAVE |
| U00d3 | `Ó` | `"'O"` | LATIN CAPITAL LETTER O WITH ACUTE |
| U00d4 | `Ô` | `"^O"` | LATIN CAPITAL LETTER O WITH CIRCUMFLEX |
| U00d5 | `Õ` | `"~O"` | LATIN CAPITAL LETTER O WITH TILDE |
| U00d6 | `Ö` | `""O"` | LATIN CAPITAL LETTER O WITH DIAERESIS |
| U00d7 | `×` | `"xx"` | MULTIPLICATION SIGN |
| U00d8 | `Ø` | `"/O"` | LATIN CAPITAL LETTER O WITH STROKE |
| U00d9 | `Ù` | ``"`U"`` | LATIN CAPITAL LETTER U WITH GRAVE |
| U00da | `Ú` | `"'U"` | LATIN CAPITAL LETTER U WITH ACUTE |
| U00db | `Û` | `"^U"` | LATIN CAPITAL LETTER U WITH CIRCUMFLEX |
| U00dc | `Ü` | `""U"` | LATIN CAPITAL LETTER U WITH DIAERESIS |
| U00dd | `Ý` | `"'Y"` | LATIN CAPITAL LETTER Y WITH ACUTE |
| U00de | `Þ` | `"TH"` | LATIN CAPITAL LETTER THORN |
| U00df | `ß` | `"ss"` | LATIN SMALL LETTER SHARP S |
| U00e0 | `à` | ``"`a"`` | LATIN SMALL LETTER A WITH GRAVE |
| U00e1 | `á` | `"'a"` | LATIN SMALL LETTER A WITH ACUTE |
| U00e2 | `â` | `"^a"` | LATIN SMALL LETTER A WITH CIRCUMFLEX |
| U00e3 | `ã` | `"~a"` | LATIN SMALL LETTER A WITH TILDE |
| U00e4 | `ä` | `""a"` | LATIN SMALL LETTER A WITH DIAERESIS |
| U00e5 | `å` | `"oa"` | LATIN SMALL LETTER A WITH RING ABOVE |
| U00e6 | `æ` | `"ae"` | LATIN SMALL LETTER AE |
| U00e7 | `ç` | `",c"` | LATIN SMALL LETTER C WITH CEDILLA |
| U00e8 | `è` | ``"`e"`` | LATIN SMALL LETTER E WITH GRAVE |
| U00e9 | `é` | `"'e"` | LATIN SMALL LETTER E WITH ACUTE |
| U00ea | `ê` | `"^e"` | LATIN SMALL LETTER E WITH CIRCUMFLEX |
| U00eb | `ë` | `""e"` | LATIN SMALL LETTER E WITH DIAERESIS |
| U00ec | `ì` | ``"`i"`` | LATIN SMALL LETTER I WITH GRAVE |
| U00ed | `í` | `"'i"` | LATIN SMALL LETTER I WITH ACUTE |
| U00ee | `î` | `"^i"` | LATIN SMALL LETTER I WITH CIRCUMFLEX |
| U00ef | `ï` | `""i"` | LATIN SMALL LETTER I WITH DIAERESIS |
| U00f0 | `ð` | `"dh"` | LATIN SMALL LETTER ETH |
| U00f1 | `ñ` | `"~n"` | LATIN SMALL LETTER N WITH TILDE |
| U00f2 | `ò` | ``"`o"`` | LATIN SMALL LETTER O WITH GRAVE |
| U00f3 | `ó` | `"'o"` | LATIN SMALL LETTER O WITH ACUTE |
| U00f4 | `ô` | `"^o"` | LATIN SMALL LETTER O WITH CIRCUMFLEX |
| U00f5 | `õ` | `"~o"` | LATIN SMALL LETTER O WITH TILDE |
| U00f6 | `ö` | `""o"` | LATIN SMALL LETTER O WITH DIAERESIS |
| U00f7 | `÷` | `":-" "-:"` | DIVISION SIGN |
| U00f8 | `ø` | `"/o"` | LATIN SMALL LETTER O WITH STROKE |
| U00f9 | `ù` | ``"`u"`` | LATIN SMALL LETTER U WITH GRAVE |
| U00fa | `ú` | `"'u"` | LATIN SMALL LETTER U WITH ACUTE |
| U00fb | `û` | `"^u"` | LATIN SMALL LETTER U WITH CIRCUMFLEX |
| U00fc | `ü` | `""u"` | LATIN SMALL LETTER U WITH DIAERESIS |
| U00fd | `ý` | `"'y"` | LATIN SMALL LETTER Y WITH ACUTE |
| U00fe | `þ` | `"th"` | LATIN SMALL LETTER THORN |
| U00ff | `ÿ` | `""y"` | LATIN SMALL LETTER Y WITH DIAERESIS |
| U0100 | `Ā` | `"_A"` | LATIN CAPITAL LETTER A WITH MACRON |
| U0101 | `ā` | `"_a"` | LATIN SMALL LETTER A WITH MACRON |
| U0102 | `Ă` | `"UA" "bA"` | LATIN CAPITAL LETTER A WITH BREVE |
| U0103 | `ă` | `"Ua" "ba"` | LATIN SMALL LETTER A WITH BREVE |
| U0104 | `Ą` | `";A"` | LATIN CAPITAL LETTER A WITH OGONEK |
| U0105 | `ą` | `";a"` | LATIN SMALL LETTER A WITH OGONEK |
| U0106 | `Ć` | `"'C"` | LATIN CAPITAL LETTER C WITH ACUTE |
| U0107 | `ć` | `"'c"` | LATIN SMALL LETTER C WITH ACUTE |
| U0108 | `Ĉ` | `"^C"` | LATIN CAPITAL LETTER C WITH CIRCUMFLEX |
| U0109 | `ĉ` | `"^c"` | LATIN SMALL LETTER C WITH CIRCUMFLEX |
| U010c | `Č` | `"cC"` | LATIN CAPITAL LETTER C WITH CARON |
| U010d | `č` | `"cc"` | LATIN SMALL LETTER C WITH CARON |
| U010e | `Ď` | `"cD"` | LATIN CAPITAL LETTER D WITH CARON |
| U010f | `ď` | `"cd"` | LATIN SMALL LETTER D WITH CARON |
| U0110 | `Đ` | `"-D" "/D"` | LATIN CAPITAL LETTER D WITH STROKE |
| U0111 | `đ` | `"-d" "/d"` | LATIN SMALL LETTER D WITH STROKE |
| U0112 | `Ē` | `"_E"` | LATIN CAPITAL LETTER E WITH MACRON |
| U0113 | `ē` | `"_e"` | LATIN SMALL LETTER E WITH MACRON |
| U0114 | `Ĕ` | `"UE" "bE"` | LATIN CAPITAL LETTER E WITH BREVE |
| U0115 | `ĕ` | `"Ue" "be"` | LATIN SMALL LETTER E WITH BREVE |
| U0118 | `Ę` | `";E"` | LATIN CAPITAL LETTER E WITH OGONEK |
| U0119 | `ę` | `";e"` | LATIN SMALL LETTER E WITH OGONEK |
| U011a | `Ě` | `"cE"` | LATIN CAPITAL LETTER E WITH CARON |
| U011b | `ě` | `"ce"` | LATIN SMALL LETTER E WITH CARON |
| U011c | `Ĝ` | `"^G"` | LATIN CAPITAL LETTER G WITH CIRCUMFLEX |
| U011d | `ĝ` | `"^g"` | LATIN SMALL LETTER G WITH CIRCUMFLEX |
| U011e | `Ğ` | `"UG" "bG"` | LATIN CAPITAL LETTER G WITH BREVE |
| U011f | `ğ` | `"Ug" "bg"` | LATIN SMALL LETTER G WITH BREVE |
| U0122 | `Ģ` | `",G"` | LATIN CAPITAL LETTER G WITH CEDILLA |
| U0123 | `ģ` | `",g"` | LATIN SMALL LETTER G WITH CEDILLA |
| U0124 | `Ĥ` | `"^H"` | LATIN CAPITAL LETTER H WITH CIRCUMFLEX |
| U0125 | `ĥ` | `"^h"` | LATIN SMALL LETTER H WITH CIRCUMFLEX |
| U0126 | `Ħ` | `"/H"` | LATIN CAPITAL LETTER H WITH STROKE |
| U0127 | `ħ` | `"/h"` | LATIN SMALL LETTER H WITH STROKE |
| U0128 | `Ĩ` | `"~I"` | LATIN CAPITAL LETTER I WITH TILDE |
| U0129 | `ĩ` | `"~i"` | LATIN SMALL LETTER I WITH TILDE |
| U012a | `Ī` | `"_I"` | LATIN CAPITAL LETTER I WITH MACRON |
| U012b | `ī` | `"_i"` | LATIN SMALL LETTER I WITH MACRON |
| U012c | `Ĭ` | `"UI" "bI"` | LATIN CAPITAL LETTER I WITH BREVE |
| U012d | `ĭ` | `"Ui" "bi"` | LATIN SMALL LETTER I WITH BREVE |
| U012e | `Į` | `";I"` | LATIN CAPITAL LETTER I WITH OGONEK |
| U012f | `į` | `";i"` | LATIN SMALL LETTER I WITH OGONEK |
| U0131 | `ı` | `"i."` | LATIN SMALL LETTER DOTLESS I |
| U0134 | `Ĵ` | `"^J"` | LATIN CAPITAL LETTER J WITH CIRCUMFLEX |
| U0135 | `ĵ` | `"^j"` | LATIN SMALL LETTER J WITH CIRCUMFLEX |
| U0136 | `Ķ` | `",K"` | LATIN CAPITAL LETTER K WITH CEDILLA |
| U0137 | `ķ` | `",k"` | LATIN SMALL LETTER K WITH CEDILLA |
| U0138 | `ĸ` | `"kk"` | LATIN SMALL LETTER KRA |
| U0139 | `Ĺ` | `"'L"` | LATIN CAPITAL LETTER L WITH ACUTE |
| U013a | `ĺ` | `"'l"` | LATIN SMALL LETTER L WITH ACUTE |
| U013b | `Ļ` | `",L"` | LATIN CAPITAL LETTER L WITH CEDILLA |
| U013c | `ļ` | `",l"` | LATIN SMALL LETTER L WITH CEDILLA |
| U013d | `Ľ` | `"cL"` | LATIN CAPITAL LETTER L WITH CARON |
| U013e | `ľ` | `"cl"` | LATIN SMALL LETTER L WITH CARON |
| U0141 | `Ł` | `"/L"` | LATIN CAPITAL LETTER L WITH STROKE |
| U0142 | `ł` | `"/l"` | LATIN SMALL LETTER L WITH STROKE |
| U0143 | `Ń` | `"'N"` | LATIN CAPITAL LETTER N WITH ACUTE |
| U0144 | `ń` | `"'n"` | LATIN SMALL LETTER N WITH ACUTE |
| U0145 | `Ņ` | `",N"` | LATIN CAPITAL LETTER N WITH CEDILLA |
| U0146 | `ņ` | `",n"` | LATIN SMALL LETTER N WITH CEDILLA |
| U0147 | `Ň` | `"cN"` | LATIN CAPITAL LETTER N WITH CARON |
| U0148 | `ň` | `"cn"` | LATIN SMALL LETTER N WITH CARON |
| U014a | `Ŋ` | `"NG"` | LATIN CAPITAL LETTER ENG |
| U014b | `ŋ` | `"ng"` | LATIN SMALL LETTER ENG |
| U014c | `Ō` | `"_O"` | LATIN CAPITAL LETTER O WITH MACRON |
| U014d | `ō` | `"_o"` | LATIN SMALL LETTER O WITH MACRON |
| U014e | `Ŏ` | `"UO" "bO"` | LATIN CAPITAL LETTER O WITH BREVE |
| U014f | `ŏ` | `"Uo" "bo"` | LATIN SMALL LETTER O WITH BREVE |
| U0150 | `Ő` | `"=O"` | LATIN CAPITAL LETTER O WITH DOUBLE ACUTE |
| U0151 | `ő` | `"=o"` | LATIN SMALL LETTER O WITH DOUBLE ACUTE |
| U0152 | `Œ` | `"OE"` | LATIN CAPITAL LIGATURE OE |
| U0153 | `œ` | `"oe"` | LATIN SMALL LIGATURE OE |
| U0154 | `Ŕ` | `"'R"` | LATIN CAPITAL LETTER R WITH ACUTE |
| U0155 | `ŕ` | `"'r"` | LATIN SMALL LETTER R WITH ACUTE |
| U0156 | `Ŗ` | `",R"` | LATIN CAPITAL LETTER R WITH CEDILLA |
| U0157 | `ŗ` | `",r"` | LATIN SMALL LETTER R WITH CEDILLA |
| U0158 | `Ř` | `"cR"` | LATIN CAPITAL LETTER R WITH CARON |
| U0159 | `ř` | `"cr"` | LATIN SMALL LETTER R WITH CARON |
| U015a | `Ś` | `"'S"` | LATIN CAPITAL LETTER S WITH ACUTE |
| U015b | `ś` | `"'s"` | LATIN SMALL LETTER S WITH ACUTE |
| U015c | `Ŝ` | `"^S"` | LATIN CAPITAL LETTER S WITH CIRCUMFLEX |
| U015d | `ŝ` | `"^s"` | LATIN SMALL LETTER S WITH CIRCUMFLEX |
| U015e | `Ş` | `",S"` | LATIN CAPITAL LETTER S WITH CEDILLA |
| U015f | `ş` | `",s"` | LATIN SMALL LETTER S WITH CEDILLA |
| U0160 | `Š` | `"cS"` | LATIN CAPITAL LETTER S WITH CARON |
| U0161 | `š` | `"cs"` | LATIN SMALL LETTER S WITH CARON |
| U0162 | `Ţ` | `",T"` | LATIN CAPITAL LETTER T WITH CEDILLA |
| U0163 | `ţ` | `",t"` | LATIN SMALL LETTER T WITH CEDILLA |
| U0164 | `Ť` | `"cT"` | LATIN CAPITAL LETTER T WITH CARON |
| U0165 | `ť` | `"ct"` | LATIN SMALL LETTER T WITH CARON |
| U0166 | `Ŧ` | `"/T"` | LATIN CAPITAL LETTER T WITH STROKE |
| U0167 | `ŧ` | `"/t"` | LATIN SMALL LETTER T WITH STROKE |
| U0168 | `Ũ` | `"~U"` | LATIN CAPITAL LETTER U WITH TILDE |
| U0169 | `ũ` | `"~u"` | LATIN SMALL LETTER U WITH TILDE |
| U016a | `Ū` | `"_U"` | LATIN CAPITAL LETTER U WITH MACRON |
| U016b | `ū` | `"_u"` | LATIN SMALL LETTER U WITH MACRON |
| U016c | `Ŭ` | `"UU" "bU"` | LATIN CAPITAL LETTER U WITH BREVE |
| U016d | `ŭ` | `"Uu" "bu"` | LATIN SMALL LETTER U WITH BREVE |
| U016e | `Ů` | `"oU"` | LATIN CAPITAL LETTER U WITH RING ABOVE |
| U016f | `ů` | `"ou"` | LATIN SMALL LETTER U WITH RING ABOVE |
| U0170 | `Ű` | `"=U"` | LATIN CAPITAL LETTER U WITH DOUBLE ACUTE |
| U0171 | `ű` | `"=u"` | LATIN SMALL LETTER U WITH DOUBLE ACUTE |
| U0172 | `Ų` | `";U"` | LATIN CAPITAL LETTER U WITH OGONEK |
| U0173 | `ų` | `";u"` | LATIN SMALL LETTER U WITH OGONEK |
| U0174 | `Ŵ` | `"^W"` | LATIN CAPITAL LETTER W WITH CIRCUMFLEX |
| U0175 | `ŵ` | `"^w"` | LATIN SMALL LETTER W WITH CIRCUMFLEX |
| U0176 | `Ŷ` | `"^Y"` | LATIN CAPITAL LETTER Y WITH CIRCUMFLEX |
| U0177 | `ŷ` | `"^y"` | LATIN SMALL LETTER Y WITH CIRCUMFLEX |
| U0178 | `Ÿ` | `""Y"` | LATIN CAPITAL LETTER Y WITH DIAERESIS |
| U0179 | `Ź` | `"'Z"` | LATIN CAPITAL LETTER Z WITH ACUTE |
| U017a | `ź` | `"'z"` | LATIN SMALL LETTER Z WITH ACUTE |
| U017d | `Ž` | `"cZ"` | LATIN CAPITAL LETTER Z WITH CARON |
| U017e | `ž` | `"cz"` | LATIN SMALL LETTER Z WITH CARON |
| U017f | `ſ` | `"fs" "fS"` | LATIN SMALL LETTER LONG S |
| U0180 | `ƀ` | `"/b"` | LATIN SMALL LETTER B WITH STROKE |
| U0197 | `Ɨ` | `"/I"` | LATIN CAPITAL LETTER I WITH STROKE |
| U01b5 | `Ƶ` | `"/Z"` | LATIN CAPITAL LETTER Z WITH STROKE |
| U01b6 | `ƶ` | `"/z"` | LATIN SMALL LETTER Z WITH STROKE |
| U01cd | `Ǎ` | `"cA"` | LATIN CAPITAL LETTER A WITH CARON |
| U01ce | `ǎ` | `"ca"` | LATIN SMALL LETTER A WITH CARON |
| U01cf | `Ǐ` | `"cI"` | LATIN CAPITAL LETTER I WITH CARON |
| U01d0 | `ǐ` | `"ci"` | LATIN SMALL LETTER I WITH CARON |
| U01d1 | `Ǒ` | `"cO"` | LATIN CAPITAL LETTER O WITH CARON |
| U01d2 | `ǒ` | `"co"` | LATIN SMALL LETTER O WITH CARON |
| U01d3 | `Ǔ` | `"cU"` | LATIN CAPITAL LETTER U WITH CARON |
| U01d4 | `ǔ` | `"cu"` | LATIN SMALL LETTER U WITH CARON |
| U01e4 | `Ǥ` | `"/G"` | LATIN CAPITAL LETTER G WITH STROKE |
| U01e5 | `ǥ` | `"/g"` | LATIN SMALL LETTER G WITH STROKE |
| U01e6 | `Ǧ` | `"cG"` | LATIN CAPITAL LETTER G WITH CARON |
| U01e7 | `ǧ` | `"cg"` | LATIN SMALL LETTER G WITH CARON |
| U01e8 | `Ǩ` | `"cK"` | LATIN CAPITAL LETTER K WITH CARON |
| U01e9 | `ǩ` | `"ck"` | LATIN SMALL LETTER K WITH CARON |
| U01ea | `Ǫ` | `";O"` | LATIN CAPITAL LETTER O WITH OGONEK |
| U01eb | `ǫ` | `";o"` | LATIN SMALL LETTER O WITH OGONEK |
| U01f0 | `ǰ` | `"cj"` | LATIN SMALL LETTER J WITH CARON |
| U01f4 | `Ǵ` | `"'G"` | LATIN CAPITAL LETTER G WITH ACUTE |
| U01f5 | `ǵ` | `"'g"` | LATIN SMALL LETTER G WITH ACUTE |
| U01f8 | `Ǹ` | ``"`N"`` | LATIN CAPITAL LETTER N WITH GRAVE |
| U01f9 | `ǹ` | ``"`n"`` | LATIN SMALL LETTER N WITH GRAVE |
| U021e | `Ȟ` | `"cH"` | LATIN CAPITAL LETTER H WITH CARON |
| U021f | `ȟ` | `"ch"` | LATIN SMALL LETTER H WITH CARON |
| U0228 | `Ȩ` | `",E"` | LATIN CAPITAL LETTER E WITH CEDILLA |
| U0229 | `ȩ` | `",e"` | LATIN SMALL LETTER E WITH CEDILLA |
| U0232 | `Ȳ` | `"_Y"` | LATIN CAPITAL LETTER Y WITH MACRON |
| U0233 | `ȳ` | `"_y"` | LATIN SMALL LETTER Y WITH MACRON |
| U0259 | `ə` | `"ee"` | LATIN SMALL LETTER SCHWA |
| U0268 | `ɨ` | `"/i"` | LATIN SMALL LETTER I WITH STROKE |
| U1e10 | `Ḑ` | `",D"` | LATIN CAPITAL LETTER D WITH CEDILLA |
| U1e11 | `ḑ` | `",d"` | LATIN SMALL LETTER D WITH CEDILLA |
| U1e20 | `Ḡ` | `"_G"` | LATIN CAPITAL LETTER G WITH MACRON |
| U1e21 | `ḡ` | `"_g"` | LATIN SMALL LETTER G WITH MACRON |
| U1e26 | `Ḧ` | `""H"` | LATIN CAPITAL LETTER H WITH DIAERESIS |
| U1e27 | `ḧ` | `""h"` | LATIN SMALL LETTER H WITH DIAERESIS |
| U1e28 | `Ḩ` | `",H"` | LATIN CAPITAL LETTER H WITH CEDILLA |
| U1e29 | `ḩ` | `",h"` | LATIN SMALL LETTER H WITH CEDILLA |
| U1e30 | `Ḱ` | `"'K"` | LATIN CAPITAL LETTER K WITH ACUTE |
| U1e31 | `ḱ` | `"'k"` | LATIN SMALL LETTER K WITH ACUTE |
| U1e3e | `Ḿ` | `"'M"` | LATIN CAPITAL LETTER M WITH ACUTE |
| U1e3f | `ḿ` | `"'m"` | LATIN SMALL LETTER M WITH ACUTE |
| U1e54 | `Ṕ` | `"'P"` | LATIN CAPITAL LETTER P WITH ACUTE |
| U1e55 | `ṕ` | `"'p"` | LATIN SMALL LETTER P WITH ACUTE |
| U1e7c | `Ṽ` | `"~V"` | LATIN CAPITAL LETTER V WITH TILDE |
| U1e7d | `ṽ` | `"~v"` | LATIN SMALL LETTER V WITH TILDE |
| U1e80 | `Ẁ` | ``"`W"`` | LATIN CAPITAL LETTER W WITH GRAVE |
| U1e81 | `ẁ` | ``"`w"`` | LATIN SMALL LETTER W WITH GRAVE |
| U1e82 | `Ẃ` | `"'W"` | LATIN CAPITAL LETTER W WITH ACUTE |
| U1e83 | `ẃ` | `"'w"` | LATIN SMALL LETTER W WITH ACUTE |
| U1e84 | `Ẅ` | `""W"` | LATIN CAPITAL LETTER W WITH DIAERESIS |
| U1e85 | `ẅ` | `""w"` | LATIN SMALL LETTER W WITH DIAERESIS |
| U1e8c | `Ẍ` | `""X"` | LATIN CAPITAL LETTER X WITH DIAERESIS |
| U1e8d | `ẍ` | `""x"` | LATIN SMALL LETTER X WITH DIAERESIS |
| U1e90 | `Ẑ` | `"^Z"` | LATIN CAPITAL LETTER Z WITH CIRCUMFLEX |
| U1e91 | `ẑ` | `"^z"` | LATIN SMALL LETTER Z WITH CIRCUMFLEX |
| U1e97 | `ẗ` | `""t"` | LATIN SMALL LETTER T WITH DIAERESIS |
| U1e98 | `ẘ` | `"ow"` | LATIN SMALL LETTER W WITH RING ABOVE |
| U1e99 | `ẙ` | `"oy"` | LATIN SMALL LETTER Y WITH RING ABOVE |
| U1ebc | `Ẽ` | `"~E"` | LATIN CAPITAL LETTER E WITH TILDE |
| U1ebd | `ẽ` | `"~e"` | LATIN SMALL LETTER E WITH TILDE |
| U1ef2 | `Ỳ` | ``"`Y"`` | LATIN CAPITAL LETTER Y WITH GRAVE |
| U1ef3 | `ỳ` | ``"`y"`` | LATIN SMALL LETTER Y WITH GRAVE |
| U1ef8 | `Ỹ` | `"~Y"` | LATIN CAPITAL LETTER Y WITH TILDE |
| U1ef9 | `ỹ` | `"~y"` | LATIN SMALL LETTER Y WITH TILDE |
| U2008 | ` ` | `" ."` | PUNCTUATION SPACE |
| U2013 | `–` | `"--."` | EN DASH |
| U2014 | `—` | `"---"` | EM DASH |
| U2018 | `‘` | `"<'" "'<"` | LEFT SINGLE QUOTATION MARK |
| U2019 | `’` | `">'" "'>"` | RIGHT SINGLE QUOTATION MARK |
| U201a | `‚` | `",'" "',"` | SINGLE LOW-9 QUOTATION MARK |
| U201c | `“` | `"<"" ""<"` | LEFT DOUBLE QUOTATION MARK |
| U201d | `”` | `">"" "">"` | RIGHT DOUBLE QUOTATION MARK |
| U201e | `„` | `","" "","` | DOUBLE LOW-9 QUOTATION MARK |
| U2030 | `‰` | `"%o"` | PER MILLE SIGN |
| U2039 | `‹` | `".<"` | SINGLE LEFT-POINTING ANGLE QUOTATION MARK |
| U203a | `›` | `".>"` | SINGLE RIGHT-POINTING ANGLE QUOTATION MARK |
| U2070 | `⁰` | `"^0"` | SUPERSCRIPT ZERO |
| U2071 | `ⁱ` | `"^_i"` | SUPERSCRIPT LATIN SMALL LETTER I |
| U2074 | `⁴` | `"^4"` | SUPERSCRIPT FOUR |
| U2075 | `⁵` | `"^5"` | SUPERSCRIPT FIVE |
| U2076 | `⁶` | `"^6"` | SUPERSCRIPT SIX |
| U2077 | `⁷` | `"^7"` | SUPERSCRIPT SEVEN |
| U2078 | `⁸` | `"^8"` | SUPERSCRIPT EIGHT |
| U2079 | `⁹` | `"^9"` | SUPERSCRIPT NINE |
| U207a | `⁺` | `"^+"` | SUPERSCRIPT PLUS SIGN |
| U207c | `⁼` | `"^="` | SUPERSCRIPT EQUALS SIGN |
| U207d | `⁽` | `"^("` | SUPERSCRIPT LEFT PARENTHESIS |
| U207e | `⁾` | `"^)"` | SUPERSCRIPT RIGHT PARENTHESIS |
| U207f | `ⁿ` | `"^_n"` | SUPERSCRIPT LATIN SMALL LETTER N |
| U2080 | `₀` | `"_0"` | SUBSCRIPT ZERO |
| U2081 | `₁` | `"_1"` | SUBSCRIPT ONE |
| U2082 | `₂` | `"_2"` | SUBSCRIPT TWO |
| U2083 | `₃` | `"_3"` | SUBSCRIPT THREE |
| U2084 | `₄` | `"_4"` | SUBSCRIPT FOUR |
| U2085 | `₅` | `"_5"` | SUBSCRIPT FIVE |
| U2086 | `₆` | `"_6"` | SUBSCRIPT SIX |
| U2087 | `₇` | `"_7"` | SUBSCRIPT SEVEN |
| U2088 | `₈` | `"_8"` | SUBSCRIPT EIGHT |
| U2089 | `₉` | `"_9"` | SUBSCRIPT NINE |
| U208a | `₊` | `"_+"` | SUBSCRIPT PLUS SIGN |
| U208c | `₌` | `"_="` | SUBSCRIPT EQUALS SIGN |
| U208d | `₍` | `"_("` | SUBSCRIPT LEFT PARENTHESIS |
| U208e | `₎` | `"_)"` | SUBSCRIPT RIGHT PARENTHESIS |
| U20a0 | `₠` | `"CE"` | EURO-CURRENCY SIGN |
| U20a1 | `₡` | `"C/" "/C"` | COLON SIGN |
| U20a2 | `₢` | `"Cr"` | CRUZEIRO SIGN |
| U20a3 | `₣` | `"Fr"` | FRENCH FRANC SIGN |
| U20a4 | `₤` | `"L=" "=L"` | LIRA SIGN |
| U20a5 | `₥` | `"m/" "/m"` | MILL SIGN |
| U20a6 | `₦` | `"N=" "=N"` | NAIRA SIGN |
| U20a7 | `₧` | `"Pt"` | PESETA SIGN |
| U20a8 | `₨` | `"Rs"` | RUPEE SIGN |
| U20a9 | `₩` | `"W=" "=W"` | WON SIGN |
| U20ab | `₫` | `"d-"` | DONG SIGN |
| U20ac | `€` | `"C=" "=C" "c=" "=c" "E=" "=E"` | EURO SIGN |
| U2120 | `℠` | `"^SM"` | SERVICE MARK |
| U2122 | `™` | `"^TM"` | TRADE MARK SIGN |
| U301d | `〝` | `""\"` | REVERSED DOUBLE PRIME QUOTATION MARK |
| U301e | `〞` | `""/"` | DOUBLE PRIME QUOTATION MARK |
## References
- <http://www.hermit.org/Linux/ComposeKeys.html>
- <https://help.ubuntu.com/community/GtkComposeTable>
- <http://en.wikipedia.org/wiki/Compose_key>
- <http://fsymbols.com/keyboard/linux/compose/>
- <http://duncanlock.net/blog/2013/05/03/how-to-set-your-compose-key-on-xfce-xubuntu-linux/>
- <https://help.ubuntu.com/community/ComposeKey>

132
md/conversion_table.md Normal file
View file

@ -0,0 +1,132 @@
# Conversion Table
| **Dec** | **Hex** | **Oct** | `Bin` | | **Dec** | **Hex** | **Oct** | `Bin` |
| ------- | ------- | ------- | ---------- | --- | ------- | ------- | ------- | ---------- |
| **0** | 0 | 000 | `00000000` | | **128** | 80 | 200 | `10000000` |
| **1** | 1 | 001 | `00000001` | | **129** | 81 | 201 | `10000001` |
| **2** | 2 | 002 | `00000010` | | **130** | 82 | 202 | `10000010` |
| **3** | 3 | 003 | `00000011` | | **131** | 83 | 203 | `10000011` |
| **4** | 4 | 004 | `00000100` | | **132** | 84 | 204 | `10000100` |
| **5** | 5 | 005 | `00000101` | | **133** | 85 | 205 | `10000101` |
| **6** | 6 | 006 | `00000110` | | **134** | 86 | 206 | `10000110` |
| **7** | 7 | 007 | `00000111` | | **135** | 87 | 207 | `10000111` |
| **8** | 8 | 010 | `00001000` | | **136** | 88 | 210 | `10001000` |
| **9** | 9 | 011 | `00001001` | | **137** | 89 | 211 | `10001001` |
| **10** | a | 012 | `00001010` | | **138** | 8a | 212 | `10001010` |
| **11** | b | 013 | `00001011` | | **139** | 8b | 213 | `10001011` |
| **12** | c | 014 | `00001100` | | **140** | 8c | 214 | `10001100` |
| **13** | d | 015 | `00001101` | | **141** | 8d | 215 | `10001101` |
| **14** | e | 016 | `00001110` | | **142** | 8e | 216 | `10001110` |
| **15** | f | 017 | `00001111` | | **143** | 8f | 217 | `10001111` |
| **16** | 10 | 020 | `00010000` | | **144** | 90 | 220 | `10010000` |
| **17** | 11 | 021 | `00010001` | | **145** | 91 | 221 | `10010001` |
| **18** | 12 | 022 | `00010010` | | **146** | 92 | 222 | `10010010` |
| **19** | 13 | 023 | `00010011` | | **147** | 93 | 223 | `10010011` |
| **20** | 14 | 024 | `00010100` | | **148** | 94 | 224 | `10010100` |
| **21** | 15 | 025 | `00010101` | | **149** | 95 | 225 | `10010101` |
| **22** | 16 | 026 | `00010110` | | **150** | 96 | 226 | `10010110` |
| **23** | 17 | 027 | `00010111` | | **151** | 97 | 227 | `10010111` |
| **24** | 18 | 030 | `00011000` | | **152** | 98 | 230 | `10011000` |
| **25** | 19 | 031 | `00011001` | | **153** | 99 | 231 | `10011001` |
| **26** | 1a | 032 | `00011010` | | **154** | 9a | 232 | `10011010` |
| **27** | 1b | 033 | `00011011` | | **155** | 9b | 233 | `10011011` |
| **28** | 1c | 034 | `00011100` | | **156** | 9c | 234 | `10011100` |
| **29** | 1d | 035 | `00011101` | | **157** | 9d | 235 | `10011101` |
| **30** | 1e | 036 | `00011110` | | **158** | 9e | 236 | `10011110` |
| **31** | 1f | 037 | `00011111` | | **159** | 9f | 237 | `10011111` |
| **32** | 20 | 040 | `00100000` | | **160** | a0 | 240 | `10100000` |
| **33** | 21 | 041 | `00100001` | | **161** | a1 | 241 | `10100001` |
| **34** | 22 | 042 | `00100010` | | **162** | a2 | 242 | `10100010` |
| **35** | 23 | 043 | `00100011` | | **163** | a3 | 243 | `10100011` |
| **36** | 24 | 044 | `00100100` | | **164** | a4 | 244 | `10100100` |
| **37** | 25 | 045 | `00100101` | | **165** | a5 | 245 | `10100101` |
| **38** | 26 | 046 | `00100110` | | **166** | a6 | 246 | `10100110` |
| **39** | 27 | 047 | `00100111` | | **167** | a7 | 247 | `10100111` |
| **40** | 28 | 050 | `00101000` | | **168** | a8 | 250 | `10101000` |
| **41** | 29 | 051 | `00101001` | | **169** | a9 | 251 | `10101001` |
| **42** | 2a | 052 | `00101010` | | **170** | aa | 252 | `10101010` |
| **43** | 2b | 053 | `00101011` | | **171** | ab | 253 | `10101011` |
| **44** | 2c | 054 | `00101100` | | **172** | ac | 254 | `10101100` |
| **45** | 2d | 055 | `00101101` | | **173** | ad | 255 | `10101101` |
| **46** | 2e | 056 | `00101110` | | **174** | ae | 256 | `10101110` |
| **47** | 2f | 057 | `00101111` | | **175** | af | 257 | `10101111` |
| **48** | 30 | 060 | `00110000` | | **176** | b0 | 260 | `10110000` |
| **49** | 31 | 061 | `00110001` | | **177** | b1 | 261 | `10110001` |
| **50** | 32 | 062 | `00110010` | | **178** | b2 | 262 | `10110010` |
| **51** | 33 | 063 | `00110011` | | **179** | b3 | 263 | `10110011` |
| **52** | 34 | 064 | `00110100` | | **180** | b4 | 264 | `10110100` |
| **53** | 35 | 065 | `00110101` | | **181** | b5 | 265 | `10110101` |
| **54** | 36 | 066 | `00110110` | | **182** | b6 | 266 | `10110110` |
| **55** | 37 | 067 | `00110111` | | **183** | b7 | 267 | `10110111` |
| **56** | 38 | 070 | `00111000` | | **184** | b8 | 270 | `10111000` |
| **57** | 39 | 071 | `00111001` | | **185** | b9 | 271 | `10111001` |
| **58** | 3a | 072 | `00111010` | | **186** | ba | 272 | `10111010` |
| **59** | 3b | 073 | `00111011` | | **187** | bb | 273 | `10111011` |
| **60** | 3c | 074 | `00111100` | | **188** | bc | 274 | `10111100` |
| **61** | 3d | 075 | `00111101` | | **189** | bd | 275 | `10111101` |
| **62** | 3e | 076 | `00111110` | | **190** | be | 276 | `10111110` |
| **63** | 3f | 077 | `00111111` | | **191** | bf | 277 | `10111111` |
| **64** | 40 | 100 | `01000000` | | **192** | c0 | 300 | `11000000` |
| **65** | 41 | 101 | `01000001` | | **193** | c1 | 301 | `11000001` |
| **66** | 42 | 102 | `01000010` | | **194** | c2 | 302 | `11000010` |
| **67** | 43 | 103 | `01000011` | | **195** | c3 | 303 | `11000011` |
| **68** | 44 | 104 | `01000100` | | **196** | c4 | 304 | `11000100` |
| **69** | 45 | 105 | `01000101` | | **197** | c5 | 305 | `11000101` |
| **70** | 46 | 106 | `01000110` | | **198** | c6 | 306 | `11000110` |
| **71** | 47 | 107 | `01000111` | | **199** | c7 | 307 | `11000111` |
| **72** | 48 | 110 | `01001000` | | **200** | c8 | 310 | `11001000` |
| **73** | 49 | 111 | `01001001` | | **201** | c9 | 311 | `11001001` |
| **74** | 4a | 112 | `01001010` | | **202** | ca | 312 | `11001010` |
| **75** | 4b | 113 | `01001011` | | **203** | cb | 313 | `11001011` |
| **76** | 4c | 114 | `01001100` | | **204** | cc | 314 | `11001100` |
| **77** | 4d | 115 | `01001101` | | **205** | cd | 315 | `11001101` |
| **78** | 4e | 116 | `01001110` | | **206** | ce | 316 | `11001110` |
| **79** | 4f | 117 | `01001111` | | **207** | cf | 317 | `11001111` |
| **80** | 50 | 120 | `01010000` | | **208** | d0 | 320 | `11010000` |
| **81** | 51 | 121 | `01010001` | | **209** | d1 | 321 | `11010001` |
| **82** | 52 | 122 | `01010010` | | **210** | d2 | 322 | `11010010` |
| **83** | 53 | 123 | `01010011` | | **211** | d3 | 323 | `11010011` |
| **84** | 54 | 124 | `01010100` | | **212** | d4 | 324 | `11010100` |
| **85** | 55 | 125 | `01010101` | | **213** | d5 | 325 | `11010101` |
| **86** | 56 | 126 | `01010110` | | **214** | d6 | 326 | `11010110` |
| **87** | 57 | 127 | `01010111` | | **215** | d7 | 327 | `11010111` |
| **88** | 58 | 130 | `01011000` | | **216** | d8 | 330 | `11011000` |
| **89** | 59 | 131 | `01011001` | | **217** | d9 | 331 | `11011001` |
| **90** | 5a | 132 | `01011010` | | **218** | da | 332 | `11011010` |
| **91** | 5b | 133 | `01011011` | | **219** | db | 333 | `11011011` |
| **92** | 5c | 134 | `01011100` | | **220** | dc | 334 | `11011100` |
| **93** | 5d | 135 | `01011101` | | **221** | dd | 335 | `11011101` |
| **94** | 5e | 136 | `01011110` | | **222** | de | 336 | `11011110` |
| **95** | 5f | 137 | `01011111` | | **223** | df | 337 | `11011111` |
| **96** | 60 | 140 | `01100000` | | **224** | e0 | 340 | `11100000` |
| **97** | 61 | 141 | `01100001` | | **225** | e1 | 341 | `11100001` |
| **98** | 62 | 142 | `01100010` | | **226** | e2 | 342 | `11100010` |
| **99** | 63 | 143 | `01100011` | | **227** | e3 | 343 | `11100011` |
| **100** | 64 | 144 | `01100100` | | **228** | e4 | 344 | `11100100` |
| **101** | 65 | 145 | `01100101` | | **229** | e5 | 345 | `11100101` |
| **102** | 66 | 146 | `01100110` | | **230** | e6 | 346 | `11100110` |
| **103** | 67 | 147 | `01100111` | | **231** | e7 | 347 | `11100111` |
| **104** | 68 | 150 | `01101000` | | **232** | e8 | 350 | `11101000` |
| **105** | 69 | 151 | `01101001` | | **233** | e9 | 351 | `11101001` |
| **106** | 6a | 152 | `01101010` | | **234** | ea | 352 | `11101010` |
| **107** | 6b | 153 | `01101011` | | **235** | eb | 353 | `11101011` |
| **108** | 6c | 154 | `01101100` | | **236** | ec | 354 | `11101100` |
| **109** | 6d | 155 | `01101101` | | **237** | ed | 355 | `11101101` |
| **110** | 6e | 156 | `01101110` | | **238** | ee | 356 | `11101110` |
| **111** | 6f | 157 | `01101111` | | **239** | ef | 357 | `11101111` |
| **112** | 70 | 160 | `01110000` | | **240** | f0 | 360 | `11110000` |
| **113** | 71 | 161 | `01110001` | | **241** | f1 | 361 | `11110001` |
| **114** | 72 | 162 | `01110010` | | **242** | f2 | 362 | `11110010` |
| **115** | 73 | 163 | `01110011` | | **243** | f3 | 363 | `11110011` |
| **116** | 74 | 164 | `01110100` | | **244** | f4 | 364 | `11110100` |
| **117** | 75 | 165 | `01110101` | | **245** | f5 | 365 | `11110101` |
| **118** | 76 | 166 | `01110110` | | **246** | f6 | 366 | `11110110` |
| **119** | 77 | 167 | `01110111` | | **247** | f7 | 367 | `11110111` |
| **120** | 78 | 170 | `01111000` | | **248** | f8 | 370 | `11111000` |
| **121** | 79 | 171 | `01111001` | | **249** | f9 | 371 | `11111001` |
| **122** | 7a | 172 | `01111010` | | **250** | fa | 372 | `11111010` |
| **123** | 7b | 173 | `01111011` | | **251** | fb | 373 | `11111011` |
| **124** | 7c | 174 | `01111100` | | **252** | fc | 374 | `11111100` |
| **125** | 7d | 175 | `01111101` | | **253** | fd | 375 | `11111101` |
| **126** | 7e | 176 | `01111110` | | **254** | fe | 376 | `11111110` |
| **127** | 7f | 177 | `01111111` | | **255** | ff | 377 | `11111111` |

385
md/debian_boinc_client.md Normal file
View file

@ -0,0 +1,385 @@
# Debian BOINC Client
## Contents
- [Installation](#server-installation)
- [User Setup](#user-setup)
- [Disable root Login](#disable-root-login)
- [Server Hardening](#server-hardening)
- [fail2ban Setup](#fail2ban-setup)
- [BOINC Client Setup](#boinc-client-setup)
- [BOINC Project Setup](#boinc-project-setup)
- [Final Checks](#final-checks)
- [References](#references)
## Server Installation
### Physical Device
Install **Debian 9 (Stretch) 64bit or newer** onto the device using the Minimal setup option plus the SSH server option during the final steps on the installation. BOINC does not require a lot of disk space requiring custom partitioning, the default installation options should work in most cases.
### Cloud Server
Spin up a basic **Debian 9 (Stretch) 64bit or newer** cloud instance at the provider of your choice; note that a [BOINC](https://boinc.berkeley.edu/) client is CPU and RAM intensive, so be careful which provider you choose. This guide is written for a 1 CPU, 512M to 1G RAM, 20G disk cloud server with 1TB transfer/month which is billed at a flat monthly rate by the provider; tuning is provided to avoid noisy-neighbor behavior and keep resource usage within reasonable limits.
> **Be Careful of Costs** - Cloud providers typically charge for time spent running (uptime) _plus_ bandwidth charges. Research costs carefully and ensure the client is configured to meet your budget. The below tuning is only a guide and may need tweaked to specific circumstances.
The below instructions have been tested on various standard Debian instances supplied by cloud providers. The actual CPU resource limits vary depending on type of provider/instance, however - be sure to pay attention to the CPU usage tuning details.
## User Setup
> **Use a very secure password** - at a minimum use `pwgen -sB 15`, strong password security encouraged!
Set up a non-root user and add to the `sudo` group, then add a password. Use this user for SSH access and become root once logged in with sudo; if you have used SSH keys to log in as root, copy to this new user's setup as well if needed:
```
apt-get update
apt-get install sudo
export MYUSER="frankthetank"
useradd -m -d /home/${MYUSER} -s /bin/bash -g users -G sudo ${MYUSER}
passwd ${MYUSER}
```
If you are unable to use `ssh-copy-id` from your workstation to add a new SSH key, perform the work manually:
```
mkdir /home/${MYUSER}/.ssh
cp /root/.ssh/authorized_keys /home/${MYUSER}/.ssh/
chmod 0700 /home/${MYUSER}/.ssh
chmod 0600 /home/${MYUSER}/.ssh/authorized_keys
chown -R ${MYUSER}:users /home/${MYUSER}/.ssh
```
> **SSH in as this user and test `sudo` several times** - log out completely between tests
### Disable root Login
**If the above is successful** and you are capable of gaining full root privileges via the non-root SSH session using sudo, now disable root logins in SSH from the outside world for an additional security layer. The `root` account still remains usable, just not via _direct_ SSH access.
The task is to set `PermitRootLogin no` - the setting varies from one provider to another, sometimes it's already set (either yes or no), sometimes it's commented out. This small scriptlet should handle these 2 most common cases, **be careful** and investigate for yourself:
```
_SCFG="/etc/ssh/sshd_config"
if $(grep -iEq '^PermitRootLogin[[:space:]]+yes' "${_SCFG}"); then
sed -i.bak -e 's/^PermitRootLogin.*/PermitRootLogin no/gi' "${_SCFG}"
else
sed -i.bak -e 's/^#PermitRootLogin.*/PermitRootLogin no/gi' "${_SCFG}"
fi
```
**After confirming the change is correct**, restart the SSH core daemon (it will not log you out):
```
systemctl restart sshd
```
**Test logging in again** to ensure the changes are as expected. Do not log out of the active, working SSH session as root until you've confirmed in _another_ session you can log in as your non-root user and still gain `sudo` to root.
## Server Hardening
**1.** The Debian default vimrc (`set mouse=a`, `/usr/share/vim/vim80/defaults.vim`) messes up middle-mouse click paste when remote via SSH, override the setting to just disable the mouse:
```
echo 'set mouse=' >> /root/.vimrc
```
**2.** Install a few basic packages to make life a little nicer; typically the minimal install / cloud instances are stripped down and need a few things added, both for security and ease of use. Adjust as needed, at a minimum ensure the below are in place:
```
apt-get update
echo "iptables-persistent iptables-persistent/autosave_v4 boolean true" | debconf-set-selections
echo "iptables-persistent iptables-persistent/autosave_v6 boolean true" | debconf-set-selections
echo "unattended-upgrades unattended-upgrades/enable_auto_updates boolean true" | debconf-set-selections
apt-get install sysstat unattended-upgrades iptables-persistent man less vim
```
The `smem` package will pull in a lot of X dependencies due to an embedded recommendation, install it while disabling that feature. This utility can be used to quickly query memory usage (including swap) on the memory constrained cloud server:
```
apt-get install smem --no-install-recommends
```
**3.** Enable `journald` to store logs on disk instead of just RAM. By default, the `journald` system is in automatic mode - on boot it will create the ephemeral tmpfs `/run` out of RAM, but will _only_ transition to storing the journal on disk (out of RAM) if this directory exists.
```
mkdir /var/log/journal
```
**4.** Enable _sysstat_ for ongoing statistics capture of your instance (use `sar` to view):
```
sed -i.bak -e 's|^ENABLED=".*"|ENABLED="true"|g' /etc/default/sysstat
```
**5.** Enable _unattended-upgrades_ to ensure that all Security updates are applied:
```
cat << 'EOF' > /etc/apt/apt.conf.d/02periodic
APT::Periodic::Enable "1";
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "5";
APT::Periodic::Unattended-Upgrade "1";
EOF
```
**6.** Enable the basic _iptables_ rules to allow only port 22:
```
cat << 'EOF' > /etc/iptables/rules.v4
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
EOF
cat << 'EOF' > /etc/iptables/rules.v6
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p ipv6-icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp6-adm-prohibited
-A FORWARD -j REJECT --reject-with icmp6-adm-prohibited
COMMIT
EOF
```
**7.** If using a cloud server, add a bit of swap if needed - using swap is not bad in and of itself, the Linux kernel will attempt to cache it's small bits of data if available. Using swap in place of real RAM is bad, however - the tuning below will avoid actual application swapping.
```
# 512M file, probably overkill
dd if=/dev/zero of=/swap.file bs=4096 count=128000
chmod 0600 /swap.file
mkswap /swap.file
echo '/swap.file none swap defaults 0 0' >> /etc/fstab
swapon /swap.file
```
**8.** Finally, ensure all the services are enabled and apply all outstanding updates; reboot as needed for a new kernel. If you don't reboot here, you'll need to `service` _foo_ `restart` each one individually (just reboot, it's easier):
```
systemctl disable remote-fs.target
systemctl enable sysstat unattended-upgrades netfilter-persistent
apt-get dist-upgrade -y
reboot
```
### fail2ban Setup
Optional: configure fail2ban to keep an eye on the SSH port for brute force attacks.
> **Note**: `fail2ban` tends to consume a fair amount of memory the longer it runs; if the cloud server is memory constrained, you may wish to skip this step or disable the service later. Use `smem` to monitor it periodically.
```
apt-get install fail2ban sqlite3
cat << 'EOF' > /etc/fail2ban/jail.local
[DEFAULT]
ignoreip = 127.0.0.1/8
bantime = 600
maxretry = 3
backend = auto
destemail = root@localhost
[ssh]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 6
EOF
systemctl enable --now fail2ban
```
Additionally, add a weekly `cron` task to purge the database of old IPs (bug in 0.9.x series) and to restart the daemon to free up it's RAM usage:
```
cat << 'EOF' > /etc/fail2ban/dbpurge.sql
delete from bans where timeofban <= strftime('%s', date('now', '-7 days'));
vacuum;
.quit
EOF
cat << 'EOF' > /etc/cron.weekly/f2b-cleanup
#!/bin/sh
if [ -x /usr/bin/sqlite3 ]; then
sqlite3 /var/lib/fail2ban/fail2ban.sqlite3 < /etc/fail2ban/dbpurge.sql
fi
systemctl restart fail2ban.service
EOF
chown root:root /etc/cron.weekly/f2b-cleanup
chmod 0755 /etc/cron.weekly/f2b-cleanup
```
## BOINC Client Setup
> **Note**: Debian 9 uses `/var/lib/boinc-client` and Debian 10 uses `/var/lib/boinc` as the default data directory. If you installed using 9 and `dist-upgrade` to 10, the upgrade process will create a symlink connecting the old name to the new name.
The BOINC client is basically a wrapper application; it will "phone home" periodically and download new work units from the upstream project(s) and manage running those work units, then submit the results of the work to the upstream projects.
**1.** Install the `boinc-client` and `boinctui` software; the client will automatically start and enable as part of the package installation process. As of this writing errors appeared in the logfile about missing directories which are being added:
```
apt-get install boinc-client boinctui
# Debian 9
mkdir /var/lib/boinc-client/{slots,locale}
chown boinc:boinc /var/lib/boinc-client/{slots,locale}
# Debian 10
mkdir /var/lib/boinc-client/{slots,locale}
chown boinc:boinc /var/lib/boinc-client/{slots,locale}
systemctl restart boinc-client
```
**3.** Enable the client to always run, as this is a dedicated researching instance:
```
boinccmd --set_run_mode always
boinccmd --set_network_mode always
```
**4.** Cloud server: tune the client settings to fit within the resources, and keep it from being a noisy-neighbor; as this is a dedicated instance, most RAM and disk will be allocated, but the CPU thresholds are reduced. As an example of two very different platforms, if you're using a Google Cloud `f1-micro` VM which has only 0.2 (20%) of a vCPU, a good throttle is 19%. But if you're using a full vCPU instance, 49% is an acceptable limit for most providers to avoid being a noisy neighbor. You will need to research the exact right throttle for your specific circumstances.
> **1 TB/month** is roughly **400 KB/s** sustained bandwidth
**4.1** First, use `systemd` to control the CPU throttle, not the settings in the BOINC client. The BOINC client uses CPU idle detection and frequently pauses the process as it detects other things happening, which is undesirable in this specific setup. Instead, we will use `systemd` to create a _cgroup_ (resources control group) to confine the BOINC process - and it's children, the _work units_ - to a smaller percentage of the real CPU.
> The `CPUQuota` option was added in systemd version 213 and requires `CONFIG_CFS_BANDWIDTH=y` to be configured in the active kernel. Debian 9 meets both of the requirements.
Use the built-in edit capability of `systemctl` to create the unit override setting(s), it will create `/etc/systemd/system/boinc-client.service.d/override.conf` and place you into edit mode:
```
EDITOR=vi systemctl edit boinc-client
# Add the below to the new file being edited and save
[Service]
CPUQuota=49%
```
Inform `systemd` of the new content just created, then restart the BOINC client to activate:
```
systemctl daemon-reload
systemctl restart boinc-client
```
**4.2** Next, set up the BOINC client to use 100% of what it thinks is the whole CPU and to only pause when it thinks other processes are using 100%, such that it basically never stops running at all while confined to it's custom _cgroup_. Remember that on the outside, it's only actually using 49% of the CPU in this throttled configuration.
> Notice that we still set the real disk space and RAM thresholds, we are only telling it to use 100% of CPU as that's the only item we throttled from the outside. If you cannot use the above systemd throttle, you must set the CPU settings below to their actual lower values desired\!
```
# Debian 9
vi /var/lib/boinc-client/global_prefs_override.xml
# Debian 10
vi /var/lib/boinc/global_prefs_override.xml
<global_preferences>
<suspend_cpu_usage>100.000000</suspend_cpu_usage>
<cpu_usage_limit>100.000000</cpu_usage_limit>
<disk_max_used_pct>75.000000</disk_max_used_pct>
<ram_max_used_busy_pct>70.000000</ram_max_used_busy_pct>
<ram_max_used_idle_pct>90.000000</ram_max_used_idle_pct>
<vm_max_used_pct>50.000000</vm_max_used_pct>
<leave_apps_in_memory>1</leave_apps_in_memory>
<run_if_user_active>1</run_if_user_active>
<run_on_batteries>1</run_on_batteries>
<run_gpu_if_user_active>1</run_gpu_if_user_active>
<daily_xfer_limit_mb>8000.000000</daily_xfer_limit_mb>
<daily_xfer_period_days>30</daily_xfer_period_days>
</global_preferences>
boinccmd --read_global_prefs_override
```
> **Note**: the above step 4 instructions can also be followed on a Physical device to limit the CPU use, this is not unique to a cloud server. Throttling a physical device to 95% of the CPU power leaves a little room for other things (nightly updates, fail2ban, ssh sessions, etc.) to breathe a little easier.
## BOINC Project Setup
The easiest way to manage BOINC projects is to use an online Account Manager; much like the BOINC client manages downloading and running work units, the Account Manager handles which projects you wish to join, as well as helping connect all those projects together easily in one view. One of the most popular Account Managers is BOINC Account Manager (BAM!); it's already integrated to the `boinctui` and `boinc-client` software and has a handy setup guide:
- <https://boincstats.com/en/bam/>
Follow the above set of steps, then when reaching step 8 "Attach the BOINC client to BAM!" return here and follow these steps. There are multiple ways to do this, using `boinctui` simply makes everything a lot easier.
> **Tip**: When making your accounts, use the same username / password combination for the various projects that you use with BAM! setup. They all interconnect with each other, using the same credentials leads to a better experience and is easier to manage. However, be sure to use a unique password for this setup, do _not_ use an already known password!
**1.** As your _non-privileged user_ (not _root_!) start the application: `boinctui`. The very first time it starts a TUI dialog will appear asking where to connect for the BOINC client. Accept the defaults, `127.0.0.1` and no password.
> **Note**: a password can be added by root to `/etc/boinc-client/gui_rpc_auth.cfg`, then it's used for the above first-time connection dialog. This will add one more layer of security and is recommended; do _not_ use the BAM! password, just generate something random. I recommend `pwgen -sB 15` at a minimum.
**2.** Within the `boinctui` TUI now on screen, press `F9` to bring up the menu along the top. Arrow over to the **Projects** menu item, then choose the option to **start using an account manager** and press Enter.
**3.** From the list of account managersi now shown, select the one near the top named **BAM!** and a new dialog will appear asking for your login and password. This is the same login created on <https://boincstats.com/en/bam/> above.
At this point a few actions will be happening upstream, be patient - in general, the following is happening when you first join a new server to your BAM! account:
- A new host record is created in BAM!
- A new host record is created in the actual project(s)
- The new host records are tied together inside BAM!
- BAM! returns the list of projects to use back to your server
- Your server "phones home" to each project and initializes
- Your server runs first-time performance benchmarks
- Your server starts running it's first work units
After just a bit of time (it varies per project; as a rule of thumb, wait at least 5 minutes) perform two more first-time only actions to get everything aligned:
- Within `boinctui`, press `F9` for the Menu, scroll to **Projects**, select the project(s) and press Enter, a new submenu will appear. Select **Update project** -- this triggers your server to contact the project to sort of "cement it's existence" by uploading statistics that it's working hard for the money (so hard for the money).
- Within `boinctui`, press `F9` again, **Projects**, then **Synchronize with manager** -- this triggers BAM! to query the project you just updated above, so now BAM! knows your server is working hard and is connected.
The above two steps can be skipped and they will eventually happen on their own, they are normal regular tasks performed by the BOINC client wrapper software. Performing them manually the first time just helps make sure everything is working as expected. After a day or so (some projects only update statistics once a day) you will now be able to track the performance of your server in both BAM! and the specific projects themselves.
## Final Checks
Note that it may take a bit of time (not long) for tasks to be assigned to your new cloud client for processing; you should be able to get an immediate confirmation the client is connected and working with a general overview:
```
boinccmd --get_state
```
Once the Projects attach and start sending tasks, use `top` and you should see the active binaries crunching data using the CPU and RAM, and can be observed (see the man page for all options):
```
boinccmd --get_simple_gui_info
boinccmd --get_project_status
boinccmd --get_tasks
```
The `boinctui` application is a very nice alternative, it presents a nice windowed interface which can fully control the client options with menus, and presents all the information cleanly. Pretty much anything `boinccmd` can do, `boinctui` can do as well.
```
boinctui
```
See the above section on configuring `boinctui` it's first time run.
## References
- <https://boinc.berkeley.edu/wiki/Preferences>
- <https://boincstats.com/en/bam/>

373
md/debian_server_setup.md Normal file
View file

@ -0,0 +1,373 @@
# Debian Server Setup
## Contents
- [Server Installation](#server-installation)
- [Server User Setup](#server-user-setup)
- [Disable root Login](#disable-root-login)
- [Server Hardening](#server-hardening)
- [fail2ban Setup](#fail2ban-setup)
- [Apache Webserver](#apache-webserver)
- [Apache iptables Ports](#apache-iptables-ports)
- [Apache Default Template](#apache-default-template)
- [Apache 80 Template](#apache-80-template)
- [Apache 443 Template](#apache-443-template)
## Server Installation
Install **Debian 10** using the Minimal setup method, add the SSH server option during the final steps on the installation. This is the default image delivered from many cloud providers; it may use the default hostname `localhost` - if desired, set a new one:
```
hostnamectl set-hostname myhostname
```
Ensure the hostname resolves locally - it does not have to be `127.0.0.1` (localhost) nor a FQDN, for example this works:
```
127.0.0.1 localhost
127.0.1.1 myhostname
```
Adjust as needed based on how `/etc/hosts` is already configured from the installation.
## Server User Setup
> **Use a very secure password** - at a minimum use `pwgen -sB 15`, strong password security encouraged!
Set up a non-root user and add to the `sudo` group, then add a password. Use this user for SSH access and become root once logged in with sudo; if you have used SSH keys to log in as root, copy to this new user's setup as well if needed:
```
apt-get update
apt-get install sudo
export MYUSER="frankthetank"
useradd -m -d /home/${MYUSER} -s /bin/bash -g users -G sudo ${MYUSER}
passwd ${MYUSER}
```
If you are unable to use `ssh-copy-id` from your workstation to add a new SSH key, perform the work manually:
```
mkdir /home/${MYUSER}/.ssh
cp /root/.ssh/authorized_keys /home/${MYUSER}/.ssh/
chmod 0700 /home/${MYUSER}/.ssh
chmod 0600 /home/${MYUSER}/.ssh/authorized_keys
chown -R ${MYUSER}:users /home/${MYUSER}/.ssh
```
> **SSH in as this user and test `sudo` several times** - log out completely between tests
### Disable root Login
**If the above is successful** and you are capable of gaining full root privileges via the non-root SSH session using sudo, now disable root logins in SSH from the outside world for an additional security layer. The `root` account still remains usable, just not via _direct_ SSH access.
The task is to set `PermitRootLogin no` - the setting varies from one provider to another, sometimes it's already set (either yes or no), sometimes it's commented out. This small scriptlet should handle these 2 most common cases, **be careful** and investigate for yourself:
```
_SCFG="/etc/ssh/sshd_config"
if $(grep -iEq '^PermitRootLogin[[:space:]]+yes' "${_SCFG}"); then
sed -i.bak -e 's/^PermitRootLogin.*/PermitRootLogin no/gi' "${_SCFG}"
else
sed -i.bak -e 's/^#PermitRootLogin.*/PermitRootLogin no/gi' "${_SCFG}"
fi
```
**After confirming the change is correct**, restart the SSH core daemon (it will not log you out):
```
systemctl restart sshd
```
**Test logging in again** to ensure the changes are as expected. Do not log out of the active, working SSH session as root until you've confirmed in _another_ session you can log in as your non-root user and still gain `sudo` to root.
## Server Hardening
**1.** The Debian default vimrc (`set mouse=a`, `/usr/share/vim/vim80/defaults.vim`) messes up middle-mouse click paste when remote via SSH, override the setting to just disable the mouse:
```
echo 'set mouse=' >> ~/.vimrc
```
**2.** Install a few basic packages to make life a little nicer; typically the minimal install / cloud instances are stripped down and need a few things added, both for security and ease of use. Adjust as desired:
```
apt-get update
echo "iptables-persistent iptables-persistent/autosave_v4 boolean true" | debconf-set-selections
echo "iptables-persistent iptables-persistent/autosave_v6 boolean true" | debconf-set-selections
echo "unattended-upgrades unattended-upgrades/enable_auto_updates boolean true" | debconf-set-selections
apt-get install sysstat unattended-upgrades iptables-persistent man less vim rsync bc net-tools git strace
```
The `smem` package will pull in a lot of X dependencies due to an embedded recommendation, install it while disabling that feature. This utility can be used to quickly query memory usage (including swap) on the memory constrained cloud server:
```
apt-get install smem --no-install-recommends
```
**3.** Enable `journald` to store logs on disk instead of just RAM. By default, the `journald` system is in automatic mode - on boot it will create the ephemeral tmpfs `/run` out of RAM, but will _only_ transition to storing the journal on disk (out of RAM) if this directory exists.
```
mkdir /var/log/journal
```
**4.** Enable _sysstat_ for ongoing statistics capture of your instance (use `sar` to view):
```
sed -i.bak -e 's|^ENABLED=".*"|ENABLED="true"|g' /etc/default/sysstat
```
**5.** Enable _unattended-upgrades_ to ensure that all Security updates are applied:
```
cat << 'EOF' > /etc/apt/apt.conf.d/02periodic
APT::Periodic::Enable "1";
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "5";
APT::Periodic::Unattended-Upgrade "1";
EOF
```
**6.** Enable the basic _iptables_ rules to allow only port 22:
```
cat << 'EOF' > /etc/iptables/rules.v4
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
EOF
cat << 'EOF' > /etc/iptables/rules.v6
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p ipv6-icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp6-adm-prohibited
-A FORWARD -j REJECT --reject-with icmp6-adm-prohibited
COMMIT
EOF
```
**7.** Add a bit of swap if needed - using swap is not bad in and of itself, the Linux kernel will attempt to cache it's small bits of data if available. A cloud instance may not be delivered with any swap configured.
```
# 128M swap file
dd if=/dev/zero of=/swap.file bs=1024 count=128000
chmod 0600 /swap.file
mkswap /swap.file
echo '/swap.file none swap defaults 0 0' >> /etc/fstab
swapon /swap.file
```
**8.** Finally, ensure all the services are enabled and apply all outstanding updates; reboot as needed for a new kernel. If you don't reboot here, you'll need to `service` _foo_ `restart` each one individually (just reboot, it's easier):
```
systemctl disable remote-fs.target
systemctl enable sysstat unattended-upgrades netfilter-persistent
apt-get dist-upgrade -y
reboot
```
### fail2ban Setup
Recommended: configure fail2ban to keep an eye on the SSH port for brute force attacks.
> **Note**: `fail2ban` tends to consume a fair amount of memory the longer it runs; if the cloud server is memory constrained, you may wish to skip this step or disable the service later. Use `smem` to monitor it periodically.
```
apt-get install fail2ban sqlite3
cat << 'EOF' > /etc/fail2ban/jail.local
[DEFAULT]
ignoreip = 127.0.0.1/8
bantime = 600
maxretry = 3
backend = auto
destemail = root@localhost
EOF
systemctl enable --now fail2ban
```
Additionally, add a weekly `cron` task to purge the database of old IPs (bug in 0.9.x series) and to restart the daemon to free up it's RAM usage:
```
cat << 'EOF' > /etc/fail2ban/dbpurge.sql
delete from bans where timeofban <= strftime('%s', date('now', '-7 days'));
vacuum;
.quit
EOF
cat << 'EOF' > /etc/cron.weekly/f2b-cleanup
#!/bin/sh
if [ -x /usr/bin/sqlite3 ]; then
sqlite3 /var/lib/fail2ban/fail2ban.sqlite3 < /etc/fail2ban/dbpurge.sql
fi
systemctl restart fail2ban.service
EOF
chown root:root /etc/cron.weekly/f2b-cleanup
chmod 0755 /etc/cron.weekly/f2b-cleanup
```
## Apache Webserver
Optional: adding a webserver might be desired, the method of obtain the SSL certificate is not covered here.
### Apache Installation
The Debian package includes the SSL libraries, a few extra modules need to be enabled to support the extra security tuning in the templates.
```
apt-get update
apt-get install apache2
a2enmod ssl
a2enmod reqtimeout
a2enmod rewrite
a2enmod headers
a2enmod expires
```
### Apache iptables Ports
Ensure the ports for 80 and 443 are added to `/etc/iptables/rules.v4` and `/etc/iptables/rules.v6`, typically near where the SSH port has been opened:
```
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
```
Restart the daemon: `systemctl restart netfilter-persistent`
### Apache Default Template
This is the main template setting up parameters for all virtualhosts; the choice to include the virtual hosts in this template is not required, only a stylistic choice of the author. Save this to `/etc/apache2/sites-available/00_main.conf` (or use a symlink):
```
Timeout 60
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15
ServerName localhost
ServerTokens OS
TraceEnable off
<IfModule prefork.c>
StartServers 3
MinSpareServers 2
MaxSpareServers 4
ServerLimit 9
MaxClients 9
MaxRequestsPerChild 2000
</IfModule>
<IfModule mod_reqtimeout.c>
RequestReadTimeout header=20-40,MinRate=500 body=20,MinRate=500
</IfModule>
<Directory "/path/to/www">
AllowOverride None
Require all granted
</Directory>
# Port 80
Include /path/to/port_80.conf
# Port 443
Include /path/to/port_443.conf
```
Disable the Debian default website and enable the new one created above:
```
a2dissite 000-default
a2ensite 00_main
```
...or just manually change symlinks in `/etc/apache2/sites-enabled/` as desired.
### Apache 80 Template
Included above as `/path/to/port_80.conf`
```
<VirtualHost *:80>
ServerName example.com
ServerAlias www.example.com
ServerAdmin root@example.com
ErrorLog /var/log/apache2/example-error.log
CustomLog /var/log/apache2/example-access.log combined
DocumentRoot /path/to/www/html
<Directory /path/to/www/html>
Options FollowSymLinks
AllowOverride All
Require all granted
</Directory>
</VirtualHost>
```
### Apache 443 Template
Included above as `/path/to/port_443.conf`
```
<VirtualHost *:443>
ServerName example.com
ServerAlias www.example.com
ServerAdmin root@example.com
ErrorLog /var/log/apache2/example-error.log
CustomLog /var/log/apache2/example-access.log combined
SSLEngine on
SSLHonorCipherOrder on
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
SSLHonorCipherOrder on
SSLCompression off
SSLSessionTickets off
SSLCertificateFile /path/to/sslkeys/2020-example.crt
SSLCertificateKeyFile /path/to/sslkeys/2020-example.key
SSLCACertificateFile /path/to/sslkeys/2020-ssl-issuer-CA.pem
Header always set Strict-Transport-Security "max-age=15768000"
<Files ~ "\.(cgi|shtml|phtml|php3?)$">
SSLOptions +StdEnvVars
</Files>
SetEnvIf User-Agent ".*MSIE.*" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
DocumentRoot /path/to/www/html
<Directory /path/to/www/html>
Options FollowSymLinks
AllowOverride All
Require all granted
</Directory>
</VirtualHost>
```
Note the above 443 template does not enable HSTS on all subdomains by design, add as required.

195
md/debian_tor_relay.md Normal file
View file

@ -0,0 +1,195 @@
# Debian Tor Relay
## Contents
- [Server Installation](#server-installation)
- [Server Hardening](#server-hardening)
- [Tor Installation](#tor-installation)
- [Tor Setup](#tor-setup)
- [Tor Backup](#tor-backup)
- [Final Checks](#final-checks)
- [References](#references)
## Server Installation
Spin up a basic **Debian 8 (Jessie) 64bit** cloud instance; the use of inexpensive cloud instances from Digital Ocean are perfect for this type of project. Only basic networking with minimal disk and memory is required, these pre-prepared cloud installations of Debian 8 are ready to go with only a minor bit of work.
> **Be Careful of Costs** - Cloud providers typically charge for time spent running (uptime) _plus_ bandwidth charges. Research costs carefully and ensure the _RelayBandwidthRate_ is configured to meet your budget. Shop around cloud providers to get the best bang for your buck - low uptime and low bandwidth charges are the key factors for a tor node.
The below instructions have been tested on a Digital Ocean standard Debian 8 instance.
## Server Hardening
**1.** Install a few basic packages to make life a little nicer; typically the cloud instances are stripped down and need a few things added, both for security and ease of use. Adjust as needed, at a minimum ensure the below are in place:
```
apt-get update
apt-get install sysstat unattended-upgrades iptables-persistent fail2ban chrony vim-nox iftop sudo -y
```
**2.** Enable _sysstat_ for ongoing statistics capture of your instance (use `sar` to view):
```
sed -i.bak -e 's|^ENABLED=".*"|ENABLED="true"|g' /etc/default/sysstat
```
**3.** Enable _unattended-upgrades_ to ensure that all Security updates are applied:
```
cat << 'EOF' > /etc/apt/apt.conf.d/02periodic
APT::Periodic::Enable "1";
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "5";
APT::Periodic::Unattended-Upgrade "1";
EOF
```
**4.** Enable the basic _iptables_ rules to allow only ports 22, 80 and 443:
```
cat << 'EOF' > /etc/iptables/rules.v4
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
EOF
cat << 'EOF' > /etc/iptables/rules.v6
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p ipv6-icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp6-adm-prohibited
-A FORWARD -j REJECT --reject-with icmp6-adm-prohibited
COMMIT
EOF
```
**5.** Configure fail2ban to keep an eye on the SSH port for brute force attacks:
```
cat << 'EOF' > /etc/fail2ban/jail.local
[DEFAULT]
ignoreip = 127.0.0.1/8
bantime = 600
maxretry = 3
backend = auto
destemail = root@localhost
[ssh]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 6
EOF
```
**6.** Finally, ensure all the services are enabled and apply all outstanding updates; reboot as needed for a new kernel. If you don't reboot here, you'll need to `service` _foo_ `restart` each one individually:
```
systemctl disable remote-fs.target
systemctl enable sysstat unattended-upgrades iptables-persistent fail2ban chrony
apt-get upgrade -y
reboot
```
## Tor Installation
Add the upstream repository to the server, install the GPG key and tor itself. The `tor-arm` package provides an interesting console interface for the daemon. (run `arm` later on to see it)
```
echo "deb http://deb.torproject.org/torproject.org jessie main" > \
/etc/apt/sources.list.d/tor.list
gpg --keyserver keys.gnupg.net --recv 886DDD89
gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add -
apt-get update
apt-get install deb.torproject.org-keyring -y
apt-get install tor tor-arm -y
systemctl stop tor
```
## Tor Setup
Edit the `/etc/tor/torrc` configuration to set up the basic parameters; this config file's comments are parsed by the `arm` utility, so don't be tempted to just replace it with the below - hand edit is recommended to preserve the comments.
> **1 TB/month** is roughly **400 KB/s** sustained bandwidth
We will configure bandwidth to 300 KB/s normal and 350 KB/s burst to keep our cloud bandwidth charges in check, and ports 443 and 80 - maximum compatibility for persons in locations with strict ACLs on their network traffic. Choose **Nickname** wisely, it's how others will refer to your node in public. Be careful with ContactInfo and protect yourself from spammers\!
```
# egrep -v "^(#|$)" /etc/tor/torrc
RunAsDaemon 1
ORPort 443
Address <server IP address>
Nickname <your relay nickname>
RelayBandwidthRate 300 KB
RelayBandwidthBurst 350 KB
ContactInfo <your contact info>
DirPort 80
DirPortFrontPage /etc/tor/index.html
ExitPolicy reject *:*
```
Copy over the HTML man page to display on port 80 (see _DirPortFrontPage_ above), ensure it's set to start on reboot and get it running:
```
cp /usr/share/doc/tor/tor.html /etc/tor/index.html
systemctl enable tor
systemctl restart tor
```
## Tor Backup
Preserve a copy of your Tor node information; this is needed if you have to rebuild or move the node and want to retain the same history in the community:
```
cp /var/lib/tor/fingerprint /root/tor.fingerprint
cp /var/lib/tor/keys/secret_id_key /root/tor.secret_id_key
```
Download those two files from the cloud instance and put them in a safe place in your normal backups. The first has one line (nickname and 40-hex char ID), the second is a RSA key.
## Final Checks
Wait an hour or two, then use one (or both) of the below links to search for your relay's nickname:
- <https://atlas.torproject.org/>
- <https://globe.torproject.org/>
Once it's showing up as expected and you're happy with the results, submit your relay to the EFF Tor Challenge and sign up via Tor Weather to keep an eye on it:
- <https://www.eff.org/torchallenge/>
- <https://weather.torproject.org/subscribe/>
## References
- <https://www.torproject.org/docs/tor-relay-debian.html.en>

337
md/device_mapper_mechanics.md Normal file
View file

@ -0,0 +1,337 @@
# Device Mapper Mechanics
## Contents
- [Overview](#overview)
- [Basic Usage](#basic-usage)
- [Linear Target](#linear-target)
- [Striped Target](#striped-target)
- [Mirror Target](#mirror-target)
- [References](#references)
## Overview
Concisely, [device mapper](https://en.wikipedia.org/wiki/Device_mapper) is a method for the Linux kernel to map physical block devices into logical devices ("virtual") for further use; most notably implemented via [LVM2](lvm_mechanics.md), [Multipath](device_mapper_multipath.md) and [dm-crypt](https://en.wikipedia.org/wiki/Dm-crypt) (LUKS) in widespread use. Each implementation is known as a _target_ to the device-mapper subsystem within the kernel.
The fundamental tool used is named `dmsetup` and provided by the `device-mapper` package on most (if not all) Linux distributions. In a sense, it's analogous to common [partitioning](linux_partitioning.md) - using it requires a start, end and size component of a device. By default, when creating a logical device with `dmsetup` the human-readable name ends up symlinked in `/dev/mapper/` as evidenced with higher level tools like `lvcreate`.
> This article presents low-level examples for building targets; when a higher level subsystem such as _dm-raid_, _dm-crypt_, _LVM_ and _dm-multipath_ exists it should always be used. The higher level subsystems provide a wealth of additional features required for a production level situation.
A secondary tool named `dmstats` is also delivered which allows for collecting statistics about the underlying _regions_ and _areas_ of the device map (similar to what one might see from `sar` and the `sysstat` package). This can be useful for examining the performance characteristics of combined physical block devices and looking for deltas.
## Basic Usage
In this example we have a single 75G physical block device name `/dev/xvdb` (a Xen virtual disk). Using `dmsetup` it will be split into two virtual devices without using a partition table (aka "raw" device). Each virtual device is then formatted and mounted as normal with any block device. Key to all the work is understanding the [table mapping format](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Logical_Volume_Manager_Administration/device_mapper.html#dm-mappings) as listed in the [man page](http://linux.die.net/man/8/dmsetup), as it can be a bit confusing at first.
First, we need to determine what our starting and ending sectors on the disk will be for use; for this [a small one-liner shell script](https://gitlab.com/snippets/1720498) can do all the math for us. Note that we ensure a 2048s offset into the device for the standard performance alignment across all tiers - this is unique per situation, used here as a Best Practice.
```
DISK="/dev/xvdb"; OFFSET=2048 \
parted ${DISK} unit s print 2>/dev/null | \
grep "^Disk ${DISK}" | \
awk -v OFF=${OFFSET} '{gsub(/s$/,"",$3); \
printf "STA1=%s\nEND1=%s\nLEN1=%s\nSTA2=%s\nEND2=%s\nLEN2=%s\n",
OFF,(($3/2)-OFF),((($3/2)-OFF)-OFF),
((($3/2)-OFF)+1),$3,($3-((($3/2)-OFF)+1))
}'
STA1=2048
END1=78641152
LEN1=78639104
STA2=78641153
END2=157286400
LEN2=78645247
```
Given the start, end and size of each part of the disk we can use `dmsetup` to build the virtual maps; exactly like what may be familiar from previous LVM, LUKS and Multipath work the _real_ devices are `/dev/dm-?`, the kernel uses symlinks to inform the user of which map is which logical name for further use. We will also create a statistics area to see how that looks.
```
// Table format for linear target used below:
// <virtual start> <virtual size> linear <physical device> <physical start offset>
# dmsetup create xyzzy1 --table "0 78639104 linear /dev/xvdb 2048"
# dmsetup create xyzzy2 --table "0 78645247 linear /dev/xvdb 78641153"
# ls -og /dev/mapper/xyzzy*
lrwxrwxrwx. 1 7 Jan 10 18:32 /dev/mapper/xyzzy1 -> ../dm-0
lrwxrwxrwx. 1 7 Jan 10 18:32 /dev/mapper/xyzzy2 -> ../dm-1
# dmsetup table
xyzzy1: 0 78639104 linear 202:16 2048
xyzzy2: 0 78645247 linear 202:16 78641153
# dmstats create /dev/mapper/xyzzy1
xyzzy1: Created new region with 1 area(s) as region ID 0
# dmstats create /dev/mapper/xyzzy2
xyzzy2: Created new region with 1 area(s) as region ID 0
# dmstats list
Name RgID RgSta RgSize #Areas ArSize ProgID
xyzzy1 0 0 37.50g 1 37.50g dmstats
xyzzy2 0 0 37.50g 1 37.50g dmstats
# dmstats report
Name RgID ArID ArStart ArSize RMrg/s WMrg/s R/s W/s RSz/s WSz/s AvgRqSz QSize Util% AWait RdAWait WrAWait
xyzzy1 0 0 0 37.50g 0.00 0.00 0.00 0.00 0 0 0 0.00 0.00 0.00 0.00 0.00
xyzzy2 0 0 0 37.50g 0.00 0.00 0.00 0.00 0 0 0 0.00 0.00 0.00 0.00 0.00
```
Now it's just the same work as usual using these two new names; a simple `dd` is used below for testing.
```
# dd if=/dev/zero of=/dev/mapper/xyzzy1 bs=1024 count=100
100+0 records in
100+0 records out
102400 bytes (102 kB) copied, 0.0371653 s, 2.8 MB/s
# dd if=/dev/zero of=/dev/mapper/xyzzy2 bs=1024 count=100
100+0 records in
100+0 records out
102400 bytes (102 kB) copied, 0.0024724 s, 41.4 MB/s
# dmstats report
Name RgID ArID ArStart ArSize RMrg/s WMrg/s R/s W/s RSz/s WSz/s AvgRqSz QSize Util% AWait RdAWait WrAWait
xyzzy1 0 0 0 37.50g 0.00 0.00 88.00 25.00 556.00k 100.00k 5.50k 0.18 12.30 1.58 1.47 2.00
xyzzy2 0 0 0 37.50g 0.00 0.00 63.00 200.00 455.50k 100.00k 2.00k 0.51 10.60 1.94 1.75 2.00
# dmstats delete /dev/mapper/xyzzy2 --allregions
# dmstats delete /dev/mapper/xyzzy1 --allregions
# dmsetup remove /dev/mapper/xyzzy2
# dmsetup remove /dev/mapper/xyzzy1
```
## Linear Target
The [linear target](https://www.kernel.org/doc/Documentation/device-mapper/linear.txt) is the most basic as shown above; however in a slightly more complex example we can build our own LVM-like single filesystem that spans two physical block devices. The LVM subsystem at it's core uses this linear methodology by default, however is contains many additional features (mapping UUIDs, maintaining block device lists and the tables, checksumming, management, etc.) which make it desirable in daily use.
Two 75G block devices are presented to the host; to add a bit more complication to exemplify the math, each block device has an empty GPT partition table to simulate not being able to use the end of the disk (the [backup GPT partition table](linux_x86_storage.md) is kept in the last 34s).
```
# parted /dev/xvdb mktable gpt
# parted /dev/xvdc mktable gpt
```
Next, we need to get the last _usable_ sector of the disk and subtract our performance-oriented beginning 2048s offset from it to get the size of the fully usable disk area; for this I prefer `sgdisk` utility (part of the `gdisk` package):
```
# sgdisk -p /dev/xvdb | grep "last usable sector" | awk '{print $NF-2048}'
157284318
# sgdisk -p /dev/xvdc | grep "last usable sector" | awk '{print $NF-2048}'
157284318
```
Because this requires two lines to feed `dmsetup` (one line for each disk), we create the mapping in a text file:
```
// Table format used below is the same as the Basic example, but notice that the
// virtual start of the second disk is the same as the ending of the first -
// remember, 0 offset not 1
# cat linear.table
0 157284318 linear /dev/xvdb 2048
157284318 157284318 linear /dev/xvdc 2048
```
Now it's just a matter of creating the map using the table and testing it out by making a filesystem and writing a file larger than any one single physical device (below, 120G is used):
```
# dmsetup create xyzzy linear.table
# mkfs.ext4 -v /dev/mapper/xyzzy
# mkdir /mnt/xyzzy
# mount /dev/mapper/xyzzy /mnt/xyzzy/
# dd if=/dev/zero of=/mnt/xyzzy/testfile bs=512M count=240
240+0 records in
240+0 records out
128849018880 bytes (129 GB) copied, 367.6 s, 351 MB/s
# umount /mnt/xyzzy
# dmsetup remove xyzzy
# dmsetup create foobar linear.table
# mount /dev/mapper/foobar /mnt/xyzzy/
# ls -og /mnt/xyzzy/
total 125829148
drwx------. 2 16384 Jan 10 21:16 lost+found
-rw-r--r--. 1 128849018880 Jan 10 21:25 testfile
```
Notice that the maps are disassembled and recreated as part of the testing to simulate what will happen when the server is rebooted - **device maps are in memory only** so in real use startup/shutdown scripts would be required to implement the above correctly. We also tested giving the device map a randomly different name the second as a test.
## Striped Target
The [striped target](https://www.kernel.org/doc/Documentation/device-mapper/striped.txt) is the basis of software RAID0 and can be used with LVM. Using the same techniques as the linear target, we'll build a simple striped target of our two physical block devices with the intent of increasing performance (so we'll add `dmstats`).
First, we have to do a bit of math; when using striping technology design each group of data is written in a _chunk_ that is a multiple of 2 and typically of a size that is optimized for the data. A chunk of 256k is very common for physical RAID controllers, we'll use this as our chunk size. However, our sector size is 512b so we'll need to use 512 as our divisor; given that, we must ensure that **an entire stripe can be written**.
In order to determine the largest size we can make the striped target, we take the usable size of the disk (in sectors), divide it by 512 and then get the floor() of that value re-multiplied times 512 (in laypersons' terms, divide the size by 512, throw away the remainder and re-multiply by 512 to get the perfect multiple). For this we'll use a `bc` function:
```
# bc
// add both usable sizes to get one large size for striping
157284318*2
314568636
// now divide by 512, throw away the remainder, re-multiply by 512
define floor(x) {
auto os,xx;os=scale;scale=0
xx=x/1;if(xx>x).=xx--
scale=os;return(xx)
}
floor(314568636/512)*512
314568192
```
Armed with this perfect multiple of 512 (ergo 256k), build a striped map. Create the device as before and this time we'll create 2 `dmstats` areas (one for each physical disk's sectors used) so that we can contrast./compare performance of each one. Notice that because we have two identically sized devices the `dmstats --areas 2` usage perfectly splits it for us so we don't have to define each area by hand:
```
# cat striped.table
0 314568192 striped 2 256 /dev/xvdb 2048 /dev/xvdc 2048
# dmsetup create xyzzy striped.table
# dmstats create xyzzy --areas 2
xyzzy: Created new region with 2 area(s) as region ID 0
# dmstats list
Name RgID RgSta RgSize #Areas ArSize ProgID
xyzzy 0 0 150.00g 2 75.00g dmstats
# dmstats report
Name RgID ArID ArStart ArSize RMrg/s WMrg/s R/s W/s RSz/s WSz/s AvgRqSz QSize Util% AWait RdAWait WrAWait
xyzzy 0 0 0 75.00g 0.00 0.00 0.00 0.00 0 0 0 0.00 0.00 0.00 0.00 0.00
xyzzy 0 1 75.00g 75.00g 0.00 0.00 0.00 0.00 0 0 0 0.00 0.00 0.00 0.00 0.00
```
Now that we have the statistics gathering readied, create the filesystem and write data as per our normal testing plan:
```
# mkfs.ext4 -v /dev/mapper/xyzzy
# mount /dev/mapper/xyzzy /mnt/xyzzy/
# dd if=/dev/zero of=/mnt/xyzzy/testfile bs=512M count=240
240+0 records in
240+0 records out
128849018880 bytes (129 GB) copied, 224.258 s, 575 MB/s
# dmstats report
Name RgID ArID ArStart ArSize RMrg/s WMrg/s R/s W/s RSz/s WSz/s AvgRqSz QSize Util% AWait RdAWait WrAWait
xyzzy 0 0 0 75.00g 0.00 0.00 185.00 1786193.00 1.13m 71.35g 41.50k 23550.26 100.00 13.18 30.79 13.18
xyzzy 0 1 75.00g 75.00g 0.00 0.00 61.00 1257156.00 244.00k 51.14g 42.50k 17752.53 100.00 14.12 35.92 14.12
# umount /mnt/xyzzy
# dmsetup remove xyzzy
# dmsetup create xyzzy striped.table
# mount /dev/mapper/xyzzy /mnt/xyzzy
# ls -og /mnt/xyzzy
total 125829148
drwx------. 2 16384 Jan 10 22:28 lost+found
-rw-r--r--. 1 128849018880 Jan 10 22:33 testfile
```
Based on our data above, it appears that we're getting better performance from `/dev/xvdb` than we am from `/dev/xvdc`; this is a public cloud instance so it's expected we have two data block volumes from different back-end cloud hosts via iSCSI. This exemplifies the risk in LVM of combining such objects in the cloud, performance characteristics will vary from data block device to data block device in this environment. What we did see, though, was the raw write performance go up about 2.5x over linear as expected.
## Mirror Target
The mirror target is arguably the most difficult to construct; in essence it's a RAID1 (but again can also be used in LVM) but it requires a log device (similar to a classic journal in a filesystem if you will, a space to record metadata about writes). In order to construct this example, we're going to construct disk partitions and use a technique generally known as "mirroring the mirror".
First we'll prep out 2 partition on each physical block device, one to store log data to disk - so that when rebooting/recreating the mirror has it's data on each _leg_, otherwise a bootstrap would be required again - and one to store data. They will be exactly the same on both disks as this is a mirror configuration.
```
# sgdisk -Z /dev/xvdb
# parted /dev/xvdb mktable gpt
# parted /dev/xvdb mkpart primary ext3 2048s 18432s
# parted /dev/xvdb mkpart primary ext3 20480s 100%
// Note that we choose some arbitrarily sized numbers; 8k for logs is plenty
# sgdisk -Z /dev/xvdc
# parted /dev/xvdc mktable gpt
# parted /dev/xvdc mkpart primary ext3 2048s 18432s
# parted /dev/xvdc mkpart primary ext3 20480s 100%
# parted /dev/xvdb unit s print
[...]
Number Start End Size File system Name Flags
1 2048s 18432s 16385s primary
2 20480s 157284351s 157263872s primary
```
Now we need to build the virtual maps for these devices; in a sense it's like connecting two linear targets together but with some other items as outlined on the `dmsetup` man page, noting that _core_ is stored in memory so the regions will be pushed to disk. Notice that since partitions are already defined, the offset is 0 (beginning of partition) for each virtual map.
```
// Table format used:
// <virtual start> <virtual size> core <params = 1> <param1 = size> <devs = 2> <dev1> <offset1> <dev2> <offset 2> <features = 1> <feature = handle_errors>
# cat mirror-log.table
0 8192 mirror core 1 1024 2 /dev/xvdb1 0 /dev/xvdc1 0 1 handle_errors
// Table format used:
// <virtual start> <virtual size> disk <params = 2> >param1 = log dev> <param2 = size> <devs = 2> <dev1> <offset1> <dev2> <offset 2> <features = 1> <feature = handle_errors>
# cat mirror-data.table
0 157263872 mirror disk 2 /dev/mapper/xyzzy-log 1024 2 /dev/xvdb2 0 /dev/xvdc2 0 1 handle_errors
```
Now we create the device maps using these tables, ensure the _log_ is created first:
```
# dmsetup create xyzzy-log mirror-log.table
# dmsetup create xyzzy-data mirror-data.table
# mkfs.ext4 -v /dev/mapper/xyzzy-data
# mount /dev/mapper/xyzzy-data /mnt/xyzzy/
# dmstats create xyzzy-log
# dmstats create xyzzy-data
# dd if=/dev/zero of=/mnt/xyzzy/testfile bs=512M count=120
120+0 records in
120+0 records out
64424509440 bytes (64 GB) copied, 235.429 s, 274 MB/s
# dmstats report
Name RgID ArID ArStart ArSize RMrg/s WMrg/s R/s W/s RSz/s WSz/s AvgRqSz QSize Util% AWait RdAWait WrAWait
xyzzy-log 0 0 0 4.00m 0.00 0.00 0.00 35357.00 0 690.57m 20.00k 42.16 100.00 1.19 0.00 1.19
xyzzy-data 0 0 0 74.99g 0.00 0.00 47.00 1486089.00 188.00k 60.23g 42.00k 163854.02 100.00 110.26 66.28 110.26
# umount /mnt/xyzzy
# dmsetup remove xyzzy-data
# dmsetup remove xyzzy-log
# dmsetup create xyzzy-log mirror-log.table
# dmsetup create xyzzy-data mirror-data.table
# mount /dev/mapper/xyzzy-data /mnt/xyzzy/
# ls -og /mnt/xyzzy/
total 62914584
drwx------. 2 16384 Jan 10 23:39 lost+found
-rw-r--r--. 1 64424509440 Jan 10 23:45 testfile
# umount /mnt/xyzzy/
# dmsetup remove xyzzy-data
# dmsetup remove xyzzy-log
# mount /dev/xvdb2 /mnt/xyzzy/
# ls -og /mnt/xyzzy/
total 62914584
drwx------. 2 16384 Jan 10 23:39 lost+found
-rw-r--r--. 1 64424509440 Jan 10 23:45 testfile
# umount /mnt/xyzzy/
# mount /dev/xvdc2 /mnt/xyzzy/
# ls -og /mnt/xyzzy/
total 62914584
drwx------. 2 16384 Jan 10 23:39 lost+found
-rw-r--r--. 1 64424509440 Jan 10 23:45 testfile
```
Notice that we also test destroying the device maps and manually mounting and verifying the raw block device as might be needed in a real situation where one member of the mirror goes offline.
## References
- <https://www.kernel.org/doc/Documentation/device-mapper/>
- <https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Logical_Volume_Manager_Administration/device_mapper.html>
- <https://wiki.gentoo.org/wiki/Device-mapper>

363
md/device_mapper_multipath.md Normal file
View file

@ -0,0 +1,363 @@
# Device Mapper Multipath
## Contents
- [Overview](#overview)
- [Initial Setup](#initial-setup)
- [DAS Config](#das-config)
- [NAS iSCSI Config](#nas-iscsi-config)
- [Multipath Names](#multipath-names)
- [Administrating Multipaths](#administrating-multipaths)
- [Partitioning Multipaths](#partitioning-multipaths)
- [Clustered Multipaths](#clustered-multipaths)
- [Renaming Multipaths](#renaming-multipaths)
- [Multipath Ownership](#multipath-ownership)
- [RHEL5 / CentOS5](#rhel5--centos5)
- [RHEL6 / CentOS6](#rhel6--centos6)
- [References](#references)
## Overview
The `device-mapper-multipath` (a sub-component of `device-mapper`) subsystem is the native way of configuring 2 or more individual paths to the same storage LUN, typically used in a HA (failover) capacity. If one underlying path fails the system transfers I/O to another path; higher level operations (such as LVM) use the single multipath pseudo device and are abstracted from the underlying physical links.
## Initial Setup
A standard setup requires 2 RPMs which provide the `multipathd` service and udev rules for naming the multipaths:
1. device-mapper
2. device-mapper-multipath
For a Dell DAS such as the MD32xx 2 more packages are required, typically from the vendor install media:
1. dkms (Dynamic Kernel Module Support - framework required for the below RPM)
2. scsi\_dh\_rdac (Dell custom version, the [kernel also contains one](https://github.com/torvalds/linux/blob/master/drivers/scsi/device_handler/scsi_dh_rdac.c))
The `multipathd` service is what pulls it all together.
### DAS Config
A well formed Dell MD32xx DAS deployed config might look like:
```
# DAS /etc/multipath.conf
blacklist {
device {
vendor "*"
product "Universal Xport"
}
device {
vendor "*"
product "MD3000"
}
device {
vendor "*"
product "MD3000i"
}
device {
vendor "*"
product "Virtual Disk"
}
device {
vendor "*"
product "PERC|Perc"
}
}
defaults {
user_friendly_names yes
max_fds 8192
polling_interval 5
}
devices {
device {
vendor "DELL"
product "MD32xxi"
path_grouping_policy group_by_prio
prio rdac
path_checker rdac
path_selector "round-robin 0"
hardware_handler "1 rdac"
failback immediate
features "2 pg_init_retries 50"
no_path_retry 30
rr_min_io 100
}
device {
vendor "DELL"
product "MD32xx"
path_grouping_policy group_by_prio
prio rdac
path_checker rdac
path_selector "round-robin 0"
hardware_handler "1 rdac"
failback immediate
features "2 pg_init_retries 50"
no_path_retry 30
rr_min_io 100
}
device {
vendor "DELL"
product "MD36xxi"
path_grouping_policy group_by_prio
prio rdac
path_checker rdac
path_selector "round-robin 0"
hardware_handler "1 rdac"
failback immediate
features "2 pg_init_retries 50"
no_path_retry 30
rr_min_io 100
}
device {
vendor "DELL"
product "MD36xxf"
path_grouping_policy group_by_prio
prio rdac
path_checker rdac
path_selector "round-robin 0"
hardware_handler "1 rdac"
failback immediate
features "2 pg_init_retries 50"
no_path_retry 30
rr_min_io 100
}
}
```
### NAS iSCSI Config
An example config for a Netapp NAS iSCSI might look like:
```
# NAS iSCSI /etc/multipath.conf
blacklist {
device {
vendor "*"
product "PERC|Perc"
}
device {
vendor "*"
product "Universal Xport"
}
device {
vendor "*"
product "Virtual Disk"
}
}
defaults {
user_friendly_names yes
max_fds max
queue_without_daemon no
}
devices {
device {
vendor "NETAPP"
product "LUN"
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
#
# RHEL5 style
prio_callout "/sbin/mpath_prio_ontap /dev/%n"
# RHEL6 style
# prio ontap
#
features "1 queue_if_no_path"
hardware_handler "0"
path_grouping_policy group_by_prio
failback immediate
rr_weight uniform
rr_min_io 128
path_checker directio
flush_on_last_del yes
}
}
```
## Multipath Names
By default in RHEL/CentOS, the names of the multipath will be in `/dev/mapper/` and begin with "mpath" and be followed by a number (v5) or a letter (v6). A partition within that path will then have "p" followed by it's number. These are controlled by `udev` and a config file installed by the `device-mapper-multipath` RPM; for example on RHEL6/CentOS6 it's named `/lib/udev/rules.d/40-multipath.rules`.
Examples:
```
/dev/mapper/mpath1p2 - 2nd partition on path #1 (1) (v5)
/dev/mapper/mpathbp1 - 1st partition on path #2 (b) (v6)
```
These are a human-friendly format of the WWID triggered by the setting `user_friendly_names yes` in the config file. These can be changed to suit needs - it's easy and can save a lot of confusion later if a dozen LUNs are used as RAW devices (such as in an Oracle RAC).
## Administrating Multipaths
The main tool for administering multipaths is called `multipath` and is normally found in /sbin/ (root only). The primary use day-to-day will be the -l or -ll flags to simply list multipaths and their associated 'real' SCSI devices (paths). Using this tool you can examine the health of the (multi)paths and all associated information.
Example:
```
## DAS multipath
# multipath -l
[...]
VOTING5 (3690b11c0001b99ba0000098f5192345e) dm-5 DELL,MD32xx
size=1.0G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=0 status=active
| `- 2:0:0:4 sdw 65:96 active undef running
`-+- policy='round-robin 0' prio=0 status=enabled
`- 1:0:0:4 sdf 8:80 active undef running
```
The system knows which SCSI devices belong together by their WWID (aka WWN, UUID) that are presented from the storage host - if they match, they belong together in a multipath. From the above example LUN, using the -v3 flag will show they match:
```
## DAS WWIDs (WWN/UUID)
# multipath -v3
[...]
uuid hcil dev dev_t pri dm_st chk_st vend/p
3690b11c0001b99ba0000098f5192345e 2:0:0:4 sdw 65:96 14 undef ready DELL,M
3690b11c0001b99ba0000098f5192345e 1:0:0:4 sdf 8:80 9 undef ready DELL,M
```
The WWID 3690b11c0001b99ba0000098f5192345e matches on both SCSI devices, so now the multipath daemon knows they belong together and creates a pseudo device for us to work with. If one underlying path (device) fails, it goes over to the other one without any manual intervention. Magic.
There are other uses of the multipath tool, such as the -f/-F flags (flush paths) and -p (change policies) -- be careful using these on a live server. Check the man page for detailed information, and know there is a -d (dry run) option to test things before commit. It's sometimes easier to restart the multipathd daemon instead depending on what you're doing (such as renaming - see below).
### Partitioning Multipaths
The tool `kpartx` is what an administrator will use to have the kernel re-examine newly partitioned multipaths and create new device entries for us; it's the equivalent of using `partx` on normal devices.
```
## Normal SCSI device
# parted /dev/sdb (create new partition 1)
# partx -a /dev/sdb
# ls -1 /dev/sdb*
/dev/sdb
/dev/sdb1
## Multipath device
# parted /dev/mapper/mpathb (create new partition 1)
# kpartx -a /dev/mapper/mpathb
# ls -1 /dev/mapper/mpathb*
/dev/mapper/mpathb
/dev/mapper/mpathbp1
```
The device /dev/mapper/mpathbp1 is now used just like /dev/sdb1 would be for any other tools (mkfs, pvcreate, vgextend, etc.) -- the multipath daemon takes care of routing the actual SCSI commands out to the active device (path) in the multipath to storage.
### Clustered Multipaths
Using the WWIDs as described above will allow you to ensure that if you have a host group of LUNs presented to 2 or more servers match multipaths. **The mapping of a WWID to multipath on one node must match on all other nodes**, otherwise you're writing to different storage areas on different nodes. If your examination finds they do not match you may need to rename them manually - see below.
> Always double-check the WWID to multipath mappings match on all nodes in a cluster\! This may not be quick but it's extremely important the time be spent doing this work. Never assume it's "just right" on a new build.
### Renaming Multipaths
Renaming them is easy - add a new stanza to the bottom of multipath.conf that has a grouping, then rename each one. The setting `user_friendly_names yes` is required in multipath.conf for this to work as expected. For example, here's is a rename of a shared Oracle RAC voting LUN from the spurious name into something that makes sense for use inside Oracle as a RAW device:
```
multipaths {
multipath {
wwid 3690b11c0001b99ba0000098f5192345e
alias VOTING5
}
}
```
Restart `multipathd` service and now the multipath is named like so:
```
# ls -1 /dev/mapper/VOTING5
/dev/mapper/VOTING5
```
The partitions within a renamed multipath follow the same convention, 'p' followed by a number. You would expect names like `/dev/mapper/VOTING5p1`, `/dev/mapper/VOTING5p2`, etc. if you partitioned this LUN for use as a normal filesystem.
## Multipath Ownership
One of the other common desires is to set the UID, GID and mode on the multipaths; alas there's a different method for RHEL/CentOS v5 and v6.
### RHEL5 / CentOS5
This is done in the same block schema as renaming them like so:
```
multipaths {
multipath {
wwid 3690b11c0001b99ba0000098f5192345e
alias VOTING5
uid 503
gid 503
mode 755
}
}
```
Note that the system requires the numerical UID/GID and octal mode as shown above.
### RHEL6 / CentOS6
The above method was deprecated in RHEL6 in favor of udev rules - Red Hat's article on how to set it up is wee bit lacking; use a ruleset like this instead of their official doc:
```
/etc/udev/rules.d/12-dm-permissions.rules
ENV{DM_NAME}=="VOTING5", OWNER:="oracle", GROUP:="oinstall", MODE:="660"
```
This is based on renaming the multipath outlined above; to get the value of the DM\_NAME you are trying to rename the "udevadm" tool is used to query the raw device-map node.
- Get the raw node-name with a simple ls:
```
# ls -l /dev/mapper/VOTING5
lrwxrwxrwx 1 root root 7 May 30 22:41 /dev/mapper/VOTING5 -> ../dm-5
```
- Use that dm-?? number against the sysfs interface for it:
```
# udevadm info --query=all --path=/devices/virtual/block/dm-5/
P: /devices/virtual/block/dm-5
N: dm-5
S: mapper/VOTING5
S: disk/by-id/dm-name-VOTING5
S: disk/by-id/dm-uuid-mpath-3690b11c0001b99ba0000098f5192345e
S: block/253:5
E: UDEV_LOG=3
E: DEVPATH=/devices/virtual/block/dm-5
E: MAJOR=253
E: MINOR=5
E: DEVNAME=/dev/dm-5
E: DEVTYPE=disk
E: SUBSYSTEM=block
E: DM_SBIN_PATH=/sbin
E: DM_UDEV_PRIMARY_SOURCE_FLAG=1
E: DM_UDEV_RULES_VSN=2
E: DM_NAME=VOTING5
E: DM_UUID=mpath-3690b11c0001b99ba0000098f5192345e
E: DM_SUSPENDED=0
E: MPATH_SBIN_PATH=/sbin
E: DEVLINKS=/dev/mapper/VOTING5 /dev/disk/by-id/dm-name-VOTING5 /dev/disk/by-id/dm-uuid-mpath-3690b11c0001b99ba0000098f5192345e /dev/block/253:5
```
Use any line item that begins with "E: " as the match clause in your udev rule; it seems the most obvious to use DM\_NAME however your situation may require using one of the others.
## References
- <http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html-single/DM_Multipath/index.html#multipath_consistent_names>
- <http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html-single/DM_Multipath/index.html#config_file_defaults>
- <http://technologist.pro/storage/multipathing-netapp-lun-on-rhel-5-3>
- <https://github.com/torvalds/linux/blob/master/drivers/scsi/device_handler/scsi_dh_rdac.c>

455
md/drbd_build_steps.md Normal file
View file

@ -0,0 +1,455 @@
# DRBD Build Steps
## Contents
- [Overview](#overview)
- [Conventions](#conventions)
- [Group 1 - CentOS](#group-1---centos)
- [Group 2 - Debian](#group-2---debian)
- [Node Prep](#node-prep)
- [Hosts file](#hosts-file)
- [CentOS](#centos)
- [Debian](#debian)
- [IPTables](#iptables)
- [CentOS](#centos-1)
- [Debian](#debian-1)
- [Software Installation](#software-installation)
- [CentOS](#centos-2)
- [Debian](#debian-2)
- [Storage Prep](#storage-prep)
- [DRBD Resource Prep](#drbd-resource-prep)
- [Common Settings](#common-settings)
- [Cloud Example](#cloud-example)
- [Resource Settings](#resource-settings)
- [CentOS](#centos-3)
- [Debian](#debian-3)
- [DRBD Resource Init](#drbd-resource-init)
- [CentOS](#centos-4)
- [Debian](#debian-4)
- [Filesystem Build](#filesystem-build)
- [Testing](#testing)
- [CentOS](#centos-5)
- [Debian](#debian-5)
- [HA Failover](#ha-failover)
- [References](#references)
## Overview
From <http://www.drbd.org/>:
> "DRBD® refers to block devices designed as a building block to form high availability (HA) clusters. This is done by mirroring a whole block device via an assigned network. DRBD can be understood as network based raid-1."
DRBD is the concept of taking two similar block storage devices and performing a network RAID-1 to connect them for HA redundancy. The block devices can be CBS (Cloud Block Storage), VMware vDisks, local in-chassis RAID arrays and so forth. The only requirements are they be unique and have a (preferably) private network between them for the replication. As with a traditional RAID-1 array, only one of the block devices is usable (live) - DRBD performs a block-level replication between the two devices in one direction only. However, DRBD allows for making either one of the devices Master (live) and shifting back and forth dynamically with a few commands.
## Conventions
This article article will use 2 groups of cloud servers as examples:
```
[root@drbd1 ~]# ip a | grep "inet " | grep 192
inet 192.168.5.4/24 brd 192.168.5.255 scope global eth2
[root@drbd2 ~]# ip a | grep "inet " | grep 192
inet 192.168.5.2/24 brd 192.168.5.255 scope global eth2
root@drbd3:~# ip a | grep "inet " | grep 192
inet 192.168.5.1/24 brd 192.168.5.255 scope global eth2
root@drbd4:~# ip a | grep "inet " | grep 192
inet 192.168.5.3/24 brd 192.168.5.255 scope global eth2
```
### Group 1 - CentOS
- drbd1 and drbd2
- CentOS 6.5
- /dev/xvde block devices
### Group 2 - Debian
- drbd3 and drbd4
- Debian 7 Stable
- 20G /dev/xvde block devices
## Node Prep
A working DRBD setup requires at a minimum:
- 2x servers with similar block devices
- DRBD kernel module and userspace utilities
- Private network between the servers
- iptables port 7788 open between servers on the Private network
- /etc/hosts configured
- NTP synchronized
For future growth, LVM should be used underneath the DRBD implementation; the underlying PV/VG/LV can then be grown and the DRBD device ("resource") resized with the [drbdadm resize resource](http://www.drbd.org/users-guide-8.3/s-resizing.html#s-growing-online) command online.
> Timing is critical to proper operation - ensure NTP is configured properly.
### Hosts file
We need to ensure that the 2 servers can find each other on the private network as typical with any type of cluster build. When initializing the resource below, the drbdadm tool uses the hostname to match what's in the resource configuration file so it's important they align. In our examples the servers are in the domain .local as shown below, their FQDN hostnames are properly configured with drbdX.local as expected.
#### CentOS
```
/etc/hosts
192.168.5.4 drbd1.local
192.168.5.2 drbd2.local
```
#### Debian
```
/etc/hosts
192.168.5.1 drbd3.local
192.168.5.3 drbd4.local
```
### IPTables
We'll add a basic rule to allow all communication on the private 192.168.5.0/24 subnet between the nodes. This can be tuned to be more granular as required.
#### CentOS
```
# vi /etc/sysconfig/iptables
...
-A INPUT -s 192.168.5.0/24 -j ACCEPT
...
# service iptables restart
```
#### Debian
```
# apt-get update; apt-get install iptables-persistent
# vi /etc/iptables/rules.v4
...
-A INPUT -s 192.168.5.0/24 -j ACCEPT
...
# service iptables-persistent restart
# insserv iptables-persistent
```
### Software Installation
#### CentOS
CentOS requires the use of the <http://www.elrepo.org/> RPM packages; this provides the DKMS-based kernel module and userspace toolset.
```
rpm -Uvh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
yum repolist
yum install drbd83-utils kmod-drbd83 dkms lvm2 ntp ntpdate
service ntpd restart && chkconfig ntpd on
reboot
```
Note that this installation will pull in kernel-devel and gcc (for DKMS) and a few device-mapper packages (for LVM2).
#### Debian
The Debian 7 kernel includes the drbd.ko module as a stock item; all that's needed is to install the userspace toolset on all nodes.
```
apt-get update
apt-get install --no-install-recommends drbd8-utils lvm2 ntp ntpdate
service ntp restart && insserv -v ntp
reboot
```
Note that without --no-install-recommends apt will install perl and other tools.
## Storage Prep
Create a single volume group and logical volume from the storage on each node in the cluster, but do not create a filesystem - that comes later.
```
parted -s -- /dev/xvde mktable gpt
parted -s -- /dev/xvde mkpart primary ext3 2048s 100%
parted -s -- /dev/xvde set 1 lvm on
pvcreate /dev/xvde1
vgcreate vgdata00 /dev/xvde1
lvcreate -l 100%VG -n drbd00 vgdata00
```
## DRBD Resource Prep
### Common Settings
The file `/etc/drbd.d/global_common.conf` exists on both nodes; as the default content will vary from release to release it's best to edit the file provided instead of creating a new one overtop – in general you most likely want to disable the usage-count for performance and set the syncer rate – changes made to this file and the default options should be researched to provide optimum settings for the platform DRBD is being deployed on.
> TODO: provide common default configurations of the global settings for various scenarios
#### Cloud Example
```
/etc/drbd.d/global_common.conf
global { usage-count no; }
common {
syncer { rate 10M; }
}
```
### Resource Settings
We create configuration files on both nodes that ties the two servers together with their new storage - note that the name of the file should be the name of the resource as a Best Practice. As we're building two different clusters on the same IP subnet, we'll be careful to name them uniquely to prevent any chance of collision at runtime. A shared secret was generated using pwgen as shown below.
#### CentOS
```
/etc/drbd.d/cent00.res
resource cent00 {
protocol C;
startup { wfc-timeout 0; degr-wfc-timeout 120; }
disk { on-io-error detach; }
net { cram-hmac-alg "sha1"; shared-secret "m9bTmbsK4quE"; }
on drbd1.local {
device /dev/drbd0;
disk /dev/vgdata00/drbd00;
meta-disk internal;
address 192.168.5.4:7788;
}
on drbd2.local {
device /dev/drbd0;
disk /dev/vgdata00/drbd00;
meta-disk internal;
address 192.168.5.2:7788;
}
}
```
#### Debian
```
/etc/drbd.d/deb00.res
resource deb00 {
protocol C;
startup { wfc-timeout 0; degr-wfc-timeout 120; }
disk { on-io-error detach; }
net { cram-hmac-alg "sha1"; shared-secret "m9bTmbsK4quE"; }
on drbd3.local {
device /dev/drbd0;
disk /dev/vgdata00/drbd00;
meta-disk internal;
address 192.168.5.1:7788;
}
on drbd4.local {
device /dev/drbd0;
disk /dev/vgdata00/drbd00;
meta-disk internal;
address 192.168.5.3:7788;
}
}
```
## DRBD Resource Init
On both nodes, the drbdadm tool is used to initialize the resource. After the initialization and service start, on one node only we start the synchronization process. We then track the progress of the init - in our example, we'll use drbd1 as CentOS primary and drbd4 as Debian primary to show how it works in either node.
### CentOS
Create the resource, start the service and start the sync:
```
[root@drbd1 ~]# drbdadm create-md cent00
[root@drbd2 ~]# drbdadm create-md cent00
[root@drbd1 ~]# service drbd start; chkconfig drbd on
[root@drbd2 ~]# service drbd start; chkconfig drbd on
[root@drbd1 ~]# drbdadm -- --overwrite-data-of-peer primary cent00
```
Check progress:
```
[root@drbd1 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2013-09-27 16:00:43
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:1124352 nr:0 dw:0 dr:1125016 al:0 bm:68 lo:0 pe:1 ua:0 ap:0 ep:1 wo:f oos:19842524
[>...................] sync'ed: 5.4% (19376/20472)M
finish: 0:31:21 speed: 10,536 (10,312) K/sec
[root@drbd1 ~]# drbdadm -- status cent00
<drbd-status version="8.3.16" api="88">
<resources config_file="/etc/drbd.conf">
<resource minor="0" name="cent00" cs="SyncSource" ro1="Primary" ro2="Secondary" ds1="UpToDate" ds2="Inconsistent" resynced_percent="14.6" />
</resources>
</drbd-status>
```
### Debian
Create the resource, start the service and start the sync:
```
root@drbd3:~# drbdadm create-md deb00
root@drbd4:~# drbdadm create-md deb00
root@drbd3:~# service drbd start; insserv drbd
root@drbd4:~# service drbd start; insserv drbd
root@drbd4:~# drbdadm -- --overwrite-data-of-peer primary deb00
```
Check progress:
```
root@drbd4:~# cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: F937DCB2E5D83C6CCE4A6C9
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:808960 nr:0 dw:0 dr:809624 al:0 bm:49 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:20157788
[>....................] sync'ed: 3.9% (19684/20472)Mfinish: 0:32:40 speed: 10,264 (10,112) K/sec
root@drbd4:~# drbdadm -- status deb00
<drbd-status version="8.3.13" api="88">
<resources config_file="/etc/drbd.conf">
<resource minor="0" name="deb00" cs="SyncSource" ro1="Primary" ro2="Secondary" ds1="UpToDate" ds2="Inconsistent" resynced_percent="15.0" />
</resources>
</drbd-status>
```
## Filesystem Build
> Remember this is **not** a shared filesystem, you can only format/mount/etc. on the Primary node
It's best to wait until the initial synchronization is complete; as per the above use one of these methods to watch and wait for 100% completion; the sync can take awhile depending on the size of the block storage and the speed of the network between them.
```
cat /proc/drbd
drbdadm -- status <resource> ('cent00' or 'deb00' in these examples)
```
A proper looking sync shows **both datastores** (ds1, ds2) as **UpToDate** like so:
```
[root@drbd1 ~]# drbdadm -- status cent00
<drbd-status version="8.3.16" api="88">
<resources config_file="/etc/drbd.conf">
<resource minor="0" name="cent00" cs="Connected" ro1="Primary" ro2="Secondary" ds1="UpToDate" ds2="UpToDate" />
</resources>
</drbd-status>
root@drbd4:~# drbdadm -- status deb00
<drbd-status version="8.3.13" api="88">
<resources config_file="/etc/drbd.conf">
<resource minor="0" name="deb00" cs="Connected" ro1="Primary" ro2="Secondary" ds1="UpToDate" ds2="UpToDate" />
</resources>
</drbd-status>
```
After that's complete, normal methodology is use to format and mount the device node as defined in the resource config. Typically - as listed in the config - the first resource is `/dev/drbd0`, the second `/dev/drbd1` and so forth. If you've lost track the devfs tree can help you with a simple ls:
```
[root@drbd1 ~]# ls -l /dev/drbd/by-res/
total 0
lrwxrwxrwx 1 root root 11 Jul 18 19:03 cent00 -> ../../drbd0
root@drbd4:~# ls -l /dev/drbd/by-res/
total 0
lrwxrwxrwx 1 root root 11 Jul 18 19:03 deb00 -> ../../drbd0
```
Double-check who is the Primary – this is the **ro1** and **ro2** information shown with the `drbdadm status` command as shown above. Notice in this example that drbd1 shows r01=Primary and drbd2 shows ro1=Secondary – we know we should be formatting, mounting, etc. on drbd1 for this work once synchronization is complete. On the Debian nodes, we see that drbd4 shows r01=Primary as expected.
```
[root@drbd1 ~]# drbdadm -- status cent00
<drbd-status version="8.3.16" api="88">
<resources config_file="/etc/drbd.conf">
<resource minor="0" name="cent00" cs="Connected" ro1="Primary" ro2="Secondary" ds1="UpToDate" ds2="UpToDate" />
</resources>
</drbd-status>
[root@drbd2 ~]# drbdadm -- status cent00
<drbd-status version="8.3.13" api="88">
<resources config_file="/etc/drbd.conf">
<resource minor="0" name="deb00" cs="Connected" ro1="Primary" ro2="Secondary" ds1="UpToDate" ds2="UpToDate" />
</resources>
</drbd-status>
```
We'll use a standard ext4 filesystem for this build on both CentOS and Debian:
```
mkfs.ext4 -v -m0 /dev/drbd0
mkdir /data
mount /dev/drbd0 /data
df -h
```
They look like what you'd expect:
```
[root@drbd1 ~]# df -h /data
Filesystem Size Used Avail Use% Mounted on
/dev/drbd0 20G 172M 20G 1% /data
root@drbd4:~# df -h /data
Filesystem Size Used Avail Use% Mounted on
/dev/drbd0 20G 172M 20G 1% /data
```
## Testing
Test that the resource can be made active and mounted on the Secondary node. We'll write a test file, unmount, demote the Primary to Secondary, mount on the partner node and check the test file.
### CentOS
```
[root@drbd1 ~]# touch /data/test.file
[root@drbd1 ~]# umount /data
[root@drbd1 ~]# drbdadm secondary cent00
[root@drbd2 ~]# drbdadm primary cent00
[root@drbd2 ~]# mkdir /data
[root@drbd2 ~]# mount /dev/drbd0 /data
[root@drbd2 ~]# ls -l /data/
total 16
drwx------ 2 root root 16384 Jul 18 19:41 lost+found
-rw-r--r-- 1 root root 0 Jul 18 19:47 test.file
```
### Debian
```
root@drbd4:~# touch /data/test.file
root@drbd4:~# umount /data
root@drbd4:~# drbdadm secondary deb00
root@drbd3:~# drbdadm primary deb00
root@drbd3:~# mkdir /data
root@drbd3:~# mount /dev/drbd0 /data
root@drbd3:~# ls -l /data/
total 16
drwx------ 2 root root 16384 Jul 18 19:41 lost+found
-rw-r--r-- 1 root root 0 Jul 18 19:47 test.file
```
## HA Failover
> TODO: Build this section some day
## References
- <http://www.drbd.org/users-guide-8.3/>
- <http://www.drbd.org/users-guide-8.3/s-resizing.html#s-growing-online>

228
md/fonts_and_linux.md Normal file
View file

@ -0,0 +1,228 @@
# Fonts and Linux
## Contents
- [Configuration Files](#configuration-files)
- [Remapping Fonts](#remapping-fonts)
- [Whitelisting and Blacklisting](#whitelisting-and-blacklisting)
- [Disable Hinting](#disable-hinting)
- [LCD Optimization](#lcd-optimization)
- [Local Fonts](#local-fonts)
- [Luxi Sans](#luxi-sans)
- [gVim Fonts](#gvim-fonts)
- [LightDM Fonts](#lightdm-fonts)
- [References](#references)
## Configuration Files
The exact location of the configuration depends on which release of which distro and the Desktop Environment being used.
| Distro | Location |
| ---------------- | ------------------------------- |
| Fedora 17- | ~/.fonts.conf |
| Fedora 18+, Arch | ~/.config/fontconfig/fonts.conf |
## Remapping Fonts
This will remap Courier to Liberation Mono and Caladea to Carlito - Google Fonts have a habit of "stealing" these typefaces resulting in unexpected fonts when browsing the general web. Substitute **DejaVu Sans Mono** for Liberation Mono for a similar result depending on your desktop.
```
<?xml version="1.0"?><!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
<match target="font">
<edit name="hinting" mode="assign"><bool>false</bool></edit>
</match>
<match target="pattern">
<test qual="any" name="family" compare="eq">
<string>Courier</string>
</test>
<edit name="family" mode="prepend" binding="same">
<string>Liberation Mono</string>
</edit>
</match>
<alias>
<family>Courier</family>
<prefer>
<family>Liberation Mono</family>
</prefer>
</alias>
<selectfont>
<rejectfont>
<pattern>
<patelt name="scalable"><bool>false</bool></patelt>
</pattern>
</rejectfont>
</selectfont>
<selectfont>
<rejectfont>
<pattern>
<patelt name="family">
<string>Caladea</string>
</patelt>
</pattern>
<pattern>
<patelt name="family">
<string>Carlito</string>
</patelt>
</pattern>
</rejectfont>
</selectfont>
</fontconfig>
```
## Whitelisting and Blacklisting
The element `<selectfont>` is used in conjunction with the `<acceptfont>` and `<rejectfontfont>` elements to selectively whitelist or blacklist fonts from the resolve list and match requests. The simplest and most typical use case it to reject one font that is needed to be installed, however is getting matched for a generic font query that is causing problems within application user interfaces.
First obtain the Family name as listed in the font itself:
```
$ fc-scan .fonts/lklug.ttf --format='%{family}\n'
LKLUG
```
Then use that Family name in a `<rejectfontfont>` stanza:
```
<selectfont>
<rejectfont>
<pattern>
<patelt name="family" >
<string>LKLUG</string>
</patelt>
</pattern>
</rejectfont>
</selectfont>
```
Typically when both elements are combined, `<rejectfontfont>` is first used on a more general matching glob to reject a large group (such as a whole directory), then `<acceptfont>` is used after it to whitelist individual fonts out of the larger blacklisted group.
```
<selectfont>
<rejectfont>
<glob>/usr/share/fonts/OTF/*</glob>
</rejectfont>
<acceptfont>
<pattern>
<patelt name="family" >
<string>Monaco</string>
</patelt>
</pattern>
<cceptfont>
</selectfont>
```
## Disable Hinting
Chrome in XFCE can have issues with hinting resulting in bad display for instance, it might be handy to disable hinting:
```
<?xml version="1.0"?><!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
<match target="font">
<edit name="hinting" mode="assign"><bool>false</bool></edit>
</match>
</fontconfig>
```
## LCD Optimization
Assuming a standard RGB subpixel ordering:
```
<?xml version="1.0"?><!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
<match target="font">
<edit name="antialias" mode="assign">
<bool>true</bool>
</edit>
<edit name="autohint" mode="assign">
<bool>false</bool>
</edit>
<edit name="hinting" mode="assign">
<bool>true</bool>
</edit>
<edit name="hintstyle" mode="assign">
<const>hintslight</const>
</edit>
<edit name="rgba" mode="assign">
<const>rgb</const>
</edit>
<edit mode="assign" name="lcdfilter">
<const>lcddefault</const>
</edit>
<edit mode="assign" name="embeddedbitmap">
<bool>false</bool>
</edit>
</match>
</fontconfig>
```
## Local Fonts
Keeping fonts locally in your home directory:
- Make a new directory in your home called `.fonts` (note leading `.`)
- Copy the downloaded TTF file into this directory
- Change directory to .fonts (`cd ~/.fonts/`)
- Run the command: `mkfontscale` (creates fonts.scale)
- Run the command: `mkfontdir` (creates fonts.dir)
- Run the command: `fc-cache -fv ~/.fonts` (rebuilds local cache)
- Test with: `fc-cache`
## Luxi Sans
This disappeared with Fedora 8 due to [licensing issues](https://fedoraproject.org/wiki/Luxi_fonts), Google for this file and unpack it to get the TTF files you can install in your [home directory](#Local_Fonts).
- `xorg-x11-fonts-truetype-7.2-3.fc8.noarch.rpm`
## gVim Fonts
A snippet for your `~/.vimrc` to map the editing font in [gVim](http://www.vim.org/):
```
~/.vimrc
" GVIM preferences
if has("gui_running")
let os=substitute(system('uname'), '\n', '', '')
if os == 'Darwin' || os == 'Mac'
set guifont=Menlo:h16
elseif os == 'Linux'
set guifont=Monospace\ 12
endif
endif
```
## LightDM Fonts
The typical config file is `/etc/lightdm/lightdm-gtk-greeter.conf`:
```
/etc/lightdm/lightdm-gtk-greeter.conf
[greeter]
font-name=Luxi Sans 12
xft-antialias=true
xft-hintstyle=hintnone
xft-rgba=rgb
```
## References
- <http://www.freedesktop.org/software/fontconfig/fontconfig-user.html>

483
md/glusterfs_build_steps.md Normal file
View file

@ -0,0 +1,483 @@
# GlusterFS Build Steps
## Contents
- [Overview](#overview)
- [Prerequisites](#prerequisites)
- [Build Document Setup](#build-document-setup)
- [Node Prep](#node-prep)
- [Configure /etc/hosts and iptables](#configure-etchosts-and-iptables)
- [Granular iptables](#granular-iptables)
- [Install Packages](#install-packages)
- [Prepare Bricks](#prepare-bricks)
- [GlusterFS Setup](#glusterfs-setup)
- [Start glusterfsd daemon](#start-glusterfsd-daemon)
- [Build Peer Group](#build-peer-group)
- [Volume Creation](#volume-creation)
- [Replicated Volume](#replicated-volume)
- [Distributed-Replicated Volume](#distributed-replicated-volume)
- [Volume Deletion](#volume-deletion)
- [Clearing Bricks](#clearing-bricks)
- [Adding Bricks](#adding-bricks)
- [Volume Options](#volume-options)
- [Client Mounts](#client-mounts)
- [FUSE Client](#fuse-client)
- [NFS Client](#nfs-client)
- [References](#references)
## Overview
Prior to starting work, a fundamental decision must be made - what type of Volume(s) need to be used for the given scenario. While 6 methods exist, two are used most often to achieve different results:
- **Replicated**: This type of Volume provides a file replication across multiple bricks, it is a best choice for environments where High Availability and High Reliability are CRITICAL, as well as if you wish to self-mount the volume on every node such as with a webserver DocumentRoot - the GlusterFS nodes are their own clients.
- Files are copied to each brick in the volume similar to a RAID-1, however you can have 3+ bricks and an odd number as well; usable space is the size of one brick, and all files written to one brick are replicated to all others. This makes the most sense if you are going to self-mount the GlusterFS volume, for instance as the web docroot (/var/www) or similar where all files must reside on that node. The value passed to `replica` is the same number of nodes in the volume.
- **Distributed-Replicated**: In this scenario files are distributed across replicated bricks in the volume. You can use this type of volume in environments where the requirement is to scale storage as well as having high availability. Volumes of this type also offer improved read performance in most environments, and are most common type of volumes used when clients are external to the GlusterFS nodes themselves.
- Somewhat like a RAID-10, an even number of bricks must be used; usable space is the size of the combined bricks passed to the `replica` value. For example, if there are **4 bricks of 20G** and you pass `replica 2` to the creation, your files will distribute to 2 nodes (40G) and replicate to 2 nodes. With **6 bricks of 20G** and `replica 3` it would distribute to 3 nodes (60G) and replicate to 3 nodes, but if you used `replica 2` it would distribute to 2 nodes (40G) and replicate to 4 nodes in pairs. This would be used when your clients are external to the cluster, not local self-mounts.
All the fundamental work in this document is the same except for the one step where the Volume is created as outlined above with the `replica` keyword. Using Striped-based volumes is not covered here.
## Prerequisites
1. 2 or more servers with separate Storage
2. Private network between servers
## Build Document Setup
This build document will use the following setup that can be easily stood up; using Cloud block devices is no different than VMware vDisks, SAN/DAS LUNs, iSCSI, etc.
- 4x Performance 1 Tier 2 Rackspace Cloud servers - a 20G /dev/xvde ready to use for each brick
- 1x Cloud Private Network on 192.168.3.0/24 for GlusterFS communication
- GlusterFS 3.7 installed from Vendor package repository
## Node Prep
- Configure /etc/hosts and iptables
- Install base toolset(s)
- Install GlusterFS software
- Connect GlusterFS nodes
### Configure /etc/hosts and iptables
In lieu of using DNS, we prepare /etc/hosts so that every machine and ensure they can talk to each other. All servers have the name `gluster`_N_ as a hostname, so we'll use `glus`_N_ for our private communication layer between nodes.
```
# vi /etc/hosts
192.168.3.2 glus1
192.168.3.4 glus2
192.168.3.1 glus3
192.168.3.3 glus4
# ping -c2 glus1; ping -c2 glus2; ping -c2 glus3; ping -c2 glus4
## Red Hat oriented:
# vi /etc/sysconfig/iptables
-A INPUT -s 192.168.3.0/24 -j ACCEPT
# service iptables restart
## Debian oriented
# vi /etc/iptables/rules.v4
-A INPUT -s 192.168.3.0/24 -j ACCEPT
# service iptables-persistent restart
```
#### Granular iptables
The above generic iptables rule opens all ports to the subnet; if more granular setup is required:
- **111** - portmap / rpcbind
- **24007** - GlusterFS Daemon
- **24008** - GlusterFS Management
- **38465** to **38467** - Required for GlusterFS NFS service
- **24009** to +X - GlusterFS versions less than 3.4, OR
- **49152** to +X - GlusterFS versions 3.4 and later
Each brick for every volume on the host requires it’s own port. For every new brick, one new port will be used starting at **24009** for GlusterFS versions below 3.4 and **49152** for version 3.4 and above.
**Example**: If you have one volume with two bricks, you will need to open 24009 - 24010, or 49152 - 49153.
### Install Packages
1. Install the basic packages for partitioning, LVM2 and XFS
2. Install the GlusterFS repository and glusterfs\* packages
3. Disable automatic updates of gluster\* packages
Some of the required packages may already be installed on the cluster nodes.
```
## YUM/RPM Based:
# yum -y install parted lvm2 xfsprogs
# wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo
# yum -y install glusterfs glusterfs-fuse glusterfs-server
## Ubuntu based (Default Ubuntu repo has glusterfs 3.4, here's how to install 3.7):
# apt-get install lvm2 xfsprogs python-software-properties
# add-apt-repository ppa:gluster/glusterfs-3.7
# apt-get update
# apt-get install glusterfs-server
```
Ensure that the gluster\* packages are filtered out of automatic updates; upgrades while it's running can crash the bricks.
```
# grep ^exclude /etc/yum.conf
exclude=kernel* gluster*
## Ubuntu method:
# apt-mark hold glusterfs*
```
### Prepare Bricks
1. Partition block devices
2. Create LVM foundation
3. Prepare volume bricks
The underlying bricks are a standard filesystem and mount point. However, make sure to mount each brick in such a way so as to discourage any use from changing to the directory and writing to the underlying bricks themselves. **Writing directly to a Brick will corrupt your Volume\!**
The bricks must be unique per node, and there should be a directory within the mount to use in volume creation. Attempting to create a replicated volume using the top-level of the mounts results in an error with instructions to use a subdirectory.
```
all nodes:
# parted -s -- /dev/xvde mktable gpt
# parted -s -- /dev/xvde mkpart primary 2048s 100%
# parted -s -- /dev/xvde set 1 lvm on
# partx -a /dev/xvde
# pvcreate /dev/xvde1
# vgcreate vgglus1 /dev/xvde1
Logical Volumes
---------------
Standard LVM:
# lvcreate -l 100%VG -n gbrick1 vgglus1
For GlusterFS snapshot support:
# lvcreate -l 100%FREE --thinpool lv_thin vgglus1
# lvcreate --thin -V $(lvdisplay /dev/vgglus1/lv_thin | awk '/LV\ Size/ { print $3 }')G -n gbrick1 vgglus1/lv_thin
Filesystems for bricks
----------------------
For XFS bricks: (recommended)
# mkfs.xfs -i size=512 /dev/vgglus1/gbrick1
# echo '/dev/vgglus1/gbrick1 /data/gluster/gvol0 xfs inode64,nobarrier 0 0' >> /etc/fstab
# mkdir -p /data/gluster/gvol0
# mount /data/gluster/gvol0
For ext4 bricks:
# mkfs.ext4 /dev/vgglus1/gbrick1
# echo '/dev/vgglus1/gbrick1 /data/gluster/gvol0 ext4 defaults,user_xattr,acl 0 0' >> /etc/fstab
# mkdir -p /data/gluster/gvol0
# mount /data/gluster/gvol0
glus1:
# mkdir -p /data/gluster/gvol0/brick1
glus2:
# mkdir -p /data/gluster/gvol0/brick1
glus3:
# mkdir -p /data/gluster/gvol0/brick1
glus4:
# mkdir -p /data/gluster/gvol0/brick1
```
## GlusterFS Setup
### Start glusterfsd daemon
The daemon can be restarted at runtime as well:
```
## Red Hat based:
# service glusterd start
# chkconfig glusterd on
```
### Build Peer Group
This is what's known as a **Trusted Storage Pool** in the GlusterFS world. Note that as of early release of version 3, you only need to probe all other nodes from glus1. The peer list is then automatically distributed to all peers from there.
```
glus1:
# gluster peer probe glus2
# gluster peer probe glus3
# gluster peer probe glus4
# gluster peer status
[root@gluster1 ~]# gluster pool list
UUID Hostname State
734aea4c-fc4f-4971-ba3d-37bd5d9c35b8 glus4 Connected
d5c9e064-c06f-44d9-bf60-bae5fc881e16 glus3 Connected
57027f23-bdf2-4a95-8eb6-ff9f936dc31e glus2 Connected
e64c5148-8942-4065-9654-169e20ed6f20 localhost Connected
```
### Volume Creation
We will set up basic auth restrictions to only our private subnet as by default glusterd NFS allows global read/write during Volume creation. glusterd automatically starts NFSd on each server and exports the volume through it from each of the nodes. The reason for this behaviour is that in order to use native client (FUSE) for mounting the volume on clients, the clients have to run exactly same version of GlusterFS packages. If the versions are different there might be differences in the hashing algorithms used by servers and clients and the clients won't be able to connect.
#### Replicated Volume
This example will create replication to all 4 nodes - each node contains a copy of all data and the size of the Volume is the size of a single brick. Notice how the info shows `1 x 4 = 4` in the output.
```
one node only:
# gluster volume create gvol0 replica 4 transport tcp \
glus1:/data/gluster/gvol0/brick1 \
glus2:/data/gluster/gvol0/brick1 \
glus3:/data/gluster/gvol0/brick1 \
glus4:/data/gluster/gvol0/brick1
# gluster volume set gvol0 auth.allow 192.168.3.*,127.0.0.1
# gluster volume set gvol0 nfs.disable off
# gluster volume set gvol0 nfs.addr-namelookup off
# gluster volume set gvol0 nfs.export-volumes on
# gluster volume set gvol0 nfs.rpc-auth-allow 192.168.3.*
# gluster volume set gvol0 performance.io-thread-count 32
# gluster volume start gvol0
[root@gluster1 ~]# gluster volume info gvol0
Volume Name: gvol0
Type: Replicate
Volume ID: 65ece3b3-a4dc-43f8-9b0f-9f39c7202640
Status: Started
Number of Bricks: 1 x 4 = 4
Transport-type: tcp
Bricks:
Brick1: glus1:/data/gluster/gvol0/brick1
Brick2: glus2:/data/gluster/gvol0/brick1
Brick3: glus3:/data/gluster/gvol0/brick1
Brick4: glus4:/data/gluster/gvol0/brick1
Options Reconfigured:
nfs.rpc-auth-allow: 192.168.3.*,127.0.0.1
nfs.export-volumes: on
nfs.addr-namelookup: off
nfs.disable: off
auth.allow: 192.168.3.*
performance.io-thread-count: 32
```
#### Distributed-Replicated Volume
This example will create distributed replication to 2x2 nodes - each pair of nodes contains the data and the size of the Volume is the size of a two bricks. Notice how the info shows `2 x 2 = 4` in the output.
```
one node only:
# gluster volume create gvol0 replica 2 transport tcp \
glus1:/data/gluster/gvol0/brick1 \
glus2:/data/gluster/gvol0/brick1 \
glus3:/data/gluster/gvol0/brick1 \
glus4:/data/gluster/gvol0/brick1
# gluster volume set gvol0 auth.allow 192.168.3.*,127.0.0.1
# gluster volume set gvol0 nfs.disable off
# gluster volume set gvol0 nfs.addr-namelookup off
# gluster volume set gvol0 nfs.export-volumes on
# gluster volume set gvol0 nfs.rpc-auth-allow 192.168.3.*
# gluster volume set gvol0 performance.io-thread-count 32
# gluster volume start gvol0
[root@gluster1 ~]# gluster volume info gvol0
Volume Name: gvol0
Type: Distributed-Replicate
Volume ID: d883f891-e38b-4565-8487-7e50ca33dbd4
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: glus1:/data/gluster/gvol0/brick1
Brick2: glus2:/data/gluster/gvol0/brick1
Brick3: glus3:/data/gluster/gvol0/brick1
Brick4: glus4:/data/gluster/gvol0/brick1
Options Reconfigured:
nfs.rpc-auth-allow: 192.168.3.*
nfs.export-volumes: on
nfs.addr-namelookup: off
nfs.disable: off
auth.allow: 192.168.3.*,127.0.0.1
performance.io-thread-count: 32
```
## Volume Deletion
After ensure that no clients (either local or remote) are mounting the Volume, stop the Volume and delete it.
```
# gluster volume stop gvol0
# gluster volume delete gvol0
```
### Clearing Bricks
If brick(s) were used in a volume and they need to be removed, there's an attribute that GlusterFS had set on the brick subdirectories. This needs to be cleared before they can be reused - or the subdir can be deleted and recreated.
```
glus1:
# setfattr -x trusted.glusterfs.volume-id /data/gluster/gvol0/brick1/
# setfattr -x trusted.gfid /data/gluster/gvol0/brick1
# rm -rf /data/gluster/gvol0/brick1/.glusterfs
glus2:
# setfattr -x trusted.glusterfs.volume-id /data/gluster/gvol0/brick1/
# setfattr -x trusted.gfid /data/gluster/gvol0/brick1
# rm -rf /data/gluster/gvol0/brick1/.glusterfs
glus3:
# setfattr -x trusted.glusterfs.volume-id /data/gluster/gvol0/brick1/
# setfattr -x trusted.gfid /data/gluster/gvol0/brick1
# rm -rf /data/gluster/gvol0/brick1/.glusterfs
glus4:
# setfattr -x trusted.glusterfs.volume-id /data/gluster/gvol0/brick1/
# setfattr -x trusted.gfid /data/gluster/gvol0/brick1
# rm -rf /data/gluster/gvol0/brick1/.glusterfs
...or just deleting all data:
glus1:
# rm -rf /data/gluster/gvol0/brick1
# mkdir /data/gluster/gvol0/brick1
glus2:
# rm -rf /data/gluster/gvol0/brick1
# mkdir /data/gluster/gvol0/brick1
glus3:
# rm -rf /data/gluster/gvol0/brick1
# mkdir /data/gluster/gvol0/brick1
glus4:
# rm -rf /data/gluster/gvol0/brick1
# mkdir /data/gluster/gvol0/brick1
```
### Adding Bricks
Additional bricks can be added to a running Volume easily:
```
# gluster volume add-brick gvol0 glus5:/data/gluster/gvol0/brick1
```
The add-brick command can also be used to change the LAYOUT of your volume. For example, to change a 2 node Distributed volume into a 4 node Distributed-Replicated Volume. After such an operation you **must rebalance** your volume. New files will be automatically created on the new nodes, but the old ones will not get moved.
```
# gluster volume add-brick gvol0 replica 2 \
glus5:/data/gluster/gvol0/brick1 \
glus6:/data/gluster/gvol0/brick1
# gluster rebalance gvol0 start
# gluster rebalance gvol0 status
## If needed (something didn't work right)
# gluster rebalance gvol0 stop
```
> When expanding distributed replicated and distributed striped volumes, you must add a number of bricks that is a multiple of the replica or stripe count. For example, to expand a distributed replicated volume with a replica count of 2, you need to add bricks in multiples of 2 (such as 4, 6, 8, etc.):
>
> ```
> # gluster volume add-brick gvol0 \
> glus5:/data/gluster/gvol0/brick1 \
> glus6:/data/gluster/gvol0/brick1
> ```
### Volume Options
To view configured volume options:
```
# gluster volume info gvol0
Volume Name: gvol0
Type: Replicate
Volume ID: bcbfc645-ebf9-4f83-b9f0-2a36d0b1f6e3
Status: Started
Number of Bricks: 1 x 4 = 4
Transport-type: tcp
Bricks:
Brick1: glus1:/data/gluster/gvol0/brick1
Brick2: glus2:/data/gluster/gvol0/brick1
Brick3: glus3:/data/gluster/gvol0/brick1
Brick4: glus4:/data/gluster/gvol0/brick1
Options Reconfigured:
performance.cache-size: 1073741824
performance.io-thread-count: 64
cluster.choose-local: on
nfs.rpc-auth-allow: 192.168.3.*,127.0.0.1
nfs.export-volumes: on
nfs.addr-namelookup: off
nfs.disable: off
auth.allow: 192.168.3.*,127.0.0.1
```
To set an option for a volume, use the `set` keyword like so:
```
# gluster volume set gvol0 performance.write-behind off
volume set: success
```
To clear an option to a Volume back to defaults, use the `reset` keyword like so:
```
# gluster volume reset gvol0 performance.read-ahead
volume reset: success: reset volume successful
```
## Client Mounts
From a client perspective the GlusterFS Volume can be mounted in two fundamental ways:
1. FUSE Client
2. NFS Client
### FUSE Client
The FUSE client allows the mount to happen with a GlusterFS "round robin" style connection; in /etc/fstab the name of one node is used, however internal mechanisms allows that node to fail and the clients will roll over to other connected nodes in the Trusted Storage Pool. The performance is slightly lower than the NFS method based on tests, however not drastically so - the gain is automatic HA client failover which is typically worth the performance hit.
```
## RPM based:
# wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo
# yum -y install glusterfs glusterfs-fuse
## Ubuntu based(glusterfs-client 3.4 works with glusterfs-server 3.5 but for most the recent version do this):
# add-apt-repository ppa:gluster/glusterfs-3.7
# apt-get update
# apt-get install glusterfs-client
##
##Common:
# vi /etc/hosts
192.168.3.2 glus1
192.168.3.4 glus2
192.168.3.1 glus3
192.168.3.3 glus4
# modprobe fuse
# echo 'glus1:/gvol0 /mnt/gluster/gvol0 glusterfs defaults,_netdev,backup-volfile-servers=glus2 0 0' >> /etc/fstab
# mkdir -p /mnt/gluster/gvol0
# mount /mnt/gluster/gvol0
```
### NFS Client
The standard Linux NFSv3 client tools are used to mount one of the GlusterFS nodes; the performance is typically a little better than the FUSE client, however the downside is the connection is 1-to-1 – of the GlusterFS node goes down the client will not round-robin out to another node. A different solution has to be added such as HAProxy/keepalived, load balancer, etc. in order to provide a floating IP proxy in this use case.
```
## RPM based:
# yum -y install rpcbind nfs-utils
# service rpcbind restart; chkconfig rpcbind on
# service nfslock restart; chkconfig on
## Ubuntu:
# apt-get install nfs-common
##
## Common:
# echo 'glus1:/repvol1 /mnt/gluster/gvol0 nfs rsize=4096,wsize=4096,hard,intr 0 0' >> /etc/fstab
# mkdir -p /mnt/gluster/gvol0
# mount /mnt/gluster/gvol0
```
## References
- <http://www.sohailriaz.com/glusterfs-howto-on-centos-6-x/>
- <http://kaivanov.blogspot.com/2012/07/deploying-glusterfs.html>
- <http://joejulian.name/blog/glusterfs-path-or-a-prefix-of-it-is-already-part-of-a-volume/>
- <http://www.jamescoyle.net/how-to/457-glusterfs-firewall-rules>

205
md/grub_2_info.md Normal file
View file

@ -0,0 +1,205 @@
# Grub 2 Info
## Contents
- [Overview](#overview)
- [Default Menu Entry](#default-menu-entry)
- [Vendor Defaults](#vendor-defaults)
- [RHEL Kernels](#rhel-kernels)
- [Simple Text Mode](#simple-text-mode)
- [Boot Options](#boot-options)
- [Disable Recovery Menus](#disable-recovery-menus)
- [Use Backup Config](#use-backup-config)
- [References](#references)
## Overview
Grub v2 is a radical departure fron grub v1 (aka "grub legacy"); at first it seems daunting, especially if you've spent years working with lilo and grub v1 and their simplistic, single-file index based configuration. These are some of the frequent tasks encountered when working with the new grub v2 environment; be sure to refer to the References links for more detailed and in-depth information. Grub2 is used in RHEL / CentOS 7 and above, Fedora 19 and above, and Ubuntu 14 and above.
## Default Menu Entry
Choose the default boot target via the `GRUB_DEFAULT` setting in `/etc/default/grub`:
- `GRUB_DEFAULT=`_n_ : specify the _n_th entry in `/boot/grub2/grub.cfg`; first entry is `0`
- `GRUB_DEFAULT=`_string_ : choose an entry by name
- `GRUB_DEFAULT=saved` : use the boot target specified in `/boot/grub/grubenv`
### Vendor Defaults
Vendors are using different out of the box defaults for the kernel to boot:
| **Distro** | **Default** |
| ------------- | -------------------- |
| RHEL / CentOS | `GRUB_DEFAULT=saved` |
| Ubuntu | `GRUB_DEFAULT=0` |
> Note that Ubuntu by default is using the 0-index method to boot a kernel; in real terms, this means it always tries to boot the top-level entry named "Ubuntu" that has no discrete kernel name listed. You must first set `GRUB_DEFAULT=saved` in `/etc/default/grub` and run `update-grub` to convert it over to using named kernels as outlined below. Failure to do these steps will result in an outcome you did not expect.
First, make sure that your setup is using a special setting in `/etc/default/grub` - add or replace as needed:
```
/etc/default/grub
GRUB_DEFAULT=saved
```
Rebuild your grub config file after making a backup. You will notice we create a backup **in the boot directory** as the Grub2 bootloader has the capability to use an alternate backup file\! So if your changes end up with a mistake, at a `grub>` menu you can tell it to boot your backup file out of the `/boot` directory. This provides an emergency rollback scenario should things go south during the reboot; see the "Use Backup Config" section in this article.
```
Fedora/RHEL/CentOS:
# cp -a /boot/grub2/grub.cfg{,.bak}
# grub2-mkconfig -o /boot/grub2/grub.cfg
Ubuntu/Debian/Mint:
# cp -a /boot/grub/grub.cfg{,.bak}
# update-grub
```
Get a list of the boot menu items with a simple grep - note that RHEL style systems use a flat top-level menu, whereas Ubuntu uses a top-level single menu "Ubuntu" with several submenus of named kernels; this is just a different implementation by the vendors, but it does require a slightly different grep to find the menu entry names.
```
Fedora/RHEL/CentOS:
# grep "^menuentry" /boot/grub2/grub.cfg | cut -d "'" -f2
Red Hat Enterprise Linux Server (3.10.0-229.1.2.el7.x86_64) 7.1 (Maipo)
Red Hat Enterprise Linux Server (3.10.0-229.1.2.el7.x86_64) 7.1 (Maipo) with debugging
Red Hat Enterprise Linux Server (3.10.0-229.4.2.el7.x86_64) 7.1 (Maipo)
Red Hat Enterprise Linux Server (3.10.0-229.4.2.el7.x86_64) 7.1 (Maipo) with debugging
Red Hat Enterprise Linux Server 7.1 (Maipo), with Linux 3.10.0-229.el7.x86_64
Red Hat Enterprise Linux Server 7.1 (Maipo), with Linux 0-rescue-2c9acae4aae44399a33ff8405cdfda12
Ubuntu/Debian/Mint:
# egrep "^[[:space:]]?(submenu|menuentry)" /boot/grub/grub.cfg | cut -d "'" -f2
Ubuntu
Advanced options for Ubuntu
Ubuntu, with Linux 4.4.0-75-generic
Ubuntu, with Linux 4.4.0-72-generic
Ubuntu, with Linux 4.4.0-43-generic
```
> Ubuntu uses submenus. This means that the line above "Advanced options for Ubuntu" is the top level item, then the lines below it are children; visually:
>
> - Ubuntu
> - Advanced options for Ubuntu
> - Ubuntu, with Linux 4.4.0-75-generic
> - Ubuntu, with Linux 4.4.0-72-generic
> - Ubuntu, with Linux 4.4.0-43-generic
>
> Below you will prepend "Advanced options for Ubuntu", a '\>' symbol, then the name to make it work correctly.
Use one of these lines to set the default with `grub2-set-default` / `grub-set-default` - a 0-based index can be used instead of a name, however on RHEL `grubby` uses the long name so stick to that. `grub(2)-set-default` is just a fancy shell script that runs `grub(2)-editenv` to unset old entries and set the `saved_entry` with your new choice. It updates the `/boot/grub(2)/grubenv` file:
```
Fedora/RHEL/CentOS:
# grub2-set-default "Red Hat Enterprise Linux Server (3.10.0-229.1.2.el7.x86_64) 7.1 (Maipo)"
# grub2-editenv list
saved_entry=Red Hat Enterprise Linux Server (3.10.0-229.1.2.el7.x86_64) 7.1 (Maipo)
# grep saved_entry /boot/grub2/grubenv
saved_entry=Red Hat Enterprise Linux Server (3.10.0-229.1.2.el7.x86_64) 7.1 (Maipo)
===
Ubuntu/Debian/Mint:
# grub-set-default 'Advanced options for Ubuntu>Ubuntu, with Linux 4.4.0-72-generic'
# grub-editenv list
saved_entry=Advanced options for Ubuntu>Ubuntu, with Linux 4.4.0-72-generic
# grep saved_entry /boot/grub2/grubenv
saved_entry=Advanced options for Ubuntu>Ubuntu, with Linux 4.4.0-72-generic
```
## RHEL Kernels
A note about how Red Hat builds their kernel packages; in the above examples `grub2-mkconfig` is used to build new config files; be aware that the RHEL kernel package installs do not use it. Instead, they use a shim `/sbin/new-kernel-pkg` which in turn uses a program called `grubby`. The critical differences:
- **grubby does not use /etc/default/grub** – it uses the as-booted kernel parameters to build a new commandline for the new kernel. If you have custom kernel arguments, be sure to first update `/etc/default/grub`, use `grub2-mkconfig`, reboot to activate the new options then upgrade the new kernel package. This is most commonly done on a freshly installed server before a `yum update` is issued.
- **the grubby menu entries are different** – the Title used in the menu entries is of a different text string than the one produced by `grub2-mkconfig`; this means you should always check and if required set the default kernel to boot when manipulating the grub config file. A quick example:
```
# Build the new config
grub2-mkconfig -o /boot/grub2/grub.cfg
# Get the name of the one you want, let's call it "Red Hat FOOBAR" here
grep "^menuentry" /boot/grub2/grub.cfg | cut -d "'" -f2
# Update the grub environment file /boot/grub2/grubenv saved_entry
grub2-set-default "Red Hat FOOBAR"
grub2-editenv list
```
The Red Hat kernel packages will automatically reconfigure the default kernel to boot as part of the `grubby` process; if simply doing a standard kernel package upgrade or downgrade the environment file will be updated to that package version.
## Simple Text Mode
This is handy for servers - set up grub2 to use basic text mode and show all kernels by default:
```
# vi /etc/default/grub
GRUB_DISABLE_SUBMENU=true
GRUB_GFXMODE=1024x768
GRUB_GFXPAYLOAD_LINUX=keep
Comment this out if present:
#GRUB_TERMINAL_OUTPUT="console"
# cp -a /boot/grub2/grub.cfg{,.bak}
# grub2-mkconfig -o /boot/grub2/grub.cfg
```
## Boot Options
The config variable `GRUB_CMDLINE_LINUX` will be applied to all the auto-detected linux kernels when you rebuild the config; typically the config already has some options, just add to the end as needed. Some common options might be disabling IPv6, setting the LANG variable or blacklisting initrd modules (such as SAN/DAS HBAs).
```
# vi /etc/default/grub
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=vglocal00/swap00 rd.lvm.lv=vglocal00/root00 net.ifnames=0 biosdevname=0 rdblacklist=bfa nomodeset"
# cp -a /boot/grub2/grub.cfg{,.bak}
# grub2-mkconfig -o /boot/grub2/grub.cfg
```
## Disable Recovery Menus
You may wish to disable the "(recovery mode)" (single-user mode) menu entries by setting `GRUB_DISABLE_RECOVERY` and rebuilding your config. In reality all the menu entry does is add the boot option `single` so there's no real value in cluttering up your grub menus.
```
# vi /etc/default/grub
GRUB_DISABLE_RECOVERY="true"
# cp -a /boot/grub2/grub.cfg{,.bak}
# grub2-mkconfig -o /boot/grub2/grub.cfg
```
## Use Backup Config
It's possible something might go wrong during `grub2-mkconfig` and leave you with a corrupt config file, but you won't notice it until you've rebooted. Typically grub will drop you to the shell with an error (or just empty menus). If that happens and you've made a backup use the `configfile` command to use it - as soon as the command is issued the menu should appear.
```
grub2> ls
(hd0) (hd0,msdos3) (hd0,msdos2) (hd0,msdos1)
grub2> configfile (hd0,1)/boot/grub2/grub.cfg.bak
```
## References
- <https://www.gnu.org/software/grub/manual/grub.html>
- <http://fedoraproject.org/wiki/GRUB_2>
- <https://help.ubuntu.com/community/Grub2>
- <https://wiki.archlinux.org/index.php/GRUB>
- <https://www.kernel.org/pub/linux/utils/boot/dracut/dracut.html>
- <https://fedoraproject.org/wiki/Features/DracutHostOnly>

197
md/jumbo_frames.md Normal file
View file

@ -0,0 +1,197 @@
# Jumbo Frames
## Contents
- [Overview](#overview)
- [Configuration](#configuration)
- [RHEL/CentOS](#rhelcentos)
- [Basic Testing](#basic-testing)
- [Node A to B](#node-a-to-b)
- [Bode B to A](#node-b-to-a)
- [Performance Testing](#performance-testing)
- [Node A to B](#node-a-to-b-1)
- [Node B to A](#node-b-to-a-1)
- [Protocol Overhead](#protocol-overhead)
## Overview
[Jumbo frames](http://en.wikipedia.org/wiki/Jumbo_frames) are the concept of opening up the Ethernet frames to a large MTU to be able to push large packets without fragmenting; this can be a common need on a private GigE switched network for Oracle RAC Interconnect between nodes. The accepted standard is 9000 (large enough for 8k payload plus packet overhead). Once your GigE switches have been configured for the new MTU, configure and test your servers.
## Configuration
**Example Setup**:
- Node **A**: 192.168.100.101 (bond1, eth1 + eth5)
- Node **B**: 192.168.100.102 (bond1, eth1 + eth5)
### RHEL/CentOS
On both nodes:
```
# ip link set dev bond1 mtu 9000
# vi /etc/sysconfig/networking-scripts/ifcfg-bond1
add: MTU=9000
```
## Basic Testing
### Node A to B
```
# ifenslave -c bond1 eth5
# ip route get 192.168.100.102
# tracepath -n 192.168.100.102
# ping -c 5 -s 8972 -M do 192.168.100.102
# ifenslave -c bond1 eth1
# ip route get 192.168.100.102
# tracepath -n 192.168.100.102
# ping -c 5 -s 8972 -M do 192.168.100.102
```
### Node B to A
```
# ifenslave -c bond1 eth5
# ip route get 192.168.100.101
# tracepath -n 192.168.100.101
# ping -c 5 -s 8972 -M do 192.168.100.101
# ifenslave -c bond1 eth1
# ip route get 192.168.100.101
# tracepath -n 192.168.100.101
# ping -c 5 -s 8972 -M do 192.168.100.101
```
## Performance Testing
Using `iperf` (available via EPEL) for throughput measurements.
### Node A to B
```
Node B: (receiver)
# ifenslave -c bond1 eth5
# iperf -B 192.168.100.102 -s -u -l 8972 -w 768k
Node A: (sender)
# ifenslave -c bond1 eth5
# iperf -B 192.168.100.101 -c 192.168.100.102 -u \
-b 10G -l 8972 -w 768k -i 2 -t 30
Node B: (receiver)
# ifenslave -c bond1 eth1
# iperf -B 192.168.100.102 -s -u -l 8972 -w 768k
Node A: (sender)
# ifenslave -c bond1 eth1
# iperf -B 192.168.100.101 -c 192.168.100.102 -u \
-b 10G -l 8972 -w 768k -i 2 -t 30
```
### Node B to A
```
Node A: (receiver)
# ifenslave -c bond1 eth5
# iperf -B 192.168.100.101 -s -u -l 8972 -w 768k
Node B: (sender)
# ifenslave -c bond1 eth5
# iperf -B 192.168.100.102 -c 192.168.100.101 -u \
-b 10G -l 8972 -w 768k -i 2 -t 30
Node A: (receiver)
# ifenslave -c bond1 eth1
# iperf -B 192.168.100.101 -s -u -l 8972 -w 768k
Node B: (sender)
# ifenslave -c bond1 eth1
# iperf -B 192.168.100.102 -c 192.168.100.101 -u \
-b 10G -l 8972 -w 768k -i 2 -t 30
```
## Protocol Overhead
Reference: [Theoretical Maximums and Protocol Overhead](http://sd.wareonearth.com/~phil/net/overhead/):
```
Theoretical maximum TCP throughput on GigE using jumbo frames:
(9000-20-20-12)/(9000+14+4+7+1+12)*1000000000/1000000 = 990.042 Mbps
| | | | | | | | | | | |
MTU | | | MTU | | | | | GigE Mbps
| | | | | | | |
IP | | Ethernet | | | | InterFrame Gap (IFG), aka
Header | | Header | | | | InterPacket Gap (IPG), is
| | | | | | a minimum of 96 bit times
TCP | FCS | | | from the last bit of the
Header | | | | FCS to the first bit of
| Preamble | | the preamble
TCP | |
Options Start |
(Timestamp) Frame |
Delimiter |
(SFD) |
|
Inter
Frame
Gap
(IFG)
Theoretical maximum UDP throughput on GigE using jumbo frames:
(9000-20-8)/(9000+14+4+7+1+12)*1000000000/1000000 = 992.697 Mbps
Theoretical maximum TCP throughput on GigE without using jumbo frames:
(1500-20-20-12)/(1500+14+4+7+1+12)*1000000000/1000000 = 941.482 Mbps
Theoretical maximum UDP throughput on GigE without using jumbo frames:
(1500-20-8)/(1500+14+4+7+1+12)*1000000000/1000000 = 957.087 Mbps
Ethernet frame format:
* 6 byte dest addr
* 6 byte src addr
* [4 byte optional 802.1q VLAN Tag]
* 2 byte length/type
* 46-1500 byte data (payload)
* 4 byte CRC
Ethernet overhead bytes:
12 gap + 8 preamble + 14 header + 4 trailer = 38 bytes/packet w/o 802.1q
12 gap + 8 preamble + 18 header + 4 trailer = 42 bytes/packet with 802.1q
Ethernet Payload data rates are thus:
1500/(38+1500) = 97.5293 % w/o 802.1q tags
1500/(42+1500) = 97.2763 % with 802.1q tags
TCP over Ethernet:
Assuming no header compression (e.g. not PPP)
Add 20 IPv4 header or 40 IPv6 header (no options)
Add 20 TCP header
Add 12 bytes optional TCP timestamps
Max TCP Payload data rates over ethernet are thus:
(1500-40)/(38+1500) = 94.9285 % IPv4, minimal headers
(1500-52)/(38+1500) = 94.1482 % IPv4, TCP timestamps
(1500-52)/(42+1500) = 93.9040 % 802.1q, IPv4, TCP timestamps
(1500-60)/(38+1500) = 93.6281 % IPv6, minimal headers
(1500-72)/(38+1500) = 92.8479 % IPv6, TCP timestamps
(1500-72)/(42+1500) = 92.6070 % 802.1q, IPv6, ICP timestamps
UDP over Ethernet:
Add 20 IPv4 header or 40 IPv6 header (no options)
Add 8 UDP header
Max UDP Payload data rates over ethernet are thus:
(1500-28)/(38+1500) = 95.7087 % IPv4
(1500-28)/(42+1500) = 95.4604 % 802.1q, IPv4
(1500-48)/(38+1500) = 94.4083 % IPv6
(1500-48)/(42+1500) = 94.1634 % 802.1q, IPv6
```

318
md/kernel_module_weak_updates.md Normal file
View file

@ -0,0 +1,318 @@
# Kernel Module Weak Updates
## Contents
- [Overview](#overview)
- [Module Loading](#module-loading)
- [Functional Examples](#functional-examples)
- [Example: bfa](#example-bfa)
- [Example: lpfc](#example-lpfc)
- [Compatible Modules](#compatible-modules)
- [Kernel Symbols](#kernel-symbols)
- [Module Symbols](#module-symbols)
- [Methodology](#methodology)
- [Dependencies](#dependencies)
- [Weak Modules](#weak-modules)
- [Incompatible Example](#incompatible-example)
- [Caveats](#caveats)
- [References](#references)
## Overview
The Red Hat oriented Linux kernel architecture has a method for 3rd party entities to provide a kernel module for an entire family of kernel releases, based on the fundamental understanding the kernel's entry tables and module interface does not change within that family. This document goes over the basic design behind the solution.
The use of this methodology is popular amongst 3rd party vendors who provide a pre-compiled kernel module for their hardware and allow that same binary module to work for a number of compatible kernels. Shipping a new binary module for each and every Red Hat kernel release is therefore not required, reducing the complexity of producing the module and it's runtime maintenance on a server.
In common usage, these types of modules are delivered in packages named `kmod-<foo>`, where `foo>` is the name of the existing stock kernel module as shipped by the distribution. The overall compatibility is referred to as **kABI** or _Kernel Application Binary Interface_.
## Module Loading
Key to understanding the method is how the kernel will look for the modules to load. It varies vendor by vendor depending on the distribution, but generally speaking the modules are looked for in this order:
- `/lib/modules/(kernel-version)/updates`
- manually controlled area for use by sysadmins to insert a module by hand and override everything
- `/lib/modules/(kernel-version)/extra`
- override everything shipped with the kernel and weak-updates (see below)
- `/lib/modules/(kernel-version)/*`
- stock kernel modules (usually in a subdirectory kernel) and other named directories; a vendor may choose to have a top-level directory here, such as the EMC PowerPath software using `/lib/modules/(kernel-version)/powerpath` as it's standard location
- `/lib/modules/(kernel-version)/weak-updates`
- compatible kernel modules for this kernel, but were actually compiled against another similar kernel in the family
The concept named _weak-updates_ works in tandem with the extra module location; typically the original module is installed in an 'extra' directory named where it was compiled, and a symlink exists in the weak-updates directory from another kernel.
## Functional Examples
Using a Red Hat Enterprise (RHEL) 7 system, we first note that only one kernel is installed:
```
# rpm -qa | grep ^kernel-3
kernel-3.10.0-327.49.1.el7.x86_64
```
In Red Hat's versioning scheme, this is read as two parts of a design:
- 3.10.0-327.\* - the suite or "family" of kernel releases
- .49.1.el7 - the specific patched release of this kernel within the family
This design indicates that any kernel module built for the 3.10.0-327.\* family of kernels _should be_ compatible with any specific kernel in the family; but as it's not possible to 100% guarantee this ahead of time, safety checks exist (more on this below). On this server, we have a kmod kernel module that is replacing one of the stock ones.
### Example: bfa
This package `kmod-bfa` was obtained from a 3rd party provider for the Brocade fiber channel adapters.
```
/lib/modules/3.10.0-327.el7.x86_64/extra/bfa:
-rw-r--r--. 1 root root 23431886 Apr 22 2016 bfa.ko
/lib/modules/3.10.0-327.49.1.el7.x86_64/weak-updates/bfa:
lrwxrwxrwx. 1 root root 51 Feb 14 12:54 bfa.ko -> /lib/modules/3.10.0-327.el7.x86_64/extra/bfa/bfa.ko
```
Notice that the real file is in a directory for a kernel that is not installed; it's located in the "base" kernel for the family (the first one released in the family, in this case RHEL 7.2) `extra/` directory, and from our running kernel has a symlink from it's `weak-updates/` directory back to the module. This module is compatible for weak-updates; it was compiled against kernel 3.10.0-327 but functionally works with kernel 3.10.0-327.49.1 as is, no modifications needed.
### Example: lpfc
This package `kmod-lpfc` is provided in the main RHEL7 software repository by Red Hat, providing newer upstream code for Emulex fiber channel adapters.
```
/lib/modules/3.10.0-327.el7.x86_64/extra/lpfc:
-rw-r--r--. 1 root root 1180268 Sep 5 02:51 lpfc.ko
/lib/modules/3.10.0-327.49.1.el7.x86_64/weak-updates/lpfc:
lrwxrwxrwx. 1 root root 53 Feb 16 15:09 lpfc.ko -> /lib/modules/3.10.0-327.el7.x86_64/extra/lpfc/lpfc.ko
```
The design is exactly that of the previous example; this module is compatible for weak-updates; it was compiled against kernel 3.10.0-327 but functionally works with kernel 3.10.0-327.49.1 as is, no modifications needed.
## Compatible Modules
Ensuring that a module compiled for one version of a kernel is compatible with another kernel is key to the system working correctly; the topic deals a great deal with compilers, assemblers and linkers which provide the needed data to compare for compatibility. When a binary is compiled it has a _symbol table_ which basically indicates the structural location of all usable functions; this is both the kernel itself, and any modules trying to load themselves into that kernel.
The location address of all kernel functions a module expects to use are embedded in itself, as well as what it exports for others (imagine a module using a module) - the process is at it's simplest asking the target kernel if the map the module knows about has changed or not. If nothing has changed, it's compatible so long as some sort of internal change has not happened that is not visible to the outside world. This is the **kABI** in effect, the module is kernel ABI compatible between several compiled kernels.
### Kernel Symbols
The kernel(s) ship with a pre-exported symbol table stored in the /boot directory next to the kernel:
```
# ls -l /boot/symvers-3.10.0-327.49.1.el7.x86_64.gz
-rw-r--r--. 1 root root 252731 Jan 25 11:37 /boot/symvers-3.10.0-327.49.1.el7.x86_64.gz
# zgrep blk_queue_init_tags /boot/symvers-3.10.0-327.49.1.el7.x86_64.gz
0x00a006aa blk_queue_init_tags vmlinux EXPORT_SYMBOL
```
The output above shows the function `blk_queue_init_tags` is exported for all to use (`EXPORT_SYMBOL`) by the binary `vmlinux` (the kernel) with address `0x00a006aa` in the stack. This is a general function being used for example purposes herein, there are many more in use.
Due to specifically how the kernel operates, the shipped `vmlinuz` file (a compressed, stripped copy of `vmlinux`) typically does not contain the symbols; hence, they are extracted while the kernel package is being compiled and packaged and saved as a separate file for use in userspace. The `symvers` file contains all the symbols of every module as well as just the main kernel itself, making it quite a large set of data. If a symbol is exported by a module the module's name will be located where `vmlinux` is shown above.
Also note that some exports are for GPL compliant module use only; they have `EXPORT_SYMBOL_GPL` type and can only be used by GPL compliant modules.
### Module Symbols
A module is nothing more fancy than a standard library (shared object) designed specifically to work with the kernel. As such, all the normal commands to deal with symbol tables can be used like so with GNU `nm`:
```
# nm /lib/modules/3.10.0-327.el7.x86_64/extra/bfa/bfa.ko | grep blk_queue_init_tags
U blk_queue_init_tags
# nm /lib/modules/3.10.0-327.el7.x86_64/extra/lpfc/lpfc.ko | grep blk_queue_init_tags
U blk_queue_init_tags
```
You'll notice that this information is not super useful as shown; how a binary is assembled is more complex and requires a bit of work to get the data required in a format which makes sense. The `modprobe` tool with a bit of `sed` can be used to reassemble the data in a way that makes more sense for the task at hand, namely comparisons of addresses to names:
```
# modprobe --dump-modversions /lib/modules/3.10.0-327.el7.x86_64/extra/bfa/bfa.ko \
| sed -r -e 's:^(0x[0]*[0-9a-f]{8}\t.*):\1:' | grep blk_queue_init_tags
0x00a006aa blk_queue_init_tags
# modprobe --dump-modversions /lib/modules/3.10.0-327.el7.x86_64/extra/lpfc/lpfc.ko \
| sed -r -e 's:^(0x[0]*[0-9a-f]{8}\t.*):\1:' | grep blk_queue_init_tags
0x00a006aa blk_queue_init_tags
```
This shows us the kernel is exporting the function `blk_queue_init_tags` at `0x00a006aa` and the _weak module_ (compiled for another kernel) is expecting to find this same function at address `0x00a006aa` - this is a compatible function entry point, nothing has changed. From here, all that's left is to ensure each and every function the module uses or exports undergoes the same scrutiny for kABI compatibility.
## Methodology
There are several steps to ensuring a weak-updates kernel module is integrated well with the system and is compatible with a given target kernel. Each kernel is checked on it's own, so it is possible to have one kernel in a family using the kernel module (a symlink exists from it to the older file), or to not be using it (no symlink exists).
### Dependencies
The dependencies first must be taken care of; in the chance the module being inserted as a _weak-update_ is used by another module, the system needs to know about the symbols in the weak-update version as they may have changed (therefore causing a cascaded incompatibility by accident).
The entity shipping the module creates a file in `/etc/depmod.d/` with the override, like so:
```
# cat /etc/depmod.d/bfa.conf
override bfa 3.10.0-* weak-updates/bfa
# cat /etc/depmod.d/lpfc.conf
override lpfc 3.10.0-327.* weak-updates/lpfc
```
This is telling the system to use the `weak-updates/bfa` module version for all kernels in the 3.10.0-\* suite (which is all of RHEL 7 in this example) if it is found for bfa, but for `lpfc` the wildcard is more refined to only work with 3.10.0-327.\* kernels as an alternate example.
The entity shipping the module then runs this command after the module has been added (via RPM post-install, etc.); in this example, the module was compiled for 3.10.0-327 so the `depmod` command is using that version to update the symbols:
```
# depmod -aeF "/boot/System.map-3.10.0-327.el7.x86_64" "3.10.0-327.el7.x86_64"
```
As might be inferred from the above, this updated the `/boot/System.map-3.10.0-327.el7.x86_64` file with all symbols from the new file in the `extra/` directory.
### Weak Modules
The second step is to now create all the compatibility symlinks in the `weak-updates/` subdirectories of all kernels installed on the system which are in fact 100% compatible with this new module. From the outside, it's all built into a script that can just be used by the entity shipping the module (again, in their package post-install):
```
# weak-modules --add-modules /lib/modules/3.10.0-327.el7.x86_64/extra/bfa/bfa.ko
...or:
# weak-modules --add-modules /lib/modules/3.10.0-327.el7.x86_64/extra/lpfc/lpfc.ko
```
> The `weak-updates` script will also rebuild the _initramfs_ files in `/boot` for all the kernels found, inserting the new module setup for stage 1 boot. When installing a kmod package this is the perceived lag, after the RPM has placed the bits down it's updating all initramfs files for kernels it adjusted.
The process inside the script can be broken down into these basic steps:
1. Take the kernel symbols file and massage it into a format that works with `diff` and `join` later (loops for every kernel found):
```
# krel=$(uname -r)
# zcat /boot/symvers-$krel.gz \
| sed -r -ne 's:^(0x[0]*[0-9a-f]{8}\t[0-9a-zA-Z_.]+)\t.*:\1:p' \
> symvers-$krel
```
2. If required (the kernel may not have any), extract and prepare the same information from any `extra/` modules in the target kernel (this will loop for every installed kernel). Notice that we're only extracting data of the installed kernels and if they have something in `extra/` - this file may be zero bytes if none are there:
```
# krel=$(uname -r)
# find /lib/modules/$krel/extra -name '*.ko' \
| xargs nm \
| sed -nre 's:^[0]*([0-9a-f]{8}) A __crc_(.*):0x\1 \2:p' \
> addon-symvers-$krel
```
3. Do the same action as the above, but **specifically for the kernel the module was built against** known as `vermagic` within the module's data:
```
# modinfo -F vermagic bfa lpfc
3.10.0-327.el7.x86_64 SMP mod_unload modversions
3.10.0-327.el7.x86_64 SMP mod_unload modversions
# module_krel=3.10.0-327.el7.x86_64
# find /lib/modules/$module_krel/extra -name '*.ko' \
| xargs nm \
| sed -nre 's:^[0]*([0-9a-f]{8}) A __crc_(.*):0x\1 \2:p' \
> extra-symvers-$module_krel
```
4. Take the data from the above steps and simply combine and sort it for use:
```
# sort -u symvers-$krel \
extra-symvers-$module_krel \
addon-symvers-$krel \
> all-symvers-$krel-$module_krel
```
5. Now extract the data from the new module physically being added to the system and extract it's symbols as well:
```
# module="/lib/modules/3.10.0-327.el7.x86_64/extra/bfa/bfa.ko"
...or:
# module="/lib/modules/3.10.0-327.el7.x86_64/extra/lpfc/lpfc.ko"
# /sbin/modprobe --dump-modversions "$module" \
| sed -r -e 's:^(0x[0]*[0-9a-f]{8}\t.*):\1:' \
| sort -u \
> modvers
```
6. Last, use the `join` command in reverse mode (think `grep -v`) to tell us if any lines from all the known symbols provided does **not** match the symbols the new module is expecting:
```
join -j 1 -v 2 all-symvers-$krel-$module_krel modvers
```
This set of steps tells us if the incoming module being added is identical in symbols to what's actually running an expected on the system; any output from the last step is indicating that something was found that differs in either address or availability and this module is not compatible. No output means it's fully compatible and cane be symlinked to the target kernel safely.
## Incompatible Example
Using the above methodology, we can examine a different kernel module which is incompatible. This specific version of `kmod-bna` has 4 occurrences of incompatible function addresses with a specific (older) kernel that has been installed. Each of the items above is covered in order:
```
The setup:
# rpm -qa | egrep "^(kernel-3|kmod-bna)"
kernel-3.10.0-229.20.1.el7.x86_64
kernel-3.10.0-327.49.1.el7.x86_64
kmod-bna-3.2.7.0-0.el7.x86_64
Step 1:
# krel=3.10.0-229.20.1.el7.x86_64
# zcat /boot/symvers-$krel.gz \
> | sed -r -ne 's:^(0x[0]*[0-9a-f]{8}\t[0-9a-zA-Z_.]+)\t.*:\1:p' \
> > symvers-$krel
Step 2:
# find /lib/modules/$krel/extra -name '*.ko' \
> | xargs nm \
> | sed -nre 's:^[0]*([0-9a-f]{8}) A __crc_(.*):0x\1 \2:p' \
> > addon-symvers-$krel
Step 3:
# modinfo -F vermagic bna | cut -f1 -d' '
3.10.0-327.el7.x86_64
# module_krel=3.10.0-327.el7.x86_64
# find /lib/modules/$module_krel/extra -name '*.ko' \
> | xargs nm \
> | sed -nre 's:^[0]*([0-9a-f]{8}) A __crc_(.*):0x\1 \2:p' \
> > extra-symvers-$module_krel
Step 4:
# sort -u symvers-$krel \
> extra-symvers-$module_krel \
> addon-symvers-$krel \
> > all-symvers-$krel-$module_krel
Step 5:
# module="/lib/modules/3.10.0-327.el7.x86_64/extra/bna/bna.ko"
# /sbin/modprobe --dump-modversions "$module" \
> | sed -r -e 's:^(0x[0]*[0-9a-f]{8}\t.*):\1:' \
> | sort -u \
> > modvers
Step 6:
# join -j 1 -v 2 all-symvers-$krel-$module_krel modvers
0x7efd609f __netif_napi_add
0x905307be napi_complete_done
0xd93737a0 napi_disable
0xe1d1af76 __dev_kfree_skb_any
```
The methodology is showing us there are 4 addresses which do not match up between the older kernel and this newer module, making them incompatible for use together.
## Caveats
A compatible kernel module as determined by the _weak-updates_ methodology is an observation from the symbol addresses from the outside only; there is no way to functionally test the module works at runtime transparently, only that it can be inserted to the target kernel without error. It is entirely possible for a coding error internally to occur and the module not work; the kernel engineers patching a given kernel may have changed something which causes breakage.
Testing a newly updated kernel against any existing weak module must performed to ensure all functionality is retained.
## References
- <https://lists.fedoraproject.org/pipermail/devel/2006-August/088293.html>

333
md/linux_partitioning.md Normal file
View file

@ -0,0 +1,333 @@
# Linux Partitioning
## Contents
- [Overview](#overview)
- [MBR vs. GPT](#mbr-vs-gpt)
- [MBR Extended/Logical Partitions](#mbr-extendedlogical-partitions)
- [Partition Alignment](#partition-alignment)
- [Common Tools](#common-tools)
- [fdisk](#fdisk)
- [parted](#parted)
- [gdisk](#gdisk)
- [partx](#partx)
- [Usage Comparison](#usage-comparison)
- [Resizing Partitions](#resizing-partitions)
- [Advanced Examples](#advanced-examples)
- [Expanding MBR Primary](#expanding-mbr-primary)
- [Adding MBR Logical](#adding-mbr-logical)
- [Zapping Devices](#zapping-devices)
- [References](#references)
- [Citations](#citations)
## Overview
After understanding the design of how [x86 Storage works](linux_x86_storage.md) the next logical step is learning and using the various utilities to manipulate the MBR/GPT and partitions themselves. We'll focus on the most common utilities herein and common tasks and how they differ and their use with both partition types.
## MBR vs. GPT
This topic is covered in detail in the [Linux x86 Storage](linux_x86_storage.md) article; a quick recap of the basics:
| **Purpose** | **MBR** | **GPT** |
| -------------- | ----------------------------------------------------- | ------- |
| Max Partitions | 4 Primary or 3 Primary, 1 Extended, Unlimited Logical | 128 |
| Max Size | 2 TiB | 8 ZiB |
Extrapolating from this table we can then make a set of rules to live by:
- MBR format is limited to 2 TiB
- GPT format is limited to 8 ZiB
- Use GPT format if your storage is greater than 2 TiB
- Use GPT if your storage could grow larger than 2 TiB
- If a 4th MBR partition is marked Primary, you cannot use Extended
- If 3 MBR Primary partitions exist, make the 4th one Extended rest of device
- Logical MBR partitions 5+ live inside the 4th Extended partition
- Adding a new Logical partition requires growing the Extended partition first
## MBR Extended/Logical Partitions
Having only 4 primary partitions is a limit of the original MBR design - as such, an extension was invented called the [extended partition](http://en.wikipedia.org/wiki/Extended_partition) with a very wide open design. The 4th primary partition is created as a type _extended_ which points to the first Extended Boot Record (EBR) of the first logical partition within. The extended partition is normally created with the rest of the disk as it's size, it will contain logical partitions inside.
Each EBR contains a pointer to the next EBR (along with it's logical partition info) which allows chaining EBRs together. The number of logical partitions within an extended partition is limited only by the amount of available disk space in the extended partition; all of this work is handled by the various tools and does not need to be manually manipulated by the end user.
## Partition Alignment
With the traditional design of a 512-byte sector, starting the first partition at LBA 63 poses no problem in alignment of the physical device (sometimes called the _radial geometry_) with the logical use by the CPU. Modern devices are starting to use 4096-byte (4k) sector sizes [(1)][c1] - and in the case of SSD possibly 8192-bytes (8k), sometimes referred to as _pages_ to match the CPU terminology. Externally they may emulate 512-byte sectors for compatibility (called **512e** mode) but internally these devices are working with 4096.[(2)][c2]</sup>
Starting a partition at LBA 63 with 4k pages it problematic -- mathematically this is one 512-byte sector short of a natural 4096 boundary of the physical geometry. If history had worked out better, using LBA 64 would have worked out mathematically:
```
(63*512)/4096 = 7.8750
(63*512)/8192 = 3.9375
(64*512)/4096 = 8.0000
(64*512)/8192 = 4.0000
```
To overcome this issue, in modern use the first partition is started with at LBA 2048 - with a 512-byte sector this is 1 MiB, or 256 pages of a 4096-byte sector. This conservative approach allows for future changes that are not foreseen today, allowing enough space and performing meticulous boundary alignment on the classic power of 2^8. SSD drives use multiples of 128 KiB, 256 KiB or 512 KiB depending on device, again creating a natural alignment mechanism.[(3)][c3]
On a modern Linux distribution the userspace tools described below understand these alignment needs and default to using 2048 sector offset when creating the first partition on a drive.
## Common Tools
### fdisk
The fdisk utility is the classic partitioning tool. It's usage is based around a menu driven text interface; all changes performed are held into memory until a command is issued to write them out to disk; this technique allows one to cancel/exit if desired without changes, making it an enticing tool to use. The fdisk binary itself is part of the `util-linux` (or `util-linux-ng`) package of tools that also includes utilities like `fsck`, `mount` and `umount`.
One of the most important things to note about using fdisk: the version shipped on most enterprise class distributions (RHEL/CentOS for instance) tends to be an older but stable version. The downside is that these older stable releases only support the MBR (aka "dos" or "msdos") style and partition design, limiting your disk to 2 TiB. The most recent fdisk release are beginning to support GPT style, but as they are not yet packaged as for these distros means another tool must be used.
> Use the `-c` and `-u` options to fdisk to disable DOS compatibility mode (C/H/S) and use Sector mode. These options can also be toggled while in fdisk with the `c` and `u` commands.
### parted
The parted utility works the opposite of fdisk; as commands are issued the changes are made, whether it's used in pure commandline-only mode or when using the menu driven interface. This concept tends to make the usage of parted a bit scary to the beginning tech, and rightly so - once you do it, it's done. No backing out of a mistake (easily) like with fdisk.
The draw to parted tends to come in two parts: it's easily commandline scriptable and it fully supports GPT style disks (as well as MBR). This makes it the current _de-facto_ utility for dealing with storage larger than 2 TiB and GPT partitions (required for UEFI). The parted package also includes the `partprobe` tool that many like.
### gdisk
The `gdisk` utility (sometimes called _gptdisk_ or _GPT fdisk_) is a newer tool designed to combine the two worlds of fdisk and parted; it provides a bit of commandline and a bit of menu driven interface. It allows all the GPT functionality but with the capability of staging in memory first before writing to disk, allowing for an exit without write as needed. The gdisk package includes complementary targeted tools such as `cgdisk`, `sgdisk` and `fixparts` which are designed for GPT/MBR manipulation.
One of the largest draws to gdisk is it's ability to repair (or attempt to repair) corrupted or broken MBR and GPT partition style types. It can extract and back up partition tables, load them from backups, repair the primary GPT from the secondary copy, convert GPT to MBR and all sorts of advanced features. The use of gdisk in certain situations can achieve results that fdisk and parted cannot.
At this time the gdisk package is hosted via the EPEL project for RHEL/CentOS distributions and is not present in the standard vendor repositories. Conversely the Ubuntu LTS repositories do contain it, however they may have an older version (0.8.1 for 12.04 LTS). If working on GPT disks be sure and use a stable relatively bug-free version.
### partx
The partx utility is useful for triggering the kernel subsystems to add/remove device nodes based on the partition map of a block device. It's akin to the `partprobe` tool however works a bit differently; in general it's a better choice than partprobe due to it's design and integration with the kernel. The most typical usage is to add new partition device nodes after new partitions were created, or delete in the reverse scenario. Normally these functions are handled by fdisk/parted/gdisk but in some cases they are not.
For example - if a device has partitions and devices nodes such as /dev/xvdb1, /dev/xvdb2, etc. and you utilize the `zap` feature of gdisk to zero out a disk, this does not trigger cleanup of the device nodes. Running `partx -d /dev/xvdb` will trigger a kernel level cleanup and remove them at runtime - likewise, it's common using `fdisk /dev/sda` to make a new partition results in an ioctl error when saving. You can use `partx -a /dev/sda` to trigger a kernel level refresh to create your new /dev/sdaX device node entry.
## Usage Comparison
Utilizing fdisk and gdisk menu driven interfaces are nearly identical; while parted has a menu interface, the commands typed are the same as on the commandline scripted mode. A quick comparison of the most common operations being performed:
| **Purpose** | **fdisk** | **parted** | **gdisk** |
| -------------------------- | --------------------------------------------- | ------------------------------------------------ | ---------------------------------------------- |
| **List Partitions** | fdisk -cul /dev/xvdb | parted /dev/xvdb unit s print | gdisk -l /dev/xvdb |
| **Create MBR or GPT** | fdisk -cu /dev/xvdb; o, w | parted -s -- /dev/xvdb mkpart table gpt | gdisk /dev/xvdb; o, w |
| **Create First Partition** | fdisk -cu /dev/xvdb; n, p, 1, 2048, _size_, w | parted -s -- /dev/xvdb mkpart primary 2048s 100% | gdisk /dev/xvdb; n, 1, 2048, _size_, _type_, w |
| **Set Partition to LVM** | fdisk -cu /dev/xvdb; t, 1, 8e, w | parted -s -- /dev/xvdb set 1 lvm on | gdisk /dev/xvdb; t, 1, 8e00, w |
| **Delete Partition** | fdisk -cu /dev/xvdb; d, 1, w | parted -s -- /dev/xvdb rm 1 | gdisk /dev/xvdb; d, 1, w |
## Resizing Partitions
Resizing partitions is typically encountered when a disk is grown (expanded) or shrunk (reduced); remember that a higher level infrastructure sits on top of the partitions themselves in almost all cases. Given that, remember to take precautions such as:
- If reducing, unmount the filesystem or make it inactive as required for safety
- If reducing, resize the filesystem (ext3, ext4, etc.) **before** the container/partition
- If reducing, reduce the container (LVM, etc.) **before** the partition
- If expanding, expand the container (LVM, etc.) **after** the partition
- If expanding, resize the filesystem (ext3, ext4, XFS, etc.) **after** the container/partition
- Not all filesystem types can be reduced or expanded\! XFS for example cannot be reduced
In practice using `parted` is the best tool for the job; however, remember that parted is a one shot tool - **changes are immediate**. Use `gdisk` for GPT or `fdisk` for MBR disks if you need to stage the work and double check it before executing. Do not use these tools to resize the filesystem itself\! For example the parted tool has a _resize_ command that works on some filesystem types and not others, and no way to ignore the filesystem.
## Advanced Examples
### Expanding MBR Primary
> Warning: all data could be lost if this is done incorrectly\! Pay very close attention to your start/end sectors and math.
- **Scenario**: MBR, 3 Primary partitions
- **Task**: Increase partition 3 size +10G
Unmount the partition and record the layout; for this example only I'll show the disk in GiB to help in understanding the sector math first; we'll unmount it and fsck it to ensure the filesystem is OK before starting.
```
# df -h | grep xvdb3
/dev/xvdb3 9.9G 151M 9.2G 2% /mnt
# umount /mnt
# parted -s -- /dev/xvdb unit GiB print
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdb: 100GiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 0.00GiB 10.0GiB 10.0GiB primary
2 10.0GiB 20.0GiB 10.0GiB primary
3 20.0GiB 30.0GiB 10.0GiB primary ext4
# fsck -fC /dev/xvdb3
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/xvdb3: 11/655360 files (0.0% non-contiguous), 79663/2621440 blocks
```
Now let's do the work - first, the geometry needs to be recorded **in sector mode** as we must precisely recreate the starting sector of the partition:
```
# parted -s -- /dev/xvdb unit s print
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdb: 209715200s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 2048s 20973568s 20971521s primary
2 20973569s 41945089s 20971521s primary
3 41945090s 62916610s 20971521s primary ext4
```
We use the built-in functionality of parted to simple change the starting and ending sectors. We will add 20971520 sectors (10 GiB @ 512-byte sector size) to the end of partition 3. We then tell parted to use the same starting sector (41945090s) and new ending sector (83888130s) we just computed.
Note that the numerical value **ends in s** to denote sectors\! Do not forget your trailing s. Lastly, we resize (grow) the ext4 filesystem on top of it to the new size.
```
# echo "62916610 + 20971520" | bc -l
83888130
# parted -s -- /dev/xvdb rm 3
# parted -s -- /dev/xvdb mkpart primary 41945090s 83888130s
# resize2fs /dev/xvdb3
# mount /dev/xvdb3 /mnt
# parted /dev/xvdb unit s print
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdb: 209715200s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 2048s 20973568s 20971521s primary
2 20973569s 41945089s 20971521s primary
3 41945090s 83888130s 41943041s primary ext4
# df -h | grep mnt
/dev/xvdb3 20G 156M 19G 1% /mnt
```
### Adding MBR Logical
> Warning: all data could be lost if this is done incorrectly\! Pay very close attention to your start/end sectors and math.
- **Scenario**: MBR, 3 Primary partitions, 1 Extended partition, 1 Logical partition
- **Task**: Increase Extended partition, add new Logical partition
This builds upon the basic expansion of a primary partition concept, however the `parted resize` command is our saving grace; if it were done 100% manually (say, with fdisk) the process would go like this:
1. Record geometry
2. Delete Logical partition 1
3. Delete Extended partition
4. Recreate Extended partition larger
5. Recreate Logical partition 1
6. Add new Logical partition 2
Using parted however allows us to magically resize the Extended partition without having to perform all of the above steps. Once again, pay attention to your exact geometry and math\!
```
# parted /dev/xvdb unit s print
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdb: 209715200s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 2048s 20973568s 20971521s primary
2 20973569s 41945089s 20971521s primary
3 41945090s 62916610s 20971521s primary ext4
4 62916611s 83888131s 20971521s extended
5 62918659s 83888131s 20969473s logical ext4
```
Notice the Start of 4 (Extended) and 5 (Logical) do not match - watch out for this, especially if you intended to resize 5 after growing 4. We will add 10G (20971520s) to the end of 4, then add a new 6 (Logical) using this new space starting after the existing 5. Note that I am adding 2048 to the ending sector of 5 to determine my start of 6 for optimal performance (in theory - this disk is misaligned starting at partition 2 already):
```
# echo "83888131 + 20971520" | bc -l
104859651
# parted -s -- /dev/xvdb resize 4 62916611s 104859651s
# parted /dev/xvdb unit s print
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdb: 209715200s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 2048s 20973568s 20971521s primary
2 20973569s 41945089s 20971521s primary
3 41945090s 62916610s 20971521s primary ext4
4 62916611s 104859651s 41943041s extended
5 62918659s 83888131s 20969473s logical ext4
# parted -s -- /dev/xvdb mkpart logical 83890179s 104859651s
# parted /dev/xvdb unit s print
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdb: 209715200s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 2048s 20973568s 20971521s primary
2 20973569s 41945089s 20971521s primary
3 41945090s 62916610s 20971521s primary ext4
4 62916611s 104859651s 41943041s extended
5 62918659s 83888131s 20969473s logical ext4
6 83890179s 104859651s 20969473s logical
```
### Zapping Devices
Zapping is the concept of completely wiping all traces of a MBR and/or GPT to result in clean device, as if it were brand new. All data is lost, use with caution\! Use `gdisk` for this need:
```
# gdisk /dev/xvdb
GPT fdisk (gdisk) version 0.8.10
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Command (? for help): x
Expert command (? for help): z
About to wipe out GPT on /dev/xvdb. Proceed? (Y/N): Y
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Blank out MBR? (Y/N): Y
```
## References
- <ftp://ftp.kernel.org/pub/linux/utils/util-linux/>
- <http://www.gnu.org/software/parted/manual/parted.html>
- <http://www.rodsbooks.com/gdisk/>
- <http://en.wikipedia.org/wiki/Extended_partition>
- <http://en.wikipedia.org/wiki/Disk_partitioning>
- <http://people.redhat.com/msnitzer/docs/io-limits.txt>
## Citations
1. <http://www.anandtech.com/show/2888>
2. <https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/ch-iolimits.html>
3. <http://blog.nuclex-games.com/2009/12/aligning-an-ssd-on-linux/>
[c1]: http://www.anandtech.com/show/2888
[c2]: https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/ch-iolimits.html
[c3]: http://blog.nuclex-games.com/2009/12/aligning-an-ssd-on-linux/

400
md/linux_x86_storage.md Normal file
View file

@ -0,0 +1,400 @@
# Linux x86 Storage
## Contents
- [Overview](#overview)
- [BIOS vs. UEFI](#bios-vs-uefi)
- [4096 Everywhere](#4096-everywhere)
- [Drive Geometry](#drive-geometry)
- [CHS Addressing](#chs-addressing)
- [ZBR Tracks](#zbr-tracks)
- [LBA Addressing](#lba-addressing)
- [Boot Sector](#boot-sector)
- [MBR Format](#mbr-format)
- [Bootstrap Code Area](#bootstrap-code-area)
- [Partition Table Entry](#partition-table-entry)
- [GPT Format](#gpt-format)
- [Legacy MBR Sector](#legacy-mbr-sector)
- [Partition Table Entry](#partition-table-entry-1)
- [OS Compatibility](#os-compatibility)
- [GRUB Bootstrap](#grub-bootstrap)
- [LVM Boot Volumes](#lvm-boot-volumes)
- [Scanning Devices](#scanning-devices)
- [Scan for new Devices](#scan-for-new-devices)
- [Rescanning and Deleting Devices](#rescanning-and-deleting-devices)
- [The udev Subsystem](#the-udev-subsystem)
- [HBA Blacklisting](#hba-blacklisting)
- [References](#references)
- [Citations](#citations)
## Overview
This article focuses on a number of core concepts combined to provide an overview of the process without going down the rabbit hole too far in any one subject area. In general, the areas of focus:
- [IA-32/x86/x86-64 Architecture CPU](http://en.wikipedia.org/wiki/X86)
- [Basic Input/Output System<](http://en.wikipedia.org/wiki/BIOS) (BIOS)
- [Unified Extensible Firmware Interface](http://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface) (UEFI)
- [Master Boot Record boot sector](http://en.wikipedia.org/wiki/Master_boot_record) (MBR)
- [GUI Partition Table](http://en.wikipedia.org/wiki/GUID_Partition_Table) (GPT)
- [GNU/Linux Operating System](http://en.wikipedia.org/wiki/Linux)
- [GNU GRand Unified Bootloader](http://en.wikipedia.org/wiki/GRUB) (GRUB)
The use of partitioned mass storage devices (such as fixed disks) is our target medium; discussing non-partitioned devices (such as floppies) is not considered as part of this article. This article also focuses on traditional rotating magnetic media as Storage - newer devices such as SSD drives are still in thrall to the older design mechanisms, so understanding the basic mechanics is key.
### BIOS vs. UEFI
BIOS is still the most widely deployed firmware - modern systems are embracing UEFI but until it becomes the _de-facto_ standard we must adhere to the design and limitations of the BIOS firmware infrastructure in order to address storage and boot operating systems. Most of this article is discussing traditional BIOS design.
BIOS is only capable of reading the boot sector of the disk and executing it's code, while recognizing the MBR partition format itself. BIOS has no concept of filesystem types and only executes the first 440 bytes before relinquishing control. This leads to the use of secondary (_multi-staged_ or _chained_) boot loaders such as GRUB.
UEFI by contrast does not execute boot sector code; instead there exists an **EFI System Partition** (ESP) in which the required firmware files are loaded (called an _UEFI Application_); this partition is typically a FAT32 formatted 512 MiB space and supports multiple UEFI Applications at the same time. The firmware itself has a boot menu embedded that defines the disks and partitions to be launched for the applications, effectively acting like Stage 1 of GRUB.
BIOS-GPT is a hybrid; it allows BIOS to load boot code of the protected MBR on a GPT partition and execute it. Typically this requires a [BIOS boot partition](http://en.wikipedia.org/wiki/BIOS_Boot_partition) around 1 MiB in size as the 440-byte bootstrap area is not large enough and there exist no extra sectors where the boot loader is typically located in MBR.
### 4096 Everywhere
The traditional storage sector size is 512-bytes; with the introduction of the [Advanced Format](http://en.wikipedia.org/wiki/Advanced_Format) 4096-byte sector size this brings to light a question - why is everything based on 4kb? The Linux kernel memory page size, the largest Linux filesystem block size that can be used, the disk sector size are all a max of 4kb - this is all based on the [classic x86 MMU design](http://en.wikipedia.org/wiki/Memory_management_unit#IA-32_.2F_x86).
The classic [x86 MMU architecture](http://wiki.osdev.org/Paging) contains two [page tables](http://en.wikipedia.org/wiki/Page_table) of 1024 4-bytes entries making each one 4kb in size. One table is called the _paging directory_ and the other one the _paging table_, they work together to provide virtual-to-physical access to the memory within the system. If you're doing the math that limits us to 4GiB (32bits) of memory - hence the introduction of [PAE](http://en.wikipedia.org/wiki/Physical_Address_Extension) to allow up to 64 GiB (36 bits) to be addressed. The [x86\_64 platform](http://en.wikipedia.org/wiki/X86-64#Virtual_address_space_details) further increases this _currently_ to 128 TiB (48 bits) per current spec, but a theoretical maximum of 256 TiB (64bits) of virtual address space could happen.
Modern CPUs are starting to offer larger page tables ("huge pages") but are not the norm - as such, the alignment of the x86 MMU 4kb page to other parts of the system is why we don't see 8096-byte sectors or 16kb filesystem blocks possible at this time with most Linux infrastructure even though the code itself supports it (such as XFS block size \> 4096).
The Linux kernel has various extensions such as [transparent huge pages](https://lwn.net/Articles/188056/) to perform virtual mappings, however at the end of the day performance is limited to the specific x86 CPU that is being used at the time. As such the 4096-byte sector size, block size and memory page size are our daily use scenarios.
## Drive Geometry
The root of everything is the radial geometry of the physical drive itself, how it's segregated and most importantly how do we "find" specific data at any given moment on the magnetic medium.
### CHS Addressing
At the core of our x86 history is the [cylinder-head-sector](http://en.wikipedia.org/wiki/Cylinder-head-sector) addressing scheme that was invented to determine where on a platter the needed information lives on a storage device. This design is what has imposed the 2 TiB limit for storage using the Master Boot Record (MBR) format as the spec has limits (more on that below). CHS is expressed as a _tuple_ - 0/0/1, 12/9/17, etc. - to refer to a finite physical location of data.
- **Platter**
- The platter is the thin piece of magnetic storage medium; it has two sides that can be used. All the platters are stacked on top of each other with heads in between
- **Head**
- A head is the little "arm" that moves over the platters to read the magnetic information, so for each platter you have 2 heads (one on each side). The maximum is 256 however an old bug limits this to 255 in use
- **Track**
- A track is one of the concentric rings on a platter; they start with 0 at the outer edge and increase numerically inwards
- **Cylinder**
- A cylinder is the set of "stacked" tracks of all platters in the same physical location; they start at 0 along with the tracks. The address of a cylinder is always the same as an individual track since it's just a stack of them
- **Sector**
- A sector is the segregated block within a track, maximum 63 sectors/track with 512-bytes per sector. **Sectors start counting at 1, not 0**
Where this all comes into play is the concept that BIOS will look specifically at location 0/0/1 (first cylinder/track, first head, first sector) to load the initial [machine language](http://en.wikipedia.org/wiki/Machine_code) boot code. This creates an absolute physical location for every storage device to boot and has carried forth into the more modern LBA addressing mechanism.
Note that the maximums for tracks, cylinders and sectors evolved over time and ended with the ATA-5 specification of 16383/16/63 requiring a full 24-bit value.
```
16383 cylinders * 16 heads * 63 sectors = 1032*254*63 = 16514064 sectors / 2 (512-bytes/sector) = 8257032 kb = ~ 8Gib
```
The [INT 13H](http://en.wikipedia.org/wiki/INT_13) BIOS EXT extension is what permits us to read beyond the original CHS limit; INT-13H CHS is 24-bits and ATA spec is 28-bits, BIOS routines exist to translate between the two for full compatibility. The ATA 16:4:8 bit scheme to 10:8:6 bit scheme used by INT 13H routines are what allow mapping up to 8 GiB.
#### ZBR Tracks
Initially the design was that all tracks contained the same number of sectors (MFM and RLL drives) - this was updated with a newer technique called [zone bit recording](http://en.wikipedia.org/wiki/Zone_bit_recording) in ATA/IDE drives that allowed more sectors on the outer (larger) tracks and fewer moving inwards. This technique, however, created a problem - the physical geometry of the drive no longer matched the CHS addressing.
Because data (such as a partition) needs to start/end on a track/cylinder boundary, this leaves surplus sectors at the end of the drive less than 1 cylinder in size since they almost never line up perfectly. This is why when making partitions in tools like fdisk or parted you will see unused sectors even though you specified using the whole drive - the tools are translating your request into cylinder boundaries and discarding any surplus sectors as unusable since they are not aligned.
### LBA Addressing
The limitations of the CHS design were quickly encountered; as such a more extensive format was introduced called [Logical Block Addressing](http://en.wikipedia.org/wiki/Logical_block_addressing). Now that CHS has been defined understanding LBA becomes easy and is best explained with a simple table.
| **LBA Value** | **CHS _Tuple_** |
| -------------- | --------------- |
| 0 | 0 / 0 / 1 |
| 62 | 0 / 0 / 63 |
| 1008 | 1 / 0 / 1 |
| 1070 | 1 / 0 / 63 |
| 16,514,063 | 16382 / 15 / 63 |
As exemplified, LBA addressing simply starts at 0 and increases by 1 for each CHS tuple. The original LBA was native 28-bit (see the CHS mapping above), the current ATA-6 spec is a 48-bit wide LBA allowing addressing up to 128 PiB of storage. As might be obvious there is a cutoff after 8 GiB of being able to translate CHS to LBA for backwards compatibility. Modern INT 13H extensions allow native LBA access thereby negating any need to use CHS style structures.
Our **CHS tuple 0/0/1 and LBA value 0 are aligned**, however - this is what we care about most for booting the system.
## Boot Sector
Now that we understand CHS and LBA addressing, let's look at what's going on once the BIOS reads the first 512-byte sector of the drive to get going. This breaks down into two formats - traditional [Master Boot Record](http://en.wikipedia.org/wiki/Master_boot_record) (MBR) format, and [GUID Partition Table](http://en.wikipedia.org/wiki/GUID_Partition_Table) (GPT) format. The Wikipedia pages on both are fantastic, I highly recommend reading both to gain a deeper understanding.
There are two kinds of basic boot sectors:
- [Master Boot Record](http://en.wikipedia.org/wiki/Master_boot_record) (MBR) is the first sector of the partitioned storage
- [Volume Boot Record](http://en.wikipedia.org/wiki/Volume_Boot_Record) (VBR) is the first sector of an individual partition
We are used to thinking of the boot sector as the MBR, but in fact there are two present in our x86 partitioned storage. GPT contains a 512-byte MBR protection mechanism for backwards compatibility. Essentially a MBR and VBR are the same thing, just located at different locations for different purposes. A non-partitioned device like a floppy disk uses only a VBR at the beginning, whereas a partitioned device typically uses a MBR (which may then load a VBR later).
### MBR Format
The MBR is at minimum the first 512-byte sector of the storage. There are two basic structures that are in use for our purposes as detailed in the tables; of most import are the bootstrap code area and partition table entries.
**Classic Generic MBR Structure**
| **Offset** | **Description** | **Size(bytes)** |
| ---------- | ------------------------ | --------------- |
| +0 | Bootstrap code area | 446 |
| +446 | PTE #1 | 16 |
| +462 | PTE #2 | 16 |
| +478 | PTE #3 | 16 |
| +494 | PTE #4 | 16 |
| +510 | Boot signature (55h AAh) | 2 |
**Modern Standard MBR Structure**
| **Offset** | **Description** | **Size(bytes)** |
| ---------- | ---------------------------- | --------------- |
| +0 | Bootstrap code area (part 1) | 218 |
| +218 | Disk timestamp | 6 |
| +224 | Bootstrap code area (part 2) | 216 |
| +440 | Disk signature | 6 |
| +446 | PTE #1 | 16 |
| +462 | PTE #2 | 16 |
| +478 | PTE #3 | 16 |
| +494 | PTE #4 | 16 |
| +510 | Boot signature | 2 |
The first data partition does not start until sector 63 (for historical reasons), leaving a 62 "MBR gap" present on the system. This gap of unused sectors is typically used for Stage 1.5 chained boot managers, low level device utilities and so forth.
#### Bootstrap Code Area
This area of the sector is pure [machine language code](http://en.wikipedia.org/wiki/Machine_code) run in real time mode; think of it like a BASIC program, it's read line by line and executes each one of those lines sequentially to manipulate CPU registers (more or less - it's complicated\!). This mechanism allows the CPU to execute arbitrary code without understanding anything about the higher level filesystem or storage design.
Notice that we have an extremely limited amount of space (440 bytes) - this is nowhere near enough room to run a fancy modern boot manager like [GNU GRUB](http://en.wikipedia.org/wiki/GRUB) full of graphics, features and whatnot. Hence we have the concept of _staged_ (or _chained_) boot managers. This area represents **Stage 1** of the bootloader process and serves to simply provide instructions on where to load the next bit of code from physically. More on that in the GRUB section.
#### Partition Table Entry
We come finally to the [Achilles' Heel](http://en.wikipedia.org/wiki/Achilles%27_heel) of the MBR design - partition table design and it's relation to the CHS addressing format. As each PTE is only 16-bytes we have a finite limit on what can be stored; extrapolating this is where are limit is created in how much disk can be addressed, leading to our 2 TiB limit of a MBR-based storage disk.
**16-byte PTE**
| **Length** | **Description** |
| ---------- | ------------------------------ |
| 1 | Status (active/inactive) |
| 3 | CHS address of partition start |
| 1 | Partition type |
| 3 | CHS address of partition end |
| 4 | LBA address of partition start |
| 4 | Total sectors in partition |
Given this design, at most 4-bytes (32-bits) can store the number of sectors in LBA mode and the limitations as discussed above. These are referred to as the Primary partitions of the disk and the above exemplifies why only 4 of them exist when using tools like fdisk and parted.
### GPT Format
The GUID Partition Table format was invented to solve the whole mess of CHS, MBR and 32-bit LBA limitations. It's actually part of the [Unified Extensible Firmware Interface](http://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface) (UEFI) designed to replace the aging [Basic Input/Output System](http://en.wikipedia.org/wiki/BIOS) (BIOS) design, however due to it's widespread use to utilize larger than 2 TiB of storage it's often considered it's own project.
| **Offset** | **Description** | **Size(bytes)** |
| ---------- | ----------------------------------------------------- | --------------- |
| | **LBA 0 (Legacy MBR)** | |
| +0 | Bootstrap code, Disk timestamp and signature | 446 |
| +446 | PTE #1 Type 0xEE (EFI GPT) | 16 |
| +462 | PTE #2 (unused) | 16 |
| +478 | PTE #3 (unused) | 16 |
| +494 | PTE #4 (unused) | 16 |
| +510 | Boot Signature (55h AAh) | 2 |
| | **LBA 1 (Primary GPT Header)** | |
| +512 | Definition of usable blocks on disk, PTEs, GUID, etc. | 512 |
| | **LBA 2-33 (Primary Partition Table Entries)** | |
| +1024 | 128x 128-byte PTEs | 16128 |
| | **LBA 34+ (Partitions)** | |
| +17408 | Actual partitions | n/a |
| | **LBA -33 to -2 (Secondary Partition Table Entries)** | |
| -1023 | 128x 128-byte PTEs | 16128 |
| | **LBA -1 (Secondary GPT Header)** | |
| -511 | Definition of usable blocks on disk, PTEs, GUID, etc. | 512 |
LBA is used exclusively, there is no CHS mapping. While it's possible to use LBA 34 to start partitions, due to the prevalence of MBR track boundary requirements the first partition often starts at LBA 63. This allows chained bootloaders such as GRUB to store their Stage 1.5 images similar to the MBR technique prior to sector 63.
#### Legacy MBR Sector
Notice the Legacy MBR is clearly defined per the specification; this allows booting a GPT-based storage medium using BIOS techniques as it contains the same area for bootstrap code and PTEs in the same disk locations.
The bootstrap code area remains, only the first PTE is used and it denotes a type of EFI. This sufficiently protects the disk from tools which do not understand EFI, as they should report simply a partition of type "unknown" in the worst case scenario.
#### Partition Table Entry
The PTE format of GPT is very similar to the MBR style and should come as no surprise; most notable of the structure is the use of 8-byte (64-bit) values for the LBA address. Much like MBR, this defines a hard limit on the maximum LBA value that could be addressed as useful storage within GPT.
**128-byte PTE**
| **Length** | **Description** |
| ---------- | --------------------------------------- |
| 16 | Partition type GUID |
| 16 | Unique partition GUID |
| 8 | First LBA (little endian) |
| 8 | Last LBA (inclusive, usually odd) |
| 8 | Attribute flags |
| 72 | Partition name (36 UTF-16LE code units) |
### OS Compatibility
Userspace tools such as fdisk (2.17.1+) and parted contain checks and balances for this more modern approach - one must ensure to not use "DOS Compatibility Mode" and use Sectors mode inside a utility like fdisk or parted to achieve the desired perfect alignment. Additionally the LVM subsystem starting with 2.02.73 will align to this 1 MiB boundary - previous versions used a 64 KiB alignment, akin to the LBA 63 offset. Same goes with software RAID - as long as it's using the modern Superblock Metadata v1 it will align to 1 MiB.[(1)][c1][(2)][c2][(3)][c3]
## GRUB Bootstrap
Understanding how GRUB loads becomes fairly straightforward once the mechanics of the MBR/GPT world are understood. The installation of GRUB onto the MBR (or, optionally the VBR) consists of three primary parts, the first two of which are concrete in design.
- **Stage 1**
- The `boot.img` 440-byte code is loaded into the boostrap area as defined in the MBR design, and is coded to load the first sector of core.img (the next stage) using LBA48 addressing.
- **Stage 1.5 MBR**
- The `core.img` ~30 KiB code is loaded into the 62 empty sectors between the end of the MBR and beginning of the first partition (sector 63). This code contains the ability to recognize filesystems to read stage 2 configuration.
- **Stage 1.5 GPT**
- The `core.img` ~30 KiB code is loaded starting at sector 34 after the GPT structure. This code contains the ability to recognize filesystems to read stage 2 configuration
- **Stage 2**
- This stage reads the configurations by file/path names under /boot/grub to build the TUI and present choices. The majority of userspace code and configuration is located here
Once stage 2 is loaded this is where the higher level GRUB magic begins; the most visible example of this is the user interface allowing for selection of multiple boot choices on multiple partitions. UEFI is similar, however instead of core.img a different piece of code (grub\*.efi) is copied to the EFI System Partition and acts as the UEFI Application as outlined above.
### LVM Boot Volumes
The default storage mechanism for `pvcreate` is to use the second 512-byte sector of the MBR/VBR to hold it's metadata structure; however the LVM subsystem will scan the first 4 sectors of the MBR/VBR for it's data. The physical volume label begins with the string **LABELONE** and contains 4 basic items of information:
- Physical volume UUID
- Size of block device in bytes
- NULL-terminated list of data area locations
- NULL-terminated lists of metadata area locations
Metadata locations are stored as offset and size (in bytes). There is room in the label for about 15 locations, but the LVM tools currently use 3: a single data area plus up to two metadata areas.[(4)][c4]
Historically the lack of LVM-capable boot loaders (such as LILO and GRUB1) required the /boot/ filesystem to reside at the beginning of the disk (yet another CHS legacy issue) or be of a more basic filesystem format such as ext2 or ext3. With the advent of GRUB2 (GRUB version 2) the ability exists to read from more complex filesystems such as ext4, LVM and RAID.[(5)][c5]
In the default installation of many server-oriented Linux distributions such as Ubuntu 14.04 LTS and RHEL/CentOS 7 the /boot/ partition is still non-LVM and the first partition of the disk for maximum backwards compatibility even though they use GRUB2.
## Scanning Devices
When expanding an existing device or adding a new device, the underlying controller(s) needs to be performed. There are two separate interfaces into the kernel to perform this work, each a little different than the other.
### Scan for new Devices
Given either a single controller or multiple controllers for the same storage (in the case of high availability) we need to issue a scan requests to those controllers to look for new devices presented and create /dev device nodes for the ones found. The `host0` is always the local controller, `host1` and above tend to be add-in controllers to external storage for example.
```
# echo "- - -" > /sys/class/scsi_host/hostX/scan (where X is your HBA)
```
Local controller (which includes VMware vDisks):
```
# echo "- - -" > /sys/class/scsi_host/host0/scan
```
Add-on HBA (Host Based Adapater) cards of some sort:
```
# echo "- - -" > /sys/class/scsi_host/host1/scan
# echo "- - -" > /sys/class/scsi_host/host1/scan
```
### Rescanning and Deleting Devices
The scenario is an existing block device is already presented (i.e. /dev/sda) and it's been expanded upstream of the OS already - for example the VMware vDisk was grown or the SAN/DAS LUN was expanded. In this case every block device that comprises that piece of storage has to be rescanned -- for a single controller it's only one device, but for HA situations (using Multipath for instance) all individual devices need rescanned.
```
# echo 1 > /sys/block/sda/device/rescan
```
Multiple paths to the same storage (Multipath, etc.):
```
# echo 1 > /sys/block/sdb/device/rescan
# echo 1 > /sys/block/sdc/device/rescan
```
Deleting those block device entries from the Linux kernel maps is just as easy -- the devices have to be completely unused and released from the OS itself first, and **do not force it** - a kernel panic may (and most probably will) ensue if you try and force a block device delete while the kernel still thinks it's in use.
```
# echo 1 > /sys/block/sdb/device/delete
# echo 1 > /sys/block/sdc/device/delete
```
## The udev Subsystem
Udev is the device manager for the Linux 2.6 kernel that creates/removes device nodes in the /dev directory dynamically. It is the successor of devfs and hotplug. It runs in userspace and the user can change device names using udev rules.[(6)][c6] The udev subsystem allows for a very wide variety of user control over devices, whether they be storage, network, UI (keyboard/mouse) or others - one of the common uses in udev is to name network interfaces.
When it comes to Linux storage this can have subtle yet extremely important implications on how the server finds and uses it's boot devices. When the kernel initializes it and the udev subsystem scan the bus and created device nodes for the storage devices it finds. Logistically, this means if a supported HBA (Host Based Adapter, a PCI-based Fiber/SAS card for instance) is found before the internal SCSI controller it's highly possible (and in practice and experience **does happen**) that a device consumes device node "sda" on the system from outside the chassis (SAN/DAS) instead of the internal disk or RAID array.
Care should be taken when researching modern udev - in some distributions it's now been subsumed by systemd and is no longer a discreet entity within the Linux ecosphere; the specific methodology has changed for certain parts of the process. For example, in traditional udev the bootstrap process initialized /dev from the pre-prepared data in /lib/udev/devices tree; in the systemd implementation it reads from /etc/udev/hwdb.bin instead.
### HBA Blacklisting
One graceful solution to the boot-from-HBA problem is to simply blacklist the kernel module **from initrd only** to prevent the kernel from having the device driver on boot, so it doesn't find the HBA controllers. Once the kernel switches to the real root filesystem and releases the initrd, it has already assigned "sda" to the internal expected array and can then load the HBA driver at runtime and initialize the controllers and find storage.
The `rdblacklist` mechanism is used on the kernel boot line of your GRUB configuration - just append as needed with the specific HBA to blacklist:
```
Blacklist the Brocade HBAs:
rdblacklist=bfa
Blacklist the QLogic HBAs:
rdblacklist=qla2xxx
```
The kernel will then respect the `/etc/modprobe.d/*.conf` entries to load the appropriate module once it's switched to the real root filesystem and discovers the devices during scan.
## References
- <http://en.wikipedia.org/wiki/X86>
- <http://en.wikipedia.org/wiki/X86-64>
- <http://en.wikipedia.org/wiki/Page_table>
- <http://en.wikipedia.org/wiki/Linux>
- <http://en.wikipedia.org/wiki/BIOS>
- <http://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface>
- <http://en.wikipedia.org/wiki/BIOS_Boot_partition>
- <http://en.wikipedia.org/wiki/Cylinder-head-sector>
- <http://en.wikipedia.org/wiki/Zone_bit_recording>
- <http://en.wikipedia.org/wiki/Logical_block_addressing>
- <http://en.wikipedia.org/wiki/Machine_code>
- <http://en.wikipedia.org/wiki/INT_13>
- <http://en.wikipedia.org/wiki/Master_boot_record>
- <http://en.wikipedia.org/wiki/Volume_Boot_Record>
- <http://en.wikipedia.org/wiki/GUID_Partition_Table>
- <http://en.wikipedia.org/wiki/GRUB>
- <http://en.wikipedia.org/wiki/Mass_storage_device>
- <http://en.wikipedia.org/wiki/Fixed_disk>
- <http://en.wikipedia.org/wiki/Boot_sector>
- <http://en.wikipedia.org/wiki/Advanced_Format>
- <http://en.wikipedia.org/wiki/Memory_management_unit>
- <http://en.wikipedia.org/wiki/Physical_Address_Extension>
- <http://en.wikipedia.org/wiki/Linux_startup_process>
- <http://en.wikipedia.org/wiki/Udev>
- <http://wiki.osdev.org/Paging>
- <https://lwn.net/Articles/188056/>
- <http://timocharis.com/help/udev.html>
## Citations
1. <http://www.thomas-krenn.com/en/wiki/Partition_Alignment>
2. <http://www.ibm.com/developerworks/linux/library/l-4kb-sector-disks/#benchmarks>
3. <http://www.rodsbooks.com/gdisk/advice.html>
4. <https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/lvm_metadata.html>
5. <http://www.gnu.org/software/grub/manual/grub.html#Changes-from-GRUB-Legacy>
6. <http://www.linux.com/news/hardware/peripherals/180950-udev>
[c1]: http://www.thomas-krenn.com/en/wiki/Partition_Alignment
[c2]: http://www.ibm.com/developerworks/linux/library/l-4kb-sector-disks/#benchmarks
[c3]: http://www.rodsbooks.com/gdisk/advice.html
[c4]: https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/lvm_metadata.html
[c5]: http://www.gnu.org/software/grub/manual/grub.html#Changes-from-GRUB-Legacy
[c6]: http://www.linux.com/news/hardware/peripherals/180950-udev

399
md/lvm_mechanics.md Normal file
View file

@ -0,0 +1,399 @@
# LVM Mechanics
## Contents
- [Overview](#overview)
- [Best Practices](#best-practices)
- [Device Mapper](#device-mapper)
- [kpartx](#kpartx)
- [LVM Components](#lvm-components)
- [Physical Volumes](#physical-volumes)
- [Whole Device vs Partition](#whole-device-vs-partition)
- [Volume Groups](#volume-groups)
- [Migrating Volume Groups](#migrating-volume-groups)
- [Logical Volumes](#logical-volumes)
- [LV Sizing Methods](#lv-sizing-methods)
- [LVM Filters](#lvm-filters)
- [LVM Snapshots](#lvm-snapshots)
- [Reverting Snapshots](#reverting-snapshots)
- [Selected Examples](#selected-examples)
- [Expanding VM and LV](#expand-vg-and-lv)
- [Migrate PVs](#migrate-pvs)
- [LVM Metadata Example](#lvm-metadata-example)
- [References](#references)
## Overview
After understanding the [x86 storage design](linux_x86_storage.md) and [partitioning](linux_partitioning.md), the next step is LVM - Logical Volume Management. Generically, LVM is the concept of taking individual pieces of storage (whole disks or individual partitions) and combining them together with a layer of software to make the group appear as one single entity. This single entity can then be sub-divided into logical filesystems and treated individually, even though they share the same physical storage.
While we use the short phrase LVM in practice, _technically_ we are referring to **LVM2** as opposed to LVM1. LVM1 hasn't been around in modern distros for quite some time once the Device Mapper infrastructure was introduced to the kernel. LVM2 utilizes device-mapper fully, unlike LVM1 - we are discussing only LVM2 as LVM here.
LVM has many configuration options, this article is an introduction to the overall LVM world and does not cover all the ways `lvm.conf`​ can be tuned. See `man 5 lvm.conf`​ for more information.
## Best Practices
A few rules to live by in the LVM world:
- Name the Volume Group with a name that represents the design - for example _vglocal01_, _vgiscsi05_, _vgsan00_, _vgraid5_, etc.
- Never combine two disparate objects together - for example, do not combine local (in the chassis) storage with remote (iSCSI/SAN/DAS) storage
- Never combine different performance tiers - for example, do not combine a RAID-1 array and RAID-5 array in the same group
- Never combine non-partitioned and partitioned devices - this could lead to performance issues or end-user confusion in the future
## Device Mapper
At the heart of the system is the [Device Mapper](device_mapper_mechanics.md) kernel level infrastructure introduced with kernel 2.6.x; it's a kernel framework for mapping block level devices to virtual block devices. Not only is it the underpinnings of LVM2 but also RAID (_dm-raid_), cryptsetup (_dm-crypt_) and others like _dm-cache_. It's the component that provides the snapshots feature for LVM as well.
Normally ioctls (I/O controls) are sent to the block device itself; within the DM world there is a special endpoint `/dev/mapper/control` that is used instead, all ioctls are sent to this device node. The userspace tool `dmsetup` can be used to manually investigate and manipulate the device-mapper subsystem; `dmsetup ls` is a common usage used by techs to quickly review device maps.
With LVM this is exactly what we're doing - creating virtual block devices on top of physical block devices.
### kpartx
While not technically LVM related, a mention of `kpartx` should be made here - where the tool `partx` is designed to read partition maps and create the proper device nodes, the `kpartx` tool is used to read partition tables and create device maps over partition segments detected. This tends to come into play more when using Multipath than LVM, however be aware there are two tools to perform these functions in their own discreet fashion.
## LVM Components
LVM is the name/acronym of the entire puzzle; there are three discreet components to make it all work:
- **Physical Volume (PV)**
- A physical volume is the actual storage space itself as a single item. It can be the whole object (entire local drive, entire SAN LUN, etc.) or the individual partitions on that storage device. Each partition is it's own discreet physical volume in the latter case.
- **Volume Group (VG)**
- A volume group is a collection of physical volumes as a single entity. It is used as a single block of storage to carve up into logical volumes. Space from one logical volume can be transferred to another logical volume within the same group, filesystem type permitting.
- **Logical Volume (LV)**
- A logical volume is an area of space - akin to a partition on a drive - that is used to hold the filesystem. A logical volume cannot span volume groups, and is the object which is manipulated with userspace tools like mount, umount, cryptsetup, etc.
### Physical Volumes
A PV is the most direct part of the LVM puzzle, and the most critical. The PV first has to be created using the command _pvcreate_ -- what this is doing is writing a metadata block to the 2nd sector up to 1MB of the VBR (on a partitioned device) or the beginning of the device if it is not partitioned. When scanning for metadata the LVM subsystem will read this data to determine all information needed - for example, on a 512-byte sector drive 2048 sectors might be scanned to try and locate the PV metadata.
The pvcreate utility provides basic protection - if a partition table is present but there are no partitions defined, it will still not work - the partition table (whether MBR or GPT) must be zapped before you can use a whole device as a PV without partitions. Otherwise a partition must be created and then pvcreate used on that partition - but **be careful**, if a device is _already_ a PV it is possible to use a tool like fdisk to create a partition table in the first 512-byte sector afterwards\!
The metadata area starts with the string `LABELONE` followed by several groups of data:
- Physical volume UUID
- Size of block device in bytes
- NULL-terminated list of data area locations
- NULL-terminated lists of metadata area locations
What's important to understand is that the concept of the higher level VG and LV are stored in this PV metadata. This protects the VG and LV from activating if one of the PVs in the group is missing - by every PV having a complete view of the objects it becomes self-referential and self-sufficient. Because this metadata also stores the location of data as it's written it's critical that all items be present before it becomes available for use to the end user.
This design makes the PVs themselves portable as a group - a group of disks can be removed from one server and presented to another, and so long as all PVs are present and the metadata is intact that LVM group can simply be activated on the new server with ease.
An example of the PV-stored LVM metadata can be found at the end of this page.
#### Whole Device vs Partition
The decision whether or not to use partitions with a PV has only one concrete advantage, but several disadvantages in practice. So long as the alignment to storage is correct (radial geometry) there is no performance gain or loss in using either method. The concrete advantage to using an entire device for the PV is the ability to expand that PV using `pvresize` at a later date. This simplifies the work in expanding the underlying storage and increasing the PV/VG/LV sizes.
The practical disadvantages to using an entire device all revolve around visibility. For example, when a tech uses fdisk/parted/gdisk and does not see a partition they may be inclined to think the drive is unused; this can result in adding a partition table to a device already in a VG by accident. The boot device which does need a MBR/GPT to operate cannot be used as a whole disk, so if other PVs will be added later it's considered bad practice to combine partitioned and non-partitioned PVs into the same VG. While not technical disadvantages these considerations should be taken on when setting up the PV.
### Volume Groups
A volume group is the abstract layer that sits between the PV and the LV -- it's role is to combine and hide the physical block devices, presenting one picture of unified storage to the LVs on top. The VG operates on **Physical Extents** - think of these as blocks of data of a given size, where 4 MiB is the default size when creating the VG. Much like the physical sectors of a disk, the PE is treated as a unit -- data is read/written to it as one chunk and the ability to move it is handled in the same chunk. During vgcreate different sizes can be chosen depending on expected workload - 1 KiB minimum and must be a power of 2.
The default VG allocation policy when writing PEs is to use _normal_ mode -- this means it has some basic intelligence built in to prevent parallel stripes from being placed on the same PV for instance. This can be changed to other methods - for example, _contiguous_ policy requires new PEs being written to be placed right after the existing PEs; the _cling_ policy places new PEs on the same PV as existing PEs in the same stripe of the LV. Note this is not the same as the LV _type_ of segments.
A single VG can span many, many PVs however a VG cannot be combined with another VG; ergo, a VG has a finite size of the PVs underneath it and how they're used by the LVs on top. The VG can be expanded or reduced by adding/removing PVs or expanding/reducing the existing PVs.
#### Migrating Volume Groups
Volume Groups are independent of the system itself, providing the VG is not the container for the root filesystem of the server. They can be exported from one system, physically moved, then imported to another system. The `vgexport` command will clear the VG metadata hostname and deactivate it from the current system, while the `vgimport` command will set the VG metadata hostname to the current system and activate it. The VG should be deactivated with `vgchange` first to ensure it's unmounted and not in use.
```
# vgs vglocal
VG #PV #LV #SN Attr VSize VFree
vglocal 1 1 0 wz--n- 50.00g 0
# vgexport vglocal
Volume group "vglocal" has active logical volumes
# vgchange -an vglocal
0 logical volume(s) in volume group "vglocal" now active
# vgexport vglocal
Volume group "vglocal" successfully exported
# vgimport vglocal
Volume group "vglocal" successfully imported
```
### Logical Volumes
The logical volume is the top-most container segmenting a given amount of space from the underlying VG; a LV is restricted to the single VG it is on top of, a LV cannot span 2 more more VGs for increased space. To increase space in a LV the underlying VG has to be increased first. The LV acts as the final virtual block device endpoint of the device mapper design -- this container is what is used with tools like `mkfs.ext4`, `mount` and so forth. It acts and reacts just like a real block device for all intents and purposes, save that it is more like a single partition instead of a whole device (doesn't use a MBR/GPT table).
The LV can be manipulated in 2 primary ways - by the /dev/_vgname_/_lvname_ symlink or the /dev/mapper/_vgname-lvname_ symlink. Using either is fine since they point to the same actual, real device mapper node entry in the /dev/ tree that corresponds to the virtual block device:
```
# ls -l /dev/vglocal/lvtest /dev/mapper/vglocal-lvtest
lrwxrwxrwx 1 root root 7 May 8 19:54 /dev/mapper/vglocal-lvtest -> ../dm-0
lrwxrwxrwx 1 root root 7 May 8 19:54 /dev/vglocal/lvtest -> ../dm-0
# lvdisplay | awk '/LV Name/{n=$3} /Block device/{d=$3;sub(".*:","dm-",d);print d,n;}'
dm-0 lvtest
```
So our actual DM node is /dev/dm-0 -- this should never be used as it's possible it could change after a reboot for instance; always use the symlink name instead for maximum resilience to change. These have the major character node type `253` in Linux, the minor character is simply the position it was discovered added in by the kernel. These can be examined with the `dmsetup` tool as outlined above:
```
# dmsetup ls
vglocal-lvtest (253:0)
# dmsetup info vglocal-lvtest
Name: vglocal-lvtest
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 0
Number of targets: 1
UUID: LVM-c1MITBqcCORe5icvRwAhlAQUJvVceVDfQXSRQz0T42vcwnPKbotggmXwxrTWB1l5
```
Logical volumes can be created in a number of different modes that might look familiar: linear, striped and mirrored are the three most common. The default mode is linear - use the space from beginning to end as a whole. Striped and mirrored are exactly like your basic RAID - both require minimum 2 PVs and write across both PVs like RAID-0 and RAID-1. Other modes exist, one of which is Snapshot -- the usage of striped and mirrored LVs is not being covered here as they tend to be specific use case oriented solutions.
#### LV Sizing Methods
A note here about specifying the size of the logical volume to be created, extended or reduced: two commandline options exist to perform the same work but tend to be confusing. Think of them this way:
| **Flag** | **Usage** |
| -------- | ---------------------------------------------- |
| `-l` | dynamic math - "100%VG", "+90%VG" and so forth |
| `-L` | absolute math - "100G", "+30G" and so forth |
Thinking of these flags in this manner aides in their usage later - the same operation could be used using either one of them, however it may be easier to do with one or the other depending on the exact situation. The `-l` by default with no quantifier is using PE as a value, handy if you need to move an exact number of physical extents around for the need at hand.
## LVM Filters
One of the more critical parts to using LVM within environments which contain [multiple HA paths](device_mapper_multipath.md) to the storage is setting up LVM filters to ignore the individual paths and only respect the meta (pseudo) path to the storage, whether that be SAN, DAS or iSCSI in nature. If the filters are not set correctly the underpinnings of LVM will use a single path by name – if that path dies, LVM dies.
The way filters are written is simple - "add these, remove these" in nature. Looking at a few examples reveals the concepts used in `/etc/lvm/lvm.conf`:
```
# A single /dev/sda internal device, PowerPath devices:
filter = [ "a|^/dev/sda[0-9]+$|", "a|^/dev/emcpower|", "r|.*|" ]
# Two internal devices, /dev/sda and /dev/sdb, and PowerPath devices:
filter = [ "a|^/dev/sd[ab][0-9]+$|", "a|^/dev/emcpower|", "r|.*|" ]
# Two internal devices, /dev/sda and /dev/sdb, and Device Mapper Multipath devices:
filter = [ "a|^/dev/sd[ab][0-9]+$|", "a|^/dev/mapper/mpath|", "r|.*|" ]
# Two internal devices, Device Mapper Multipath and PowerPath devices all at once:
filter = [ "a|^/dev/sd[ab][0-9]+$|", "a|^/dev/mapper/mpath|", "a|^/dev/emcpower|", "r|.*|" ]
```
Note in the above examples that regular expressions can be used for configuration.
## LVM Snapshots
A snapshot is a special type of LV that has a method called Copy on Write (CoW) used to store a point in time view of the source LV plus all changes since that point. A source LV is specified during creation and the name of the new snapshot LV; the snapshot LV **must** exist on the same VG as the source LV. The snapshot LV only needs to be large enough to hold changes made since the time it was taken, it does not need to be the same size as the source entirely. The VG must have this space free - it cannot be used by any LV already.
When creating the snapshot LV basically a copy of the inode table is taken - hence the need for the source and snapshot LVs to exist within the same VG. At that point all new changes are recorded in the CoW table to either be discarded or applied depending on usage. However, be aware that a lot of magic happens under the hood to support this\! It's **not** a simple LV, let's take a look:
```
# lvremove /dev/mapper/vglocal-lvtest
# lvcreate -l 50%VG -n lvtest vglocal
# lvcreate -L 10G -s -n lvsnap /dev/vglocal/lvtest
# lvs
LV VG Attr LSize Pool Origin Data%
lvsnap vglocal swi-a-s--- 10.00g lvtest 0.00
lvtest vglocal owi-a-s--- 50.00g
# dmsetup ls
vglocal-lvsnap-cow (253:3)
vglocal-lvsnap (253:1)
vglocal-lvtest (253:0)
vglocal-lvtest-real (253:2)
```
Notice how there are two additional device maps - "vglocal-lvsnap-cow" and "vglocal-lvtest-real" - that are used behind the scenes to store and work with those CoW changes **to the source volume** that occur while the snapshot is alive. If the snapshot fills up with changes and flips to read-only mode it can be a bit of an ordeal to get the snapshot fully released correctly if something goes wrong within LVM, so proper planning should be take to remove the snapshot in a timely fashion or plan for it's expected growth.
### Reverting Snapshots
It is possible to roll back changes made to the original logical volume (lvtest) by merging the original LV extents from the CoW volume back onto the origin volume, provided that the _snapshot-merge_ target is available.
```
Check if supported by kernel
# dmsetup targets | grep snapshot-merge
snapshot-merge v1.1.0
```
This operation is seamless to the user and starts automatically when the origin (lvtest) and snapshot (lvsnap) volumes are activated but not opened. If either the origin or snapshot volumes are opened, the merge operation is deferred until the next time both volumes are activated. As soon as the merge operation starts, the origin volume can be opened and the filesystem within it mounted.
From this point, all read and write operations to the origin volume are seamlessly routed to the correct logical extents (at the start of the merge operation, these would be the original extents on lvsnap-cow and the unchanged extents on lvtest-real) until the merge is complete. The lvsnap-cow, lvsnap and lvtest-real volumes are then removed from the system.
Following the lvtest/lvsnap the following command would start the merge/rollback operation:
```
# lvconvert --merge /dev/vglocal/lvsnap
```
## Selected Examples
### Expand VG and LV
One of the more common scenarios: your boot disk has two partitions, 1 and 2. 1 is /boot using non-LVM and 2 is LVM based / (root) filesystem. You have run out of space and wish to add more -- the new space can be either a new partition on the same storage device that was just expanded (SAN/DAS LUN, VMware vDisk, etc.) or a new device and partition entirely.
After creating your new partition and using `pvcreate` on it, review the mission goal - we're adding the new space from xvdb3 to the VG, growing the LV and resizing the ext4 filesystem.
```
# pvs; echo; vgs; echo; lvs
PV VG Fmt Attr PSize PFree
/dev/xvdb2 vglocal lvm2 a-- 10.00g 0
/dev/xvdb3 lvm2 a-- 20.00g 20.00g
VG #PV #LV #SN Attr VSize VFree
vglocal 1 1 0 wz--n- 10.00g 0
LV VG Attr LSize
lvroot vglocal -wi-a----- 10.00g
```
First, add the new PV into the VG. Then grow the LV with the newly added space. Lastly grow the ext4 filesystem itself:
```
# vgextend vglocal /dev/xvdb3
Volume group "vglocal" successfully extended
# lvextend -l +100%FREE /dev/vglocal/lvroot
Extending logical volume lvroot to 29.99 GiB
Logical volume lvroot successfully resized
# resize2fs /dev/vglocal/lvroot
Resizing the filesystem on /dev/vglocal/lvroot to 7862272 (4k) blocks.
The filesystem on /dev/vglocal/lvroot is now 7862272 blocks long.
```
Check our work again:
```
# pvs; echo; vgs; echo; lvs
PV VG Fmt Attr PSize PFree
/dev/xvdb2 vglocal lvm2 a-- 10.00g 0
/dev/xvdb3 vglocal lvm2 a-- 20.00g 0
VG #PV #LV #SN Attr VSize VFree
vglocal 2 1 0 wz--n- 29.99g 0
LV VG Attr LSize
lvroot vglocal -wi-a----- 29.99g
```
### Migrate PVs
The scenario: an existing LV contains a PV we wish to replace - this could be for migrating from one type of storage to another for instance, or replacing several small PVs with one large PV for better performance at the storage side. The `pvmove` command is used, and the PV being added must be at least as large as the one being removed\!
Existing LV has one PV in VG "vglocal" of 9.3 GiB in size:
```
# pvs
PV VG Fmt Attr PSize PFree
/dev/xvdb1 vglocal lvm2 a-- 9.31g 0
/dev/xvdb2 lvm2 a-- 10.00g 10.00g
```
We will replace xvdb1 with xvdb2 - note how it's 10 GiB, at least as large as the one being replaced. After the VG is extended to add the second PV, we check again and see that it's been added but all the PEs (_PFree_) from xvdb2 are still unused. **Do not extend the LV** on top of the VG, the new PV must show as free in order to use it as a migration device.
```
# vgextend vglocal /dev/xvdb2
# pvs
PV VG Fmt Attr PSize PFree
/dev/xvdb1 vglocal lvm2 a-- 9.31g 0
/dev/xvdb2 vglocal lvm2 a-- 10.00g 10.00g
```
Now we move all the PEs from xvdb1 to xvdb2 with a few commandline options to show verbose info and a progress update every 5 seconds. After all the PEs have been moved to xvdb2 we do a quick check again, then if all looks kosher we remove the old PV:
```
# pvmove -v -i5 /dev/xvdb1 /dev/xvdb2
# pvs
PV VG Fmt Attr PSize PFree
/dev/xvdb1 vglocal lvm2 a-- 9.31g 9.31g
/dev/xvdb2 vglocal lvm2 a-- 10.00g 704.00m
# vgreduce /dev/xvdb1
# pvs
PV VG Fmt Attr PSize PFree
/dev/xvdb1 lvm2 a-- 9.31g 9.31g
/dev/xvdb2 vglocal lvm2 a-- 10.00g 704.00m
```
## LVM Metadata Example
Using the `lvmdump -m` command is the easiest way to extract the metadata from all the PVs on the system; here is an example of the data with basic formatting added (spaces/indents, etc.) for easier readability. Note that the metadata area stores rolling revisions of the changes made, it might be useful in a given situation to determine what has transpired.
```
LABELONE LVM2 001wTDfgkU6aRyAwCheopo1LeCEFWWodQbd
vglocal {
id = "7PHX1A-PJ0n-fgdv-qRup-In2G-dah1-iOgWm4"
seqno = 2
format = "lvm2" # informational
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
extent_size = 8192
max_lv = 0
max_pv = 0
metadata_copies = 0
physical_volumes {
pv0 {
id = "wTDfgk-U6aR-yAwC-heop-o1Le-CEFW-WodQbd"
device = "/dev/xvdb1"
status = ["ALLOCATABLE"]
flags = []
dev_size = 209711104
pe_start = 2048
pe_count = 25599
}
}
logical_volumes {
lvtest {
id = "ZSHT4d-K4lc-pUma-6UtB-vJ9e-9jox-hRTibF"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_host = "r7rc-ha"
creation_time = 1399583935
segment_count = 1
segment1 {
start_extent = 0
extent_count = 12800
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv0", 0
]
}
}
}
}
# Generated by LVM2 version 2.02.100(2)-RHEL6 (2013-10-23): Thu May 8 21:18:55 2014
contents = "Text Format Volume Group"
version = 1
description = ""
creation_host = "localhost" # Linux localhost 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25 19:59:55 UTC 2014 x86_64
creation_time = 1399583935 # Thu May 8 21:18:55 2014
```
## References
- <http://www.tldp.org/HOWTO/LVM-HOWTO/lvm2faq.html>
- <http://en.wikipedia.org/wiki/Device_mapper>
- <http://www.funtoo.org/LVM_Fun>

218
md/lvm_snapshot_merging.md Normal file
View file

@ -0,0 +1,218 @@
# LVM Snapshot Merging
## Contents
- [Overview](#overview)
- [General Setup](#general-setup)
- [Preparing a Scenario](#preparing-a-scenario)
- [Merging a Snapshot](#merging-a-snapshot)
- [Advanced Usage](#advanced-usage)
- [Data Retention](#data-retention)
- [References](#references)
## Overview
Within the LVM2 Device Mapper infrastructure a method and kernel module exists to merge the contents of a snapshot back into it's source using `lvconvert`; the typical use case for many snapshots is to back up and discard, this article outlines an alternate use where the need is to re-merge the changes back to the source instead.
## General Setup
In order to work with the `lvconvert` merging process:
- The **LVM2** packages for the distro must be installed
- The kernel module `dm-snapshot.ko` must be loaded
- A snapshot to merge must exist
Check for the snapshot-merge feature using `dmsetup targets` and load the module as needed:
```
# dmsetup targets
mirror v1.14.0
striped v1.5.6
linear v1.1.0
error v1.2.0
# modprobe -v dm-snapshot
insmod /lib/modules/2.6.32-642.4.2.el6.x86_64/kernel/drivers/md/dm-bufio.ko
insmod /lib/modules/2.6.32-642.4.2.el6.x86_64/kernel/drivers/md/dm-snapshot.ko
# dmsetup targets
snapshot-merge v1.3.6
snapshot-origin v1.9.6
snapshot v1.13.6
mirror v1.14.0
striped v1.5.6
linear v1.1.0
error v1.2.0
```
## Preparing a Scenario
For instructional purposes, we need to prepare a scenario:
- A LV with snapshot in the VG is mounted and in use
- A snapshot was active/mounted and we add new data to it
The setup looks like:
```
Create a new snapshot and add dummy data:
# lvcreate -s /dev/vgcbs00/lvdata -L 10G -n lvdata_snap
# mount /dev/vgcbs00/lvdata_snap /snap
# for ii in 5 6 7 8; do dd if=/dev/zero of=/snap/data.$ii bs=4M count=10$ii; done
Examine the results:
# ls /data/ /snap/
/data/:
data.1 data.2 data.3 data.4
/snap/:
data.1 data.2 data.3 data.4 data.5 data.6 data.7 data.8
# df -h /data/ /snap/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgcbs00-lvdata
37G 1.7G 34G 5% /data
/dev/mapper/vgcbs00-lvdata_snap
37G 3.4G 32G 10% /snap
```
So here we have the origin (source) LV `lvdata` and a snapshot of it `lvdata_snap`; 4 additional data files were added to the mounted snapshot `/snap/` in order to simulate a a snapshot which has had data written to it since it was instantiated. We see there is now twice as much data in the snapshot and the additional files we want to merge back to the origin.
## Merging a Snapshot
To merge the snapshot back to it's origin:
1. Ensure the origin has enough space to hold the contents of the additional snapshot data
2. Ensure the **origin and snapshot are unmounted** from the filesystem
3. Use the `lvconvert` command to merge the snapshot to the origin
> It is possible to perform the merge while the origin is online, see the **Advanced Usage** section for this scenario.
Using the `lvconvert` command is very straightforward, be sure to use the `-i` flag to set an update interval for progress output. Be aware that the **snapshot will be deleted after merge** so ensure this is the expected outcome desired:
```
# umount /snap
# umount /data
# lvconvert --merge -i 2 vgcbs00/lvdata_snap
Merging of volume lvdata_snap started.
lvdata: Merged: 83.4%
lvdata: Merged: 84.8%
...lots of status...
lvdata: Merged: 100.0%
Merge of snapshot into logical volume lvdata has finished.
Logical volume "lvdata_snap" successfully removed
# mount /dev/vgcbs00/lvdata /data
# ls /data/
data.1 data.2 data.3 data.4 data.5 data.6 data.7 data.8
```
You have now successfully merged the snapshot content back to it's origin.
## Advanced Usage
It is possible to reduce the offline time by performing an online merge when the LV is next activated; this can be done on the fly if the volume group can be deactivated (so it cannot be the home of the root filesystem) however requires a **complete deactivation**, making it unsuitable for the root partition or another volume which cannot be deactivated while the server is online. The process is similar to the basic usage, you simply have to leave the origin mounted, perform the commands then wait for it to complete in the background.
Using the same scenario setup prepared above:
```
# umount /snap
# lvconvert --merge vgcbs00/lvdata_snap
Logical volume vgcbs00/lvdata contains a filesystem in use.
Can't merge over open origin volume.
Merging of snapshot vgcbs00/lvdata_snap will occur on next activation of vgcbs00/lvdata.
(stop service using /data)
# umount /data
# vgchange -an vgcbs00
# vgchange -ay vgcbs00
# mount /dev/vgcbs00/lvdata /data
(start service using /data)
```
At this point you can `ls` immediately and see the changed data on the source, however be aware it's still merging in the background. Using `lvs -a` to watch the `Data%` column decrease it's way to zero is how status is checked, and when it's complete it will **delete the snapshot** LV on it's own.
```
# lvs -a
LV VG Attr LSize Pool Origin Data%
lvdata vgcbs00 Owi-aos--- 37.50g
[lvdata_snap] vgcbs00 Swi-a-s--- 10.00g lvdata 4.76
...
# lvs -a
LV VG Attr LSize Pool Origin Data%
lvdata vgcbs00 Owi-aos--- 37.50g
[lvdata_snap] vgcbs00 Swi-a-s--- 10.00g lvdata 3.17
```
Eventually you will see the `Data%` column decrease to 0.00, then the snapshot will be removed.
## Data Retention
Simply put, the data changes on the snapshot will overwrite the data on the origin volume.
It does not matter if it's a new file or changed, the process will not attempt to merge the actual contents of a file (similar to diff and patch) but instead just replace the data blocks from the snapshot on the origin volume. File deletions are handled the same - if a file is deleted on the snapshot, upon merging it the file will be removed on the origin volume. The process does not differentiate between a binary or text file, all data is treated the same during the merge process.
Starting with a few data files:
```
/data/test-d1.txt
This line was edited on /data before snapshot creation
/data/test-d2.txt
This line was edited on /data before snapshot creation
/data/test-s2.txt
This line was edited on /data after snapshot creation
====
/snap/test-d1.txt
This line was edited on /data before snapshot creation
This line was edited on /data after snapshot creation
/snap/test-d2.txt
This line was edited on /data before snapshot creation
This line was edited on /snap after snapshot creation
/snap/test-s1.txt
This line was edited on /snap after snapshot creation
/snap/test-s2.txt
This line was edited on /snap after snapshot creation
```
We then run the `lvconvert` process outlined above to merge the data and end up with:
```
/data/test-d1.txt
This line was edited on /data before snapshot creation
This line was edited on /data after snapshot creation
/data/test-d2.txt
This line was edited on /data before snapshot creation
This line was edited on /snap after snapshot creation
/data/test-s1.txt
This line was edited on /snap after snapshot creation
/data/test-s2.txt
This line was edited on /snap after snapshot creation
```
As the content of the test text files shows, the snapshot data blocks simply replace the origin whether it's been edited in one place or both - there is no attempt to compare timestamps or perform otherwise more intelligent data merging, it's an all or nothing approach to the merge process.
## References
- <https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/snapshot_merge.html>

360
md/mongodb_basics.md Normal file
View file

@ -0,0 +1,360 @@
# MongoDB Basics
## Contents
- [Introduction](#introduction)
- [Fundamentals](#fundamentals)
- [Installation and Updates](#installation-and-updates)
- [RHEL / CentOS](#rhel--centos)
- [Ubuntu / Debian](#ubuntu--debian)
- [User Management](#user-management)
- [Network Connectivity](#network-connectivity)
- [MongDB Default Ports](#mongodb-default-ports)
- [System Level Software Configuration](#system-level-software-configuration)
- [Red Hat / CentOS](#red-hat--centos)
- [Ubuntu / Debian](#ubuntu--debian-1)
- [Intermediate Troubleshooting](#intermediate-troubleshooting)
- [Kill errant MongoDB Thread](#kill-errant-mongodb-thread)
- [Check Replica Status](#check-replica-status)
- [Check Status](#check-sharding-status)
- [References](#references)
## Introduction
MongoDB is a document-oriented database model; it uses JSON constructs to not only store the data but also to interact with the system itself. Many commands may look a little odd coming from a MySQL background but in general the concept of what you're trying to do is somewhat the same. The 10gen website has a great page detailing how to apply your MySQL knowledge to the MongoDB world:
- <http://docs.mongodb.org/manual/reference/glossary/>
- <http://docs.mongodb.org/manual/reference/sql-comparison/>
- <http://docs.mongodb.org/manual/reference/sql-aggregation-comparison/>
| **SQL Terms/Concepts** | **MongDB Terms/Concepts** |
| ---------------------- | ------------------------------ |
| database | database |
| table | collection |
| row | document or BSON document |
| column | field |
| index | index |
| table joins | embedded documents and linking |
| primary key | primary key |
| aggregation (group by) | aggregation pipeline |
A few select examples from the linked website:
| **SQL Select Statements** | **MongoDB find() Statements** |
| ------------------------------------------------------------ | ------------------------------------------------ |
| `SELECT * FROM users` | `db.users.find()` |
| `SELECT id, user_id, status FROM users` | `db.users.find({},{user_id:1,status:1})` |
| `SELECT * FROM users WHERE status="A" ORDER BY user_id DESC` | `db.users.find({status:"A"}).sort({user_id:-1})` |
> Using the `.explain()` method in MongoDB **runs the query** which is exactly the **opposite of MySQL**. Be very careful you are not using .explain() on a database with any sort of data altering command (think UPDATE / INSERT / DELETE in MySQL)
## Fundamentals
### Installation and Updates
#### RHEL / CentOS
Utilize the standard Yum repository style configuration:
```
# vi /etc/yum.repos.d/10gen.repo
[10gen]
name=10gen Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64
gpgcheck=0
enabled=1
# yum install mongo-10gen mongo-10gen-server
```
#### Ubuntu / Debian
Utilize the standard APT sources style configuration:
```
# apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
# echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' >> /etc/apt/sources.list.d/10gen.list
# apt-get update
# apt-get install mongodb-10gen
```
### User Management
MongoDB uses role-based access control based on database level; the system.users collection contains the data which correlates roughly to the mysql.user table in MySQL, however it is not manipulated quite the same way. The 10gen website has great introductory material on Access Control:
- <http://docs.mongodb.org/manual/core/access-control/>
- <http://docs.mongodb.org/manual/tutorial/enable-authentication/>
- <http://docs.mongodb.org/manual/tutorial/add-user-administrator/>
- <http://docs.mongodb.org/manual/tutorial/add-user-to-database/>
- <http://docs.mongodb.org/manual/tutorial/change-user-password/>
> **Authentication is disabled by default** in an out of the box installation\! Refer to the above documentation and tutorials for basic user administration tasks should they be required or repaired as most production level configurations will have had security practices applied.
### Network Connectivity
#### MongoDB Default Ports
**27017**
- default port for mongod and mongos instances
- change with port with --port / port
- bind with --bind\_ip / bind\_ip
- define Replicat set with --replSet / replSet
- set DB datadir with --dbpath / dbpath
**27018**
- default port when running with --shardsvr / shardsvr
**27019**
- default port when running with --configsvr / configsvr
**28017**
- default port for the web status page
- always accessible at a port + 1000
- disable with --nohttppinterface / nohttpinterface
- no authentication by default
- enable REST interface with --rest / rest
### System Level Software Configuration
Vendor packages place the default configurations, service scripts and data directories in the standard location methodologies. Subtle differences exist between the platforms:
#### Red Hat / CentOS
- /etc/mongod.conf
- /etc/sysconfig/mongod
- /etc/rc.d/init.d/mongod
- /var/log/mongo/mongod.log
- /var/lib/mongo/
- ~/.mongorc.js
#### Ubuntu / Debian
- /etc/mongodb.conf
- /etc/init/mongodb.conf
- /etc/init.d/mongodb
- /var/log/mongodb/mongodb.log
- /var/lib/mongodb/
- ~/.mongorc.js
## Intermediate Troubleshooting
### Kill errant MongoDB Thread
Killing an errant thread in MongoDB is directly analogous to killing one in MySQL - you examine the stack, find the one in question and issue a command to terminate it.
> Do not kill threads which are compacting databases or any background threads which are indexing data - this can lead to database corruption
First, use the `db.currentOp()` mongo shell command to list your threads; this is analogous to `show full processlist` in MySQL.
```
$ mongo
MongoDB shell version: 2.4.5
connecting to: test
> db.currentOp()
{
"inprog" : [
{
"opid" : 2506233,
"active" : true,
"secs_running" : 140,
"op" : "update",
"ns" : "generators.sensor_readings",
"query" : {
"$where" : "function(){sleep(500);return false;}"
},
"client" : "127.0.0.1:51773",
"desc" : "conn20",
"threadId" : "0x7f694753d700",
"connectionId" : 20,
"locks" : {
"^" : "w",
"^generator" : "W"
},
"waitingForLock" : false,
"numYields" : 279,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(0),
"w" : NumberLong(280242564)
},
"timeAcquiringMicros" : {
"r" : NumberLong(0),
"w" : NumberLong(140420592)
}
}
},
{
"opid" : 2507691,
"active" : false,
"op" : "query",
"ns" : "",
"query" : {
},
"client" : "127.0.0.1:51772",
"desc" : "conn19",
"threadId" : "0x7f6962e4a700",
"connectionId" : 19,
"locks" : {
"^generator" : "R"
},
"waitingForLock" : true,
"numYields" : 0,
"lockStats" : {
"timeLockedMicros" : {
},
"timeAcquiringMicros" : {
}
}
}
]
}
```
In the example above we see two threads; the keys to look for are the `waitingForLock`, `secs_running`, and `op` fields of the command. The threads we're looking for is the first one with `opid` 2506233 as it's the one locking up our database; but notice it has `W` in the `locks` subdocument. We kill it with the `db.killOp()` command only if we're sure the data it's writing can be lost – this is a dangerous operation to perform and should be examined carefully. Read operations are generally safe to kill in an emergency.
```
> db.killOp(2506233);
{ "info" : "attempting to kill op" }
> db.currentOp()
{ "inprog" : [ ] }
```
### Check Replica Status
Somewhat similar to MySQL, replication is based on two configurations working together; the core `mongod` process must be started with a config file/command line flag to tell it which replica set it lives. This is the `replSet` keyword and can be any string, so long as all instances (processes) share the same name. For example, here are three processes started on the same server for testing a replica set:
```
# mongod --dbpath 1 --port 27001 --smallfiles --oplogSize 50 \
--logpath 1.log --logappend --fork --replSet w4
# mongod --dbpath 2 --port 27002 --smallfiles --oplogSize 50 \
--logpath 2.log --logappend --fork --replSet w4
# mongod --dbpath 3 --port 27003 --smallfiles --oplogSize 50 \
--logpath 3.log --logappend --fork --replSet w4
```
Once the Replica set is initialized and configured (using `rs.initiate()` and `rs.add()` / `rs.reconfig()` commands), checking the status is done from any member of the set using the `rs.status()` command:
```
$ mongo --port 27002
MongoDB shell version: 2.4.5
connecting to: 127.0.0.1:27002/test
w4:PRIMARY> rs.status()
{
"set" : "w4",
"date" : ISODate("2013-08-19T18:53:23Z"),
"myState" : 1,
"members" : [
{
"_id" : 1,
"name" : "mongo1c:27002",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 586,
"optime" : Timestamp(1376937880, 1),
"optimeDate" : ISODate("2013-08-19T18:44:40Z"),
"self" : true
},
{
"_id" : 2,
"name" : "mongo1c:27003",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 584,
"optime" : Timestamp(1376937880, 1),
"optimeDate" : ISODate("2013-08-19T18:44:40Z"),
"lastHeartbeat" : ISODate("2013-08-19T18:53:21Z"),
"lastHeartbeatRecv" : ISODate("2013-08-19T18:53:21Z"),
"pingMs" : 0,
"syncingTo" : "mongo1c:27002"
},
{
"_id" : 3,
"name" : "mongo1c:27001",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 523,
"optime" : Timestamp(1376937880, 1),
"optimeDate" : ISODate("2013-08-19T18:44:40Z"),
"lastHeartbeat" : ISODate("2013-08-19T18:53:23Z"),
"lastHeartbeatRecv" : ISODate("2013-08-19T18:53:21Z"),
"pingMs" : 0,
"syncingTo" : "mongo1c:27002"
}
],
"ok" : 1
}
```
Notice how the `stateStr` field will help identify who is the PRIMARY (writer) of the set; unlike MySQL the PRIMARY node can be moved around on the fly - whether it's automatic by voting, or manual actions performed (such as taking a node offline for maintenance work). Actions such as `rs.freeze()`, `rs.stepDown()` and `rs.remove()` exist to manipulate the Replica set. Note that you can always query the instance you logged into with the `db.isMaster()` command to get another view of who is the PRIMARY writer.
```
w4:PRIMARY> db.isMaster()
{
"setName" : "w4",
"ismaster" : true,
"secondary" : false,
"hosts" : [
"mongo1c:27002",
"mongo1c:27001",
"mongo1c:27003"
],
"primary" : "mongo1c:27002",
"me" : "mongo1c:27002",
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"localTime" : ISODate("2013-08-19T18:58:40.488Z"),
"ok" : 1
}
```
### Check Sharding Status
- <http://docs.mongodb.org/manual/core/sharding-introduction/>
Connecting to the shard server (mongos) to view the configuration:
```
mongo localhost:27108/admin -u admin -p
mongos> sh.status()
--- Sharding Status ---
sharding version: { "_id" : 1, "version" : 3 }
shards:
{ "_id" : "db1", "host" : "db1:27001,db2:27001,db3:27001" }
{ "_id" : "db2", "host" : "db3:27002,db1:27002,db2:27002" }
{ "_id" : "db3", "host" : "db2:27003,db3:27003,db1:27003" }
databases:
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "generators", "partitioned" : true, "primary" : "db1" }
generators.sensor_readings chunks:
db3 3
db2 6
db1 6
```
## References
- <http://docs.mongodb.org/manual/reference/>
- <http://docs.mongodb.org/manual/reference/method/>
- <http://docs.mongodb.org/manual/replication/>
- <http://docs.mongodb.org/manual/sharding/>
- <http://docs.mongodb.org/manual/reference/sql-comparison/>

51
md/network_quick_reference.md Normal file
View file

@ -0,0 +1,51 @@
# Network Quick Reference
## Class Quick Reference
| **Class** | **Bits** | **Start** | **End** | **Networks** | **Default Mask** | **CIDR** |
| --------- | -------- | --------- | --------------- | ------------:| ---------------- | -------- |
| A | 0 | 0.0.0.0 | 127.255.255.255 | 128 | 255.0.0.0 | /8 |
| B | 10 | 128.0.0.0 | 191.255.255.255 | 16384 | 255.255.0.0 | /16 |
| C | 110 | 192.0.0.0 | 223.255.255.255 | 2097152 | 255.255.255.0 | /24 |
| D (mcast) | 1110 | 224.0.0.0 | 239.255.255.255 | _undef_ | _undef_ | _undef_ |
| E (rsrvd) | 1111 | 240.0.0.0 | 255.255.255.255 | _undef_ | _undef_ | _undef_ |
## Netmask Quick Reference
| **Bits** | **Total Hosts** | **Usable** | **Netmask** | **Inverse Mask** |
| -------- | -----------------:| -----------| --------------- | ----------------:|
| `/0` | 4294967296 (2^32) | 4294967294 | 0.0.0.0 | 255.255.255.255 |
| `/1` | 2147483648 (2^31) | 2147483646 | 128.0.0.0 | 127.255.255.255 |
| `/2` | 1073741824 (2^30) | 1073741822 | 192.0.0.0 | 63.255.255.255 |
| `/3` | 536870912 (2^29) | 536870910 | 224.0.0.0 | 31.255.255.255 |
| `/4` | 268435456 (2^28) | 268435454 | 240.0.0.0 | 15.255.255.255 |
| `/5` | 134217728 (2^27) | 134217726 | 248.0.0.0 | 7.255.255.255 |
| `/6` | 67108864 (2^26) | 67108862 | 252.0.0.0 | 3.255.255.255 |
| `/7` | 33554432 (2^25) | 33554430 | 254.0.0.0 | 1.255.255.255 |
| `/8` | 16777216 (2^24) | 16777214 | 255.0.0.0 | 0.255.255.255 |
| `/9` | 8388608 (2^23) | 8388606 | 255.128.0.0 | 0.127.255.255 |
| `/10` | 4194304 (2^22) | 4194302 | 255.192.0.0 | 0.63.255.255 |
| `/11` | 2097152 (2^21) | 2097150 | 255.224.0.0 | 0.31.255.255 |
| `/12` | 1048576 (2^20) | 1048574 | 255.240.0.0 | 0.15.255.255 |
| `/13` | 524288 (2^19) | 524286 | 255.248.0.0 | 0.7.255.255 |
| `/14` | 262144 (2^18) | 262142 | 255.252.0.0 | 0.3.255.255 |
| `/15` | 131072 (2^17) | 131070 | 255.254.0.0 | 0.1.255.255 |
| `/16` | 65536 (2^16) | 65534 | 255.255.0.0 | 0.0.255.255 |
| `/17` | 32768 (2^15) | 32766 | 255.255.128.0 | 0.0.127.255 |
| `/18` | 16384 (2^14) | 16382 | 255.255.192.0 | 0.0.63.255 |
| `/19` | 8192 (2^13) | 8190 | 255.255.224.0 | 0.0.31.255 |
| `/20` | 4096 (2^12) | 4094 | 255.255.240.0 | 0.0.15.255 |
| `/21` | 2048 (2^11) | 2046 | 255.255.248.0 | 0.0.7.255 |
| `/22` | 1024 (2^10) | 1022 | 255.255.252.0 | 0.0.3.255 |
| `/23` | 512 (2^9) | 510 | 255.255.254.0 | 0.0.1.255 |
| `/24` | 256 (2^8) | 254 | 255.255.255.0 | 0.0.0.255 |
| `/25` | 128 (2^7) | 126 | 255.255.255.128 | 0.0.0.127 |
| `/26` | 64 (2^6) | 62 | 255.255.255.192 | 0.0.0.63 |
| `/27` | 32 (2^5) | 30 | 255.255.255.224 | 0.0.0.31 |
| `/28` | 16 (2^4) | 14 | 255.255.255.240 | 0.0.0.15 |
| `/29` | 8 (2^3) | 6 | 255.255.255.248 | 0.0.0.7 |
| `/30` | 4 (2^2) | 2 | 255.255.255.252 | 0.0.0.3 |
| `/31` | 2 (2^1) | 0 | 255.255.255.254 | 0.0.0.1 |
| `/32` | 1 (2^0) | 1 | 255.255.255.255 | 0.0.0.0 |

248
md/nfs_debugging.md Normal file
View file

@ -0,0 +1,248 @@
# NFS Debugging
## Contents
- [Userspace Tools](#userspace-tools)
- [Kernel Interfaces](#kernel-interfaces)
- [NFSD debug flags](#nfsd-debug-flags)
- [NFS debug flags](#nfs-debug-flags)
- [NLM debug flags](#nlm-debug-flags)
- [RPC debug flags](#rpc-debug-flags)
- [General Notes](#general-notes)
- [References](#references)
- [Bugs](#bugs)
## Userspace Tools
Using `rpcdebug` is the easiest way to manipulate the kernel interfaces in place of echoing bitmasks to `/proc`.
| **Option** | **Description** |
| ----------- | --------------------------------------------- |
| `-c` | Clear the given debug flags |
| `-s` | Set the given debug flags |
| `-m module` | Specify which module's flags to set or clear. |
| `-v` | Increase the verbosity of rpcdebug's output |
| `-h` | Print a help message and exit |
| `-vh` | Print the available debug flags |
For the `-m` option, the available modules are:
| **Module** | **Description** |
| ---------- | ------------------------------------------------------------- |
| nfsd | The NFS server |
| nfs | The NFS client |
| nlm | The Network Lock Manager, in either an NFS client or server |
| rpc | The Remote Procedure Call module, either NFS client or server |
Examples:
```
rpcdebug -m rpc -s all # sets all debug flags for RPC
rpcdebug -m rpc -c all # clears all debug flags for RPC
rpcdebug -m nfsd -s all # sets all debug flags for NFS Server
rpcdebug -m nfsd -c all # clears all debug flags for NFS Server
```
## Kernel Interfaces
A bitmask of the debug flags can be echoed into the interface to enable output to syslog; 0 is the default:
```
/proc/sys/sunrpc/nfsd_debug
/proc/sys/sunrpc/nfs_debug
/proc/sys/sunrpc/nlm_debug
/proc/sys/sunrpc/rpc_debug
```
Sysctl controls are registered for these interfaces, so they can be used instead of echo:
```
sysctl -w sunrpc.rpc_debug=1023
sysctl -w sunrpc.rpc_debug=0
sysctl -w sunrpc.nfsd_debug=1023
sysctl -w sunrpc.nfsd_debug=0
```
At runtime the server holds information that can be examined:
```
grep . /proc/net/rpc/*/content
cat /proc/fs/nfs/exports
cat /proc/net/rpc/nfsd
ls -l /proc/fs/nfsd
```
A rundown of `/proc/net/rpc/nfsd` (the userspace tool `nfsstat` pretty-prints this info):
```
/proc/net/rpc/nfsd
* rc (reply cache): <hits> <misses> <nocache>
- hits: client it's retransmitting
- misses: a operation that requires caching
- nocache: a operation that no requires caching
* fh (filehandle): <stale> <total-lookups> <anonlookups> <dir-not-in-cache> <nodir-not-in-cache>
- stale: file handle errors
- total-lookups, anonlookups, dir-not-in-cache, nodir-not-in-cache
. always seem to be zeros
* io (input/output): <bytes-read> <bytes-written>
- bytes-read: bytes read directly from disk
- bytes-written: bytes written to disk
* th (threads): <threads> <fullcnt> <10%-20%> <20%-30%> ... <90%-100%> <100%>
- threads: number of nfsd threads
- fullcnt: number of times that the last 10% of threads are busy
- 10%-20%, 20%-30% ... 90%-100%: 10 numbers representing 10-20%, 20-30% to 100%
. Counts the number of times a given interval are busy
* ra (read-ahead): <cache-size> <10%> <20%> ... <100%> <not-found>
- cache-size: always the double of number threads
- 10%, 20% ... 100%: how deep it found what was looking for
- not-found: not found in the read-ahead cache
* net: <netcnt> <netudpcnt> <nettcpcnt> <nettcpconn>
- netcnt: counts every read
- netudpcnt: counts every UDP packet it receives
- nettcpcnt: counts every time it receives data from a TCP connection
- nettcpconn: count every TCP connection it receives
* rpc: <rpccnt> <rpcbadfmt+rpcbadauth+rpcbadclnt> <rpcbadfmt> <rpcbadauth> <rpcbadclnt>
- rpccnt: counts all rpc operations
- rpcbadfmt: counts if while processing a RPC it encounters the following errors:
. err_bad_dir, err_bad_rpc, err_bad_prog, err_bad_vers, err_bad_proc, err_bad
- rpcbadauth: bad authentication
. does not count if you try to mount from a machine that it's not in your exports file
- rpcbadclnt: unused
* procN (N = vers): <vs_nproc> <null> <getattr> <setattr> <lookup> <access> <readlink> <read> <write> <create> <mkdir> <symlink> <mknod> <remove> <rmdir> <rename> <link> <readdir> <readdirplus> <fsstat> <fsinfo> <pathconf> <commit>
- vs_nproc: number of procedures for NFS version
. v2: nfsproc.c, 18
. v3: nfs3proc.c, 22
- v4, nfs4proc.c, 2
- statistics: generated from NFS operations at runtime
* proc4ops: <ops> <x..y>
- ops: the definition of LAST_NFS4_OP, OP_RELEASE_LOCKOWNER = 39, plus 1 (so 40); defined in nfs4.h
- x..y: the array of nfs_opcount up to LAST_NFS4_OP (nfsdstats.nfs4_opcount[i])
```
## NFSD debug flags
```
/usr/include/linux/nfsd/debug.h (kernel 3.13.5)
/*
* knfsd debug flags
*/
#define NFSDDBG_SOCK 0x0001
#define NFSDDBG_FH 0x0002
#define NFSDDBG_EXPORT 0x0004
#define NFSDDBG_SVC 0x0008
#define NFSDDBG_PROC 0x0010
#define NFSDDBG_FILEOP 0x0020
#define NFSDDBG_AUTH 0x0040
#define NFSDDBG_REPCACHE 0x0080
#define NFSDDBG_XDR 0x0100
#define NFSDDBG_LOCKD 0x0200
#define NFSDDBG_ALL 0x7FFF
#define NFSDDBG_NOCHANGE 0xFFFF
```
## NFS debug flags
```
/usr/include/linux/nfs_fs.h (kernel 3.13.5)
/*
* NFS debug flags
*/
#define NFSDBG_VFS 0x0001
#define NFSDBG_DIRCACHE 0x0002
#define NFSDBG_LOOKUPCACHE 0x0004
#define NFSDBG_PAGECACHE 0x0008
#define NFSDBG_PROC 0x0010
#define NFSDBG_XDR 0x0020
#define NFSDBG_FILE 0x0040
#define NFSDBG_ROOT 0x0080
#define NFSDBG_CALLBACK 0x0100
#define NFSDBG_CLIENT 0x0200
#define NFSDBG_MOUNT 0x0400
#define NFSDBG_FSCACHE 0x0800
#define NFSDBG_PNFS 0x1000
#define NFSDBG_PNFS_LD 0x2000
#define NFSDBG_STATE 0x4000
#define NFSDBG_ALL 0xFFFF
```
## NLM debug flags
```
/usr/include/linux/lockd/debug.h (kernel 3.13.5)
/*
* Debug flags
*/
#define NLMDBG_SVC 0x0001
#define NLMDBG_CLIENT 0x0002
#define NLMDBG_CLNTLOCK 0x0004
#define NLMDBG_SVCLOCK 0x0008
#define NLMDBG_MONITOR 0x0010
#define NLMDBG_CLNTSUBS 0x0020
#define NLMDBG_SVCSUBS 0x0040
#define NLMDBG_HOSTCACHE 0x0080
#define NLMDBG_XDR 0x0100
#define NLMDBG_ALL 0x7fff
```
## RPC debug flags
```
/usr/include/linux/sunrpc/debug.h (kernel 3.13.5)
/*
* RPC debug facilities
*/
#define RPCDBG_XPRT 0x0001
#define RPCDBG_CALL 0x0002
#define RPCDBG_DEBUG 0x0004
#define RPCDBG_NFS 0x0008
#define RPCDBG_AUTH 0x0010
#define RPCDBG_BIND 0x0020
#define RPCDBG_SCHED 0x0040
#define RPCDBG_TRANS 0x0080
#define RPCDBG_SVCXPRT 0x0100
#define RPCDBG_SVCDSP 0x0200
#define RPCDBG_MISC 0x0400
#define RPCDBG_CACHE 0x0800
#define RPCDBG_ALL 0x7fff
```
## General Notes
- While the number of threads can be increased at runtime via an echo to `/proc/fs/nfsd/threads`, the cache size (double the threads, see the `ra` line of /proc/net/rpc/nfsd) is not dynamic. The NFS daemon must be restarted with the new thread size during initialization (`/etc/sysconfig/nfs` on RHEL and CentOS) in order for the thread cache to properly adjust.
## References
- <https://github.com/torvalds/linux/tree/master/include/linux>
- <http://linux.die.net/man/8/rpcdebug>
- <http://utcc.utoronto.ca/~cks/space/blog/linux/NFSClientDebuggingBits>
- <http://www.novell.com/support/kb/doc.php?id=7011571>
- <http://stromberg.dnsalias.org/~strombrg/NFS-troubleshooting-2.html>
- <http://www.opensubscriber.com/message/nfs@lists.sourceforge.net/7833588.html>
## Bugs
- <https://bugzilla.redhat.com/show_bug.cgi?id=465412>

1097
md/nfs_setup.md Normal file
View file

File diff suppressed because it is too large Load diff

128
md/oracle_environment.md Normal file
View file

@ -0,0 +1,128 @@
# Oracle Environment
## Contents
- [Kernel Parameters](#kernel-parameters)
- [Userspace Setup](#userspace-setup)
- [Automatic Storage Management (ASM)](#automatic-storage-management-asm)
- [Real Application Cluster (RAC)](#real-application-cluster-rac)
- [RAC Networking](#rac-networking)
- [Shared Storage](#shared-storage)
- [Virtual IP Setup](#virtual-ip-setup)
## Kernel Parameters
Oracle Global Customer Support officially recommends a maximum for SHMMAX of "1/2 of physical RAM".
The maximum size of a shared memory segment is limited by the size of the available user address space. On 64-bit systems, this is a theoretical 2<sup>64</sup>bytes. So the theoretical limit for SHMMAX is the amount of physical RAM that you have. However, to actually attempt to use such a value could potentially lead to a situation where no system memory is available for anything else. Therefore a more realistic physical limit for SHMMAX would probably be "physical RAM - 2G".
In an Oracle RDBMS application, this physical limit still leaves inadequate system memory for other necessary functions. Therefore, the common Oracle maximum for SHMMAX that you will often see is "1/2 of physical RAM". Operators may erroneously think that that setting the SHMMAX as recommended limits the total SGA, which is untrue. Setting the SHMMAX as recommended only causes a few more shared memory segments to be used for whatever total SGA that you subsequently configure in Oracle.
Modify your kernel settings in /etc/sysctl.conf as follows. If the current value for any parameter is higher than the value listed in this table, do not change the value of that parameter. Range values (such as net.ipv4.ip\_local\_port\_range) must match exactly.
```
kernel.shmall = physical RAM size / pagesize
kernel.shmmax = 1/2 of physical RAM.
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 512 x processes (for example 6815744 for 13312 processes)
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
```
Set shmall equal to the sum of all the SGAs on the system, divided by the page size. The SGA values can be calculated with a one line script:
```
# su - oracle
$ SGA=`echo "show sga"|sqlplus -s / as sysdba|grep "^Total System"|awk '{print $5}'`; PAGE=`getconf PAGE_SIZE`; echo "$SGA/$PAGE" | bc
```
## Userspace Setup
Oracle groups and user(s): normally two groups, 'oinstall' and 'dba' and one user, 'oracle'). The 'oracle' user has a hefty custom environment configured for all the variables needed. This is why we 'su - oracle' and not just a 'su oracle' when needing to run sqlplus - a full login shell with all variables initialized is required.
Add the following settings to `/etc/security/limits.conf` for the 'oracle' user (adjust as needed):
```
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
```
## Automatic Storage Management (ASM)
Oracle supplies it's own kernel module for ASM use which the DBA team will install; ASM can be summarized by three main points:
- Direct I/O to storage (bypasses kernel buffering)
- Solves 4k block size limitation of ext3
- Cluster aware filesystem on raw devices (RAC)
Unlike other kernel modules using DKMS, Oracle provides pre-compiled binaries for very specific versions of the Red Hat Enterprise kernel and can be found via their landing page below. They have a great intro article on learning more about how it actually works.
- [Oracle ASMLib Downloads](http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html)
- [Introduction to Automatic Storage Management](http://docs.oracle.com/cd/B28359_01/server.111/b31107/asmcon.htm)
## Real Application Cluster (RAC)
An Oracle RAC is typically 2+ machines in a cluster with shared storage, although it is possible to configure a single-node RAC.
### RAC Networking
Oracle RAC needs a private network between servers to use; in some cases using Jumbo Frames and/or 10G switches. The Oracle RAC nodes use this private network link for inter-node communication of large amounts of data using UDP, terabytes of both RX and TX traffic per month is not uncommon for highly active cluster nodes. It is common for the RAC nodes to use 8k UDP packets on this private network as they pass table data back and forth to stay in sync.
**Server A**
- bond0 (eth0 / eth1) - 172.16.10.5
- bond1 (eth2 / eth3) - 10.10.10.5 (RAC Interconnect)
**Server B**
- bond0 (eth0 / eth1) - 172.16.10.6
- bond1 (eth2 / eth3) - 10.10.10.6 (RAC Interconnect)
### Shared Storage
The shared storage should have a minimum of 7 LUNs presented to the servers; 5x 1G control and 2+ xxxG data. More may be used to further spread out the data for better performance; RAID-10 is the suggested design across as many spindles as possible.
- 5x 1G Raw Control LUNs
- 2x LUNs for OCR
- Oracle Cluster Registry: The OCR stores the details of the cluster configuration including the names and current status of the database, associated instances, services, and node applications such as the listener process.
- 3x LUNs for Voting
- CSS Voting Disks are used to determine which nodes are currently available within the cluster. An odd number is always used.
- 2+ xxxG ASM Data LUNs
- 1 LUN has one set of data, control, redo
- 1 LUN has one set of data, control, redo, archivelog
It is common the Server Parameter File (SPFILE) is stored on the ASM disks and there is no Flash Recovery Area (FRA) unless specifically requested. The FRA is typically twice as large as the Data LUNs.
### Virtual IP Setup
A RAC requires 5 additional IP addresses from the same subnet as the NAT IPs to be used by Oracle; they are not configured on the servers in the traditional fashion.
- The two **VIP** addresses are considered legacy for 11gR2; their use is deprecated. It's possible that an Oracle client may have an older JDBC driver that talks only "VIP" style.
- The three SCAN addresses are round-robin returned by a standard NS resolve on the client end; one node listener ("TNS LIstener") has two of the IPs configured, the other node has one IP. When an Oracle client does a NS lookup on the SCAN DNS name they connect to the TNS Listener on that IP; the TNS Listener is a load balancer and may actually communicate with another node in the cluster (so not always his local node).
- The placement of how the client resolves the listener IPs should be done in such a way that we don't insert a point of failure. For instance it's a bad idea to host the DNS record for SCAN across a site-to-site VPN link; if that link goes down then no clients can connect\! It is a best practice that the DNS record for the SCAN be hosted in such a way that it's redundant and reachable via multiple paths.
During configuration the IPs needs to be in DNS:
- Forward and reverse list the Linux server hostnames in DNS with their **primary public IPs**.
- DNS list the SCAN name with all three SCANx IP addresses; you're in effect creating a round-robin lookup in DNS for the same single-named record, not creating unique records for each IP.
```
scan01.mydomain.com IN A 172.16.30.52
scan01.mydomain.com IN A 172.16.30.53
scan01.mydomain.com IN A 172.16.30.54
```

47
md/raid_penalties.md Normal file
View file

@ -0,0 +1,47 @@
# RAID Penalties
## Contents
- [Write Penalty](#write-penalty)
- [Penalty Calculation](#penalty-calculation)
- [References](#references)
## Write Penalty
The penalty is for how we deal with the stripe (parity, etc.) where 1 means no penalty.
| **RAID Level** | **Write Penalty** |
| -------------- | ----------------- |
| 0 | 1 |
| 1 | 2 |
| 5 | 4 |
| 10 | 2 |
## Penalty Calculation
- **raw IOPS** = disk speed IOPS \* number of disks
- **functional IOPS** = (raw IOPS \* write% / write penalty) + (raw IOPS \* read%)
Given 5x 15k SAS drives @ 900 IOPS (180/disk) as an average, a sample calculation for RAID-5, with varying levels of write vs read percentage:
```
16% writes: (900*.16/4)+(900*.84) = 792 / 900 = 88.0% efficiency
10% writes: (900*.10/4)+(900*.90) = 832 / 900 = 92.4% efficiency
5% writes: (900*.05/4)+(900*.95) = 866 / 900 = 96.2% efficiency
```
Reducing the write percentage from 16% to 10% would yield an efficiency gain of 4.4%, from 16% to 5% yields an 8.2% gain. Writing to RAID5 and calculating parity is very costly - we haven't even talked about linear writes vs. scattered block writes. As an exercise, if the same 5-disk RAID-5 was converted to an 8 disk RAID-10 (to achieve the same capacity for data):
```
8*180 = 1440 raw IOPs
(1440*.16/2)+(1440*.84) = 1324 / 792 = 167% efficiency
```
This is all very basic math that doesn't take into account real world load and deals with theoretical maximums based on published standards. Use a tool such as `fio` to obtain real world performance.
## References
- <http://git.kernel.dk/?p=fio.git;a=summary>

119
md/reducing_the_root_lv.md Normal file
View file

@ -0,0 +1,119 @@
# Reducing the root LV
## Contents
- [Overview](#overview)
- [Procedure](#procedure)
- [Without lvresize -r flag](#without-lvresize--r-flag)
- [With lvresize -r flag](#with-lvresize--r-flag)
## Overview
Reducing the root logical volume requires booting into a rescue environment that has the LVM utilities.
> Some rescue images have a version of the LVM utilities that were released missing a critical shim needed to resize filesystems using the `-r` flag to `lvresize`. It may or may not be necessary to resize the filesystem as a separate step, both are outlined.
## Procedure
Given this current configuration of a 250G boot volume:
| **Mount** | **Size** | **VG / LV** |
| --------- | --------- | ---------------- |
| /boot | 250M | n/a |
| /tmp | 2G | vglocal / lvtmp |
| swap | 2G | vglocal / lvswap |
| / | remainder | vglocal / lvroot |
We'll reduce the root (/) volume and grow swap and /tmp to end up with:
| **Mount** | **Size** | **VG / LV** |
| --------- | --------- | ---------------- |
| /boot | 250M | n/a |
| /tmp | 4G | vglocal / lvtmp |
| swap | 32G | vglocal / lvswap |
| / | remainder | vglocal / lvroot |
### Without lvresize -r flag
```
##
## DO NOT MOUNT ANY FILESYSTEMS DURING RESCUE BOOT
##
# activate all LVM
lvm vgchange -a y
# fsck the filesystems; it's normal to get a message about a time error that needs fixed
fsck -fC /dev/vglocal/lvroot
fsck -fC /dev/vglocal/lvtmp
# shrink the ext3/4 root far below what we need
resize2fs -p /dev/vglocal/lvroot 200G
# reduce the root LV a bit above what we just resized (+5GB)
lvm lvresize /dev/vglocal/lvroot --size 205G
# increase swap and tmp LVs
lvm lvresize /dev/vglocal/lvswap --size 32G
lvm lvresize /dev/vglocal/lvtmp --size 4G
# re-grow the root LV back to max space
lvm lvresize -l +100%FREE /dev/vglocal/lvroot
# re-grow the / and /tmp ext3/4 to fill the increased LVs
resize2fs -p /dev/vglocal/lvroot
resize2fs -p /dev/vglocal/lvtmp
# fsck the filesystems again
fsck -fC /dev/vglocal/lvroot
fsck -fC /dev/vglocal/lvtmp
# rescue image 'mkswap' is sometimes not able to see a large swap
reboot
# finally, make a new swap signature that sees the whole LV
swapoff /dev/vglocal/lvswap
mkswap /dev/vglocal/lvswap
swapon /dev/vglocal/lvswap
```
### With lvresize -r flag
```
##
## DO NOT MOUNT ANY FILESYSTEMS DURING RESCUE BOOT
##
# activate all LVM
lvm vgchange -a y
# fsck the filesystems; it's normal to get a message about a time error that needs fixed
fsck -fC /dev/vglocal/lvroot
fsck -fC /dev/vglocal/lvtmp
# reduce the size of LV root a tad more than we need
lvm lvresize -r /dev/vglocal/lvroot --size 200G
# increase swap and tmp LVs
lvm lvresize /dev/vglocal/lvswap --size 32G
lvm lvresize /dev/vglocal/lvtmp --size 4G
# re-grow the root LV back to max space
lvm lvresize -r -l +100%FREE /dev/vglocal/lvroot
# fsck the filesystems again
fsck -fC /dev/vglocal/lvroot
fsck -fC /dev/vglocal/lvtmp
# rescue image 'mkswap' is sometimes not able to see a large swap
reboot
# finally, make a new swap signature that sees the whole LV
swapoff /dev/vglocal/lvswap
mkswap /dev/vglocal/lvswap
swapon /dev/vglocal/lvswap
```

478
md/rhcs_mechanics.md Normal file
View file

@ -0,0 +1,478 @@
# RHCS Mechanics
## Contents
- [Acronyms](#acronyms)
- [Configuration Files](#configuration-files)
- [Filesystem Locations](#filesystem-locations)
- [Operational Commands](#operational-commands)
- [Cluster Components](#cluster-components)
- [Operational Examples](#operational-examples)
- [Configuration Validation](#configuration-validation)
- [Status Check](#status-check)
- [Service Manipulation](#service-manipulation)
- [Configuration Examples](#configuration-examples)
- [Standard LVM and PgSQL Initscript](#standard-lvm-and-pgsql-initscript)
- [HA-LVM and MySQL Object](#ha-lvm-and-mysql-object)
- [Standard LVM, MySQL script and NFS](#standard-lvm-mysql-script-and-nfs)
- [HA-LVM and NFS Object](#ha-lvm-and-nfs-object)
- [References](#references)
## Acronyms
- **AIS**: Application Interface Specification
- **AMF**: Availability Management Framework
- **CCS**: Cluster Configuration System
- **CLM**: Cluster Membership
- **CLVM**: Cluster Logical Volume Manager
- **CMAN**: Cluster Manager
- **DLM**: Distributed Lock Manager
- **GFS2**: Global File System 2
- **GNDB**: Global Network Block Device
- **STONITH**: Shoot The Other Node In The Head
- **TOTEM**: Group communication algorithm for reliable group messaging among cluster members
## Configuration Files
- `/etc/cluster/cluster.conf` - The main cluster configuration file
- `/etc/lvm/lvm.conf` - The LVM configuration file - typically `locking_type` and a `filter` are being configured here
## Filesystem Locations
- `/usr/share/cluster/` - The main directory of code used for cluster objects |
- `/var/log/cluster/` - The main logging directory (**RHEL6**)
## Operational Commands
**Graphical Cluster Configuration**
- `luci` - Cluster Management Web Interface primarily used with **RHEL6**
- `system-config-cluster` - Cluster Management X11/Motif Interface primarily used with **RHEL5**
**RGManager** - Resource Group Manager
- `clustat` - Command used to display the status of the cluster, including node membership and services running
- `clusvcadm` - Command used to manually enable, disable, relocate, and restart user services in a cluster
- `rg_test` - Debug and test services and resource ordering
**CCS** - Cluster Configuration System
- `ccs_config_validate` - Verify a configuration; can validate the running config or a named file (**RHEL6**)
- `ccs_config_dump` - Tool to generate XML output of running configuration (**RHEL6**)
- `ccs_sync` - Synchronize the cluster configuration file to one or more machines in a cluster (**RHEL6**)
- `ccs_update_schema` - Update the cluster relaxng schema that validates cluster.conf (**RHEL6**)
- `ccs_test` - Diagnostic and testing command used to retrieve information from configuration files via **ccsd**
- `ccs_tool` - Used to make online updates of CCS configuration files - **considered obsolete**
**CMAN** - Cluster Manager
- `cman_tool` - The administrative front end to CMAN, starts and stops CMAN infrastructure and can perform changes
- `group_tool` - Used to get a list of groups related to fencing, DLM, GFS, and getting debug information
- `fence_XXXX` - Fence agent for XXXX type of device- for example `fence_drac` (Dell DRAC), `fence_ipmilan` (IPMI) and `fence_ilo` (HP iLO)
- `fence_check` - Test the fence configuration for each node in the cluster
- `fence_node` - A program which performs I/O fencing on a single node
- `fence_tool` - A program to join and leave the fence domain
- `dlm_tool` - Utility for the `dlm` and `dlm_controld` daemon
- `gfs_control` - Utility for the gfs\_controld daemon
**GFS2** - Global File System 2
- `mkfs.gfs2` - Creates a GFS2 file system on a storage device
- `mount.gfs2` - Mount a GFS2 file system; normally not used by the user directly
- `fsck.gfs2` - Repair an unmounted GFS2 file system
- `gfs2_grow` - Grows a mounted GFS2 file system
- `gfs2_jadd` - Adds journals to a mounted GFS2 file system
- `gfs2_quota` - Manage quotas on a mounted GFS2 file system
- `gfs2_tool` - Configures, tunes and gather information on a GFS2 file system
**Quorum Disk**
- `mkqdisk` - Cluster Quorum Disk Utility
## Cluster Components
**RGManager** - Resource Group Manager
- `rgmanager` - Daemon used to handle user service requests including service start, service disable, service relocate, and service restart; **RHEL6**
- `clurgmgrd` - Daemon used to handle user service requests including service start, service disable, service relocate, and service restart; **RHEL5**
- `cpglockd` - Utilizes the extended virtual synchrony features of Corosync to implement a simplistic, distributed lock server for rgmanager
**CLVM** - Cluster Logical Volume Manager
- `clvmd` - The daemon that distributes LVM metadata updates around a cluster. Requires `cman` to be running first
**CCS** - Cluster Configuration System
- `ricci` - CCS daemon running on all cluster nodes and provides configuration file data to cluster software; **RHEL6**
- `ccsd` - CCS daemon running on all cluster nodes and provides configuration file data to cluster software; **RHEL5**
**CMAN** - Cluster Manager
- `cman` - Cluster initscript used to start/stop all the CMAN daemons
- `corosync` - Corosync cluster communications infrastructure daemon using TOTEM; **RHEL6**
- `aisexec` - OpenAIS cluster communications infrastructure daemon using TOTEM; **RHEL5**
- `fenced` - Fences cluster nodes that have failed (fencing generally means rebooting)
- `dlm_controld` - Daemon that configures dlm according to cluster events
- `gfs_controld` - Daemon that coordinates GFS mounts and recovery
- `groupd` - Compatibility daemon for `fenced`, `dlm_controld` and `gfs_controld`
- `qdiskd` - Talks to CMAN and provides a mechanism for determining node-fitness in a cluster environment
- `cmannotifyd` - Talks to CMAN and provides a mechanism to notify external entities about cluster changes
## Operational Examples
The man pages for `clustat` and `clusvcadm` contain more in-depth explanations of all the shown options; more options exist than are shown here.
### Configuration Validation
As RHEL5 does not have the `ccs_config_validate` utility, an alternate method is possible to perform XML validation against the cluster schema instead:
```
xmllint --relaxng /usr/share/system-config-cluster/misc/cluster.ng /etc/cluster/cluster.conf
```
The well-formatted XML file and a final message about validation should be printed out when run.
### Status Check
Use the `clustat` command to check the cluster status:
```
# clustat
Cluster Status for cluster1 @ Fri Jan 17 16:49:45 2014
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
node1 1 Online, Local, rgmanager
node2 2 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:pgsql-svc node1 started
```
### Service Manipulation
Use the `clusvcadm` command to manipulate the services:
```
# Restart PostgreSQL in place on the same server
clusvcadm -R pgsql-svc
# Relocate PostgreSQL to a specific node
clusvcadm -r pgsql-svc -m <node name>
# Disable PostgreSQL
clusvcadm -d pgsql-svc
# Enable PostgreSQL
clusvcadm -e pgsql-svc
# Freeze PostgreSQL on the current node
clusvcadm -Z pgsql-svc
# Unfreeze PostgreSQL after it was frozen
clusvcadm -U pgsql-svc
```
## Configuration Examples
### Standard LVM and PgSQL Initscript
This example uses a single standard LVM mount from SAN (as opposed to HA-LVM) and a normal initscript to start the service. An IP for a secondary backup network is included as well as the Dell DRAC fencing devices on the same VLAN.
```
/etc/hosts
127.0.0.1 localhost localhost.localdomain
10.11.12.10 pgdb1.example.com pgdb1
10.11.12.11 pgdb2.example.com pgdb2
10.11.12.20 pgdb1-drac
10.11.12.21 pgdb2-drac
/etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="18" name="pgdbclus1">
<cman expected_votes="1" two_node="1"/>
<fence_daemon post_fail_delay="5" post_join_delay="15"/>
<clusternodes>
<clusternode name="pgdb1" nodeid="1">
<fence>
<method name="drac">
<device name="pgdb1-drac"/>
</method>
</fence>
</clusternode>
<clusternode name="pgdb2" nodeid="2">
<fence>
<method name="drac">
<device name="pgdb2-drac"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_drac5" cmd_prompt="admin1" ipaddr="10.11.12.20" login="root" module_name="pgdb1" name="pgdb1-drac" passwd="calvin"/>
<fencedevice agent="fence_drac5" cmd_prompt="admin1" ipaddr="10.11.12.21" login="root" module_name="pgdb2" name="pgdb2-drac" passwd="calvin"/>
</fencedevices>
<rm>
<failoverdomains>
<failoverdomain name="pgsql-fd" nofailback="1" restricted="1">
<failoverdomainnode name="pgdb1"/>
<failoverdomainnode name="pgdb2"/>
</failoverdomain>
</failoverdomains>
<resources>
<ip address="10.11.12.25" monitor_link="1" sleeptime="10"/>
<ip address="10.9.8.7" monitor_link="0" sleeptime="5"/>
<fs device="/dev/vgsan00/lvdata00" fsid="64301" fstype="ext4" mountpoint="/var/lib/pgsql/" name="pgsql-fs" options="noatime"/>
<script file="/etc/init.d/postgresql-9.3" name="pgsql-srv"/>
</resources>
<service domain="pgsql-fd" name="pgsql-svc" recovery="relocate">
<ip ref="10.11.12.25">
<fs ref="pgsql-fs">
<script ref="pgsql-srv"/>
</fs>
</ip>
<ip ref="10.9.8.7"/>
</service>
</rm>
</cluster>
```
### HA-LVM and MySQL Object
This example uses a single HA-LVM mount from SAN (activated with exclusive locks) and a RHCS-provided service object to start MySQL. An IP for a secondary backup network is included as well as the Dell DRAC fencing devices on the same VLAN.
```
/etc/hosts
127.0.0.1 localhost localhost.localdomain
10.11.12.10 mydb1.example.com mydb1
10.11.12.11 mydb2.example.com mydb2
10.11.12.20 mydb1-drac
10.11.12.21 mydb2-drac
/etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="18" name="mydbclus1">
<cman expected_votes="1" two_node="1"/>
<fence_daemon post_fail_delay="5" post_join_delay="15"/>
<clusternodes>
<clusternode name="mydb1" nodeid="1">
<fence>
<method name="drac">
<device name="mydb1-drac"/>
</method>
</fence>
</clusternode>
<clusternode name="mydb2" nodeid="2">
<fence>
<method name="drac">
<device name="mydb2-drac"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_drac5" cmd_prompt="admin1" ipaddr="10.11.12.20" login="root" module_name="mydb1" name="mydb1-drac" passwd="calvin"/>
<fencedevice agent="fence_drac5" cmd_prompt="admin1" ipaddr="10.11.12.21" login="root" module_name="mydb2" name="mydb2-drac" passwd="calvin"/>
</fencedevices>
<rm>
<failoverdomains>
<failoverdomain name="mysql-fd" nofailback="1" restricted="1">
<failoverdomainnode name="mydb1"/>
<failoverdomainnode name="mydb2"/>
</failoverdomain>
</failoverdomains>
<resources>
<ip address="10.11.12.25" monitor_link="1" sleeptime="10"/>
<ip address="10.9.8.7" monitor_link="0" sleeptime="5"/>
<lvm lv_name="data00" name="mysql-lv" vg_name="vgsan00"/>
<fs device="/dev/vgsan00/lvdata00" force_fsck="0" force_unmount="0" fsid="64301" fstype="ext4" mountpoint="/var/lib/mysql/" name="mysql-fs" options="noatime" self_fence="0"/>
<mysql config_file="/etc/my.cnf" listen_address="10.11.12.25" mysqld_options="" name="mysql" shutdown_wait="600"/>
</resources>
<service domain="mysql-fd" name="mysql-svc" recovery="relocate">
<ip ref="10.11.12.25"/>
<lvm ref="mysql-lv"/>
<fs ref="mysql-fs"/>
<mysql ref="mysql"/>
<ip ref="10.9.8.7"/>
</service>
</rm>
</cluster>
```
### Standard LVM, MySQL script and NFS
This example uses a two standard LVM mounts from DAS, a MySQL initscript and RHCS-provided NFS objects to run two services. An IP for a secondary backup network is included for each, as well as the Dell DRAC fencing devices on the same VLAN. Notice that we're also adding preferred priority (where the service will run) to ask the cluster to self-balance from a cold-start, each node will run one service.
```
/etc/hosts
127.0.0.1 localhost localhost.localdomain
10.11.12.10 node1.example.com node1
10.11.12.11 node2.example.com node2
10.11.12.20 node1-drac
10.11.12.21 node2-drac
/etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="18" name="cluster1">
<cman expected_votes="1" two_node="1"/>
<fence_daemon post_fail_delay="5" post_join_delay="15"/>
<clusternodes>
<clusternode name="node1" nodeid="1" votes="1">
<fence>
<method name="drac">
<device modulename="" name="node1-drac"/>
</method>
</fence>
</clusternode>
<clusternode name="node2" nodeid="2" votes="1">
<fence>
<method name="drac">
<device modulename="" name="node2-drac"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_drac5" cmd_prompt="admin1" ipaddr="10.11.12.20" login="root" name="node1-drac" passwd="calvin"/>
<fencedevice agent="fence_drac5" cmd_prompt="admin1" ipaddr="10.11.12.21" login="root" name="node2-drac" passwd="calvin"/>
</fencedevices>
<rm log_facility="local4" log_level="7">
<failoverdomains>
<failoverdomain name="mysql-fd" ordered="1" restricted="1">
<failoverdomainnode name="node1" priority="1"/>
<failoverdomainnode name="node2" priority="2"/>
</failoverdomain>
<failoverdomain name="nfs-fd" ordered="1" restricted="1">
<failoverdomainnode name="node1" priority="2"/>
<failoverdomainnode name="node2" priority="1"/>
</failoverdomain>
</failoverdomains>
<resources>
<ip address="10.11.12.25" monitor_link="1" sleeptime="10"/>
<ip address="10.11.12.26" monitor_link="1" sleeptime="10"/>
<ip address="10.9.8.7" monitor_link="0" sleeptime="5"/>
<ip address="10.9.8.6" monitor_link="0" sleeptime="5"/>
<fs device="/dev/vgdas00/mysql00" force_fsck="0" force_unmount="0" fsid="14404" fstype="ext3" mountpoint="/das/mysql-fs" name="mysql-fs" options="noatime" self_fence="0"/>
<fs device="/dev/vgdas01/nfs00" force_fsck="0" force_unmount="1" fsid="31490" fstype="ext3" mountpoint="/das/nfs-fs" name="nfs-fs" options="noatime" self_fence="0"/>
<script file="/etc/init.d/mysqld" name="mysql-script"/>
<nfsexport name="nfs-res"/>
<nfsclient name="nfs-export" options="rw,no_root_squash" path="/das/nfs-fs" target="10.11.12.0/24"/>
</resources>
<service autostart="1" domain="mysql-fd" name="mysql-svc">
<ip ref="10.11.12.25">
<fs ref="mysql-fs">
<script ref="mysql-script"/>
</fs>
</ip>
<ip ref="10.9.8.7"/>
</service>
<service autostart="1" domain="nfs-fd" name="nfs-svc" nfslock="1">
<ip ref="10.11.12.26">
<fs ref="nfs-fs">
<nfsexport ref="nfs-res">
<nfsclient ref="nfs-export"/>
</nfsexport>
</fs>
</ip>
<ip ref="10.9.8.6"/>
</service>
</rm>
</cluster>
```
### HA-LVM and NFS Object
This example uses a single HA-LVM mount from SAN (activated with exclusive locks) and a RHCS-provided service object to start NFS. An IP for a secondary backup network is included as well as the Dell DRAC fencing devices on the same VLAN.
```
/etc/hosts
127.0.0.1 localhost localhost.localdomain
10.11.12.10 nfs1.example.com nfs1
10.11.12.11 nfs2.example.com nfs2
10.11.12.20 nfs1-drac
10.11.12.21 nfs2-drac
/etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="18" name="nfsclus1">
<cman expected_votes="1" two_node="1"/>
<fence_daemon post_fail_delay="5" post_join_delay="15"/>
<clusternodes>
<clusternode name="nfs1" nodeid="1">
<fence>
<method name="drac">
<device name="nfs1-drac"/>
</method>
</fence>
</clusternode>
<clusternode name="nfs2" nodeid="2">
<fence>
<method name="drac">
<device name="nfs2-drac"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_drac5" cmd_prompt="admin1" ipaddr="10.11.12.20" login="root" module_name="nfs1" name="nfs1-drac" passwd="calvin"/>
<fencedevice agent="fence_drac5" cmd_prompt="admin1" ipaddr="10.11.12.21" login="root" module_name="nfs2" name="nfs2-drac" passwd="calvin"/>
</fencedevices>
<rm>
<failoverdomains>
<failoverdomain name="nfs-fd" nofailback="1" restricted="1">
<failoverdomainnode name="nfs1"/>
<failoverdomainnode name="nfs2"/>
</failoverdomain>
</failoverdomains>
<resources>
<ip address="10.11.12.25" monitor_link="1" sleeptime="10"/>
<ip address="10.9.8.7" monitor_link="0" sleeptime="5"/>
<lvm lv_name="data00" name="nfs-lv" vg_name="vgsan00"/>
<fs device="/dev/vgsan00/lvdata00" force_fsck="0" force_unmount="0" fsid="64301" fstype="ext4" mountpoint="/san/nfs-fs" name="nfs-fs" options="noatime" self_fence="0"/>
<nfsserver name="nfs-srv" nfspath=".clumanager/nfs"/>
<nfsclient allow_recover="on" name="nfsclient1" options="rw,no_root_squash,no_subtree_check" target="10.11.12.0/24"/>
</resources>
<service domain="nfs-fd" name="nfs-svc" recovery="relocate">
<lvm ref="nfs-lv"/>
<fs ref="nfs-fs">
<nfsserver ref="nfs-srv">
<ip ref="10.11.12.25"/>
<ip ref="10.9.8.7"/>
<nfsclient ref="nfsclient1"/>
</nfsserver>
</fs>
</service>
</rm>
</cluster>
```
## References
- <https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Cluster_Administration/index.html>
- <http://www.sourceware.org/cluster/conga/>
- <https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Logical_Volume_Manager_Administration/LVM_Cluster_Overview.html>
- <http://en.wikipedia.org/wiki/Fencing_%28computing%29>
- <http://en.wikipedia.org/wiki/Distributed_lock_manager>
- <http://en.wikipedia.org/wiki/STONITH>
- <http://en.wikipedia.org/wiki/Network_block_device>

673
md/rhel7_networking.md Normal file
View file

@ -0,0 +1,673 @@
# RHEL7 Networking
## Contents
- [Overview](#overview)
- [General Design](#general-design)
- [Services Required](#services-required)
- [UUID Configuration](#uuid-configuration)
- [NM\_CONTROLLED](#nm_controlled)
- [DNS Servers](#dns-servers)
- [Device Names](#device-names)
- [net.ifnames Naming](#netifnames-naming)
- [biosdevname Naming](#biosdevname-naming)
- [Example Mappings](#example-mappings)
- [Example udev rules](#example-udev-rules)
- [Using nmcli](#using-nmcli)
- [Show connections](#show-connections)
- [Configure an IP address](#configure-an-ip-address)
- [Configure an IP address with default gateway](#configure-an-ip-address-with-default-gateway)
- [Configure an additional IP address](#configure-an-additional-ip-address)
- [Add a static route](#add-a-static-route)
- [Modify DNS servers](#modify-dns-servers)
- [Activate changes](#activate-changes)
- [Reload changes](#reload-changes)
- [Delete an interface](#delete-an-interface)
- [Create new interface](#create-new-interface)
- [Well Formed Configuration](#well-formed-configuration)
- [/etc/hostname](#etchostname)
- [/etc/resolv.conf](#etcresolvconf)
- [/etc/sysconfig/network](#etcsysconfignetwork)
- [/etc/sysconfig/static-routes](#etcsysconfigstatic-routes)
- [/etc/sysconfig/network-scripts/ifcfg-eno1](#etcsysconfignetwork-scriptsifcfg-eno1)
- [/etc/sysconfig/network-scripts/ifcfg-eno2](#etcsysconfignetwork-scriptsifcfg-eno2)
- [/etc/sysconfig/network-scripts/route-eno2](#etcsysconfignetwork-scriptsroute-eno2)
- [Bonding Configuration](#bonding-configuration)
- [References](#references)
## Overview
The release of RHEL7 - and by extension CentOS7 - has changed the way core networking is managed on the system. In general the new and preferred method is to use `nmcli` to generate all configs; however it's possible to mix and match the old style and new style - sometimes with undesired results.
- <https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/>
There are now two parts to Networking:
- **NetworkManager.service** - this is a systemd unit which is started first. It will bring interfaces online if configured to do so and perform all the additional configuration work. It relies on the methodology deployed using `nmcli` commands.
- **network SysV service** - this is a traditional shim in /etc/rc3.d/ that can also be used to bring up additional networking, such as the `static-routes` file. The service uses NetworkManager internally, and also the traditional `route` command.
## General Design
The systemd unit `NetworkManager.service` is preconfigured in `/etc/NetworkManager/NetworkManager.conf` with this:
```
[main]
plugins=ifcfg-rh
```
This plugin is fully detailed in the man page `nm-settings-ifcfg-rh` which is part of the NetworkManager RPM. It provides a comprehensive set of tables of the old FOO=bar values to the new settings along with the `nmcli` man page.
1. The `/etc/sysconfig/network` file will respect a global GATEWAY and GATEWAYDEV setting, however this is not how Red Hat would like for it to be done now; they prefer the values be set as part of the interface. The file is empty by default; if it is populated with GATEWAY, any new interface created with `nmcli` will inherit and use this `GATEWAY` in the new config file.
2. Determining the default route methodology: <https://mail.gnome.org/archives/networkmanager-list/2014-July/msg00080.html>
```
NM automatically sets the default route based on two things:
(1) interface priority - all interfaces have a priority and if two
interfaces are active, and *not* prevented from getting the default
route (see #2), the one with the highest priority wins. Right now,
that's a static ordering but we're exploring how to make that dynamic.
(2) the "never-default" option: you can prevent connections (and thus
their interface when that connection is active) from ever getting the
default route by setting this option. With nmcli:
nmcli c mod eth0 ipv4.never-default true
nmcli c mod eth0 ipv6.never-default true
would prevent connection "eth0" from ever receiving the IPv4 or IPv6
default route. This would allow some other connection/interface to
receive the default route, when active.
```
3. Unlike earlier releases, leave `NM_CONTROLLED` alone and do not set it to 'no'; the NetworkManager service runs along with the traditional `/etc/init.d/network` script
4. The `route`, `netstat` and other tools part of RPM `net-tools` is not installed on a default Minimal installation; the `ifenslave` binary is now part of RPM `iputils`, so no longer a unique RPM
5. The new `ipv4.ignore-auto-dns` config is what causes NM to write out the `DNS=` lines from `ifcfg-xxx` to `/etc/resolv.conf`
Additionally, Red Hat's online documentation and the man pages have a few important design features to understand:
- The default gateway is determined by the network scripts which parse the `/etc/sysconfig/network` file first and then the network interface ifcfg files for interfaces that are "up". The ifcfg files are parsed in numerically ascending order, and the last `GATEWAY` directive to be read is used to compose a default route in the routing table.
- The default route can thus be indicated by means of the `GATEWAY` directive and can be specified either globally or in interface-specific configuration files. Specifying the gateway globally has certain advantages in static networking environments, especially if more than one network interface is present. It can make fault finding simpler if applied consistently.
- Each NetworkManager connection maps to one `ifcfg-*` file, with possible usage of `keys-*` for passwords, `route-*` for static IPv4 routes and `route6-*` for static IPv6 routes.
- The `UUID` values in the config files must be unique. You can use `uuidgen` command line tool to generate such values. Alternatively, you can leave out UUID entirely. In that case NetworkManager will generate a UUID based on the file name.
Other important concepts are outlined beyond what's presented above for other types of networking (bridges, etc.) - check the man pages for further information.
### Services Required
The `NetworkManager.service` systemd unit should be enabled, and the `network` SysV initscript should be enabled for full network collaboration between traditional methods and the new style.
```
systemctl enable NetworkManager.service
chkconfig network on
```
### UUID Configuration
The `nmcli` tool will automatically generate the UUID for an interface when created; if copying a config from one server to another, simply comment out the existing UUID and NetworkManager will dynamically assign a new one when brought online. Do not bring up two interfaces with the same UUID - it is the same hazard as using the same IP on two interfaces.
### NM\_CONTROLLED
In previous releases it was common to add or set the `NM_CONTROLLED` variable to "no" - do not do this in RHEL7. This setting should be either not in the config (the default) or set to yes if it needs to be in the config file. It is recommended to not use this setting on RHEL7 and let the system self-manage.
### DNS Servers
The standard way for DNS to be configured is using `nmcli` and the server IPs are inside the `ifcfg-*` files; when the interface is brought online NetworkManager will add/remove the values in `/etc/resolv.conf`. To disable this behaviour and use traditional static DNS configuration, the property `.ignore-auto-dns` is manipulated to `yes`:
```
# nmcli con mod eno1 ipv4.ignore-auto-dns yes ipv6.ignore-auto-dns yes
# nmcli -f ipv4.ignore-auto-dns,ipv6.ignore-auto-dns con show eno1
ipv4.ignore-auto-dns: yes
ipv6.ignore-auto-dns: yes
# grep PEERDNS /etc/sysconfig/network-scripts/ifcfg-eno1
PEERDNS=no
IPV6_PEERDNS=no
```
Changing the `nmcli` property method sets the `PEERDNS` variable, they are one in the same result.
## Device Names
There are 3 possible ways a network device name will appear in RHEL7, depending on the kernel parameters used at boot (grub2):
1. With no parameters or `net.ifnames=1` used, the names will use **systemd** [Predictable Network Interface Names](http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/)
2. With `biosdevname=1` used, the names will be managed by **udev** (technically systemd-udev) using [Consistent Network Device Naming](https://en.wikipedia.org/wiki/Consistent_Network_Device_Naming)
3. With `net.ifnames=0` and `biosdevname=0` both specified, traditional ("kernel names") _ethX_ names will be used; if udev rules are also added they can rename the interfaces to anything except _ethX_
> When method \#3 is used, the renaming from _ethX_ to _ethY_ (using udev rules or otherwise) is not supported by Red Hat in RHEL7. There was a [commit to systemd code in 2012](https://github.com/systemd/systemd/commit/97595710b77aa162ca5e20da57d0a1ed7355eaad) that removed the ability to shuffle _ethX_ kernel names properly, resulting in errors on boot if it is attempted with RHEL7. Renaming kernel names is not a feature provided (or declared) by RHEL7.
### net.ifnames Naming
Two character prefixes [based on the type of interface](http://cgit.freedesktop.org/systemd/systemd/tree/src/udev/udev-builtin-net_id.c):
| **Prefix** | **Device Topology** |
| ---------- | --------------------- |
| **en** | ethernet |
| **sl** | serial line IP (slip) |
| **wl** | wlan |
| **ww** | wwan |
Type specifics following the Prefix:
| **Designators** | **Type** |
| ----------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- |
| **b**\<*number*\> | BCMA bus core number |
| **ccw**\<*name*\> | CCW bus group name |
| **o**\<*index*\>\[**d**\<*dev\_port*\>\] | on-board device index number |
| **s**\<*slot*\>\[**f**\<*function*\>\]\[**d**\<*dev\_port*\>\] | hotplug slot index number |
| **x**\<*MAC*\> | MAC address |
| \[**P**\<*domain*\>\]**p**\<*bus*\>**s**\<*slot*\>\[**f**\<*function*\>\]\[**d**\<*dev\_port*\>\] | PCI geographical location |
| \[**P**\<*domain*\>\]**p**\<*bus*\>**s**\<*slot*\>\[**f**\<*function*\>\]\[**u**\<*port*\>\]\[..\]\[**c**\<*config*\>\]\[**i**\<*interface*\>\] | USB port number chain |
- All multi-function PCI devices will carry the \[**f**\<*function*\>\] number in the device name, including the function 0 device.
- When using PCI geography, The PCI domain is only prepended when it is not 0.
- For USB devices the full chain of port numbers of hubs is composed. If the name gets longer than the maximum number of 15 characters, the name is not exported.
- The usual USB `configuration == 1` and `interface == 0` values are suppressed., including the function 0 device.
### biosdevname Naming
The [naming convention](https://domsch.com/blog/?p=455) is as follows:
| **Convention** | **Device Type** |
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **em**\[1-*N*\] | on-board (embedded) NICs (\# matches chassis labels) |
| **p**\<*slot\_number*\>**p**\<*port\_number*\> | cards in PCI slots, ports starting at 1 (not zero) |
| **\_**\<*vf*\> suffix | NPAR and SR-IOV devices from 0..*N* depending on the number of Partitions or Virtual Functions exposed on each port |
| **.**\<*vlan*\> and **:**\<*alias*\> suffixes | Other Linux conventions remain unchanged and are still applicable |
The algorithm:
- If system BIOS exposes the new PCI Firmware Specification 3.1 ACPI\_DSM method, we get the interface label and index from ACPI, and use those.
- Else if system BIOS exposes an index and label in SMBIOS 2.6 types 9 and 41, use the index value.
- Else if system BIOS exposes index via the HP proprietary SMBIOS extension, use that.
- Else fall back to using the legacy PCI IRQ Routing Table to figure out which slots devices are in, sort the PCI device list in breadth-first order, and assign index values.
### Example Mappings
Given a HP DL380 G9 chassis with 4x onboard, 4x offboard, 2x 10G onboard (FlexiLOM) and 2x 10G offboard (Brocade) NICs this is what the name relationship looks like in it's natural state (no udev rules in place or specific `HWADDR=` lines in the `ifcfg-*` files):
| **Device** | **MAC Address** | **Traditional** | **net.ifnames=1** | **biosdevname=1** |
| --------------- | ------------------- | --------------- | ----------------- | ----------------- |
| HP 1G port 1 | `38:63:bb:44:89:e4` | eth0 | eno1 | em1 |
| HP 1G port 2 | `38:63:bb:44:89:e5` | eth1 | eno2 | em2 |
| HP 1G port 3 | `38:63:bb:44:89:e6` | eth3 | eno3 | em3 |
| HP 1G port 4 | `38:63:bb:44:89:e7` | eth4 | eno4 | em4 |
| Intel 1G port 1 | `a0:36:9f:63:2a:dc` | eth2 | ens6f0 | p6p1 |
| Intel 1G port 2 | `a0:36:9f:63:2a:dd` | eth5 | ens6f1 | p6p2 |
| Intel 1G port 3 | `a0:36:9f:63:2a:de` | eth6 | ens6f2 | p6p3 |
| Intel 1G port 4 | `a0:36:9f:63:2a:df` | eth7 | ens6f3 | p6p4 |
| HP 10G port 1 | `8c:dc:d4:a9:ea:74` | eth8 | eno49 | em49 |
| HP 10G port 2 | `8c:dc:d4:a9:ea:75` | eth10 | eno50 | em50 |
| Brocade 10G \#1 | `00:24:ff:a2:a7:00` | eth9 | ens1f0 | p1p1 |
| Brocade 10G \#2 | `00:24:ff:a2:ab:00` | eth11 | ens4f0 | p4p1 |
The `net.ifnames=1` systemd derived names are a mixture of base-0 and base-1 names that does not happen with udev derived names which are all base-1. The use of `biosdevname=1` creates the most consistently named device interfaces regardless if they are onboard or offboard NICs, all using a base-1 for the first port name as shown above.
### Example udev rules
Using the same MAC addresses from the table above, an example of renaming all the network interfaces with _netX_ as the pattern when booting (grub2) with `net.ifnames=0 biosdevname=0` kernel parameters - these names will match the `/etc/sysconfig/network-scripts/ifcfg-netX` configuration files that need to be created as well.
```
/etc/udev/rules.d/70-persistent-net.rules
# Onboard quad-port (HP)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="38:63:bb:44:89:e4", ATTR{type}=="1", KERNEL=="eth*", NAME="net0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="38:63:bb:44:89:e5", ATTR{type}=="1", KERNEL=="eth*", NAME="net1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="38:63:bb:44:89:e6", ATTR{type}=="1", KERNEL=="eth*", NAME="net2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="38:63:bb:44:89:e7", ATTR{type}=="1", KERNEL=="eth*", NAME="net3"
# Offboard quad-port (Intel)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="a0:36:9f:63:2a:dc", ATTR{type}=="1", KERNEL=="eth*", NAME="net4"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="a0:36:9f:63:2a:dd", ATTR{type}=="1", KERNEL=="eth*", NAME="net5"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="a0:36:9f:63:2a:de", ATTR{type}=="1", KERNEL=="eth*", NAME="net6"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="a0:36:9f:63:2a:df", ATTR{type}=="1", KERNEL=="eth*", NAME="net7"
# Onboard 10G 2-port (HP FlexiLOM)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:a9:ea:74", ATTR{type}=="1", KERNEL=="eth*", NAME="net8"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:a9:ea:75", ATTR{type}=="1", KERNEL=="eth*", NAME="net9"
# Offboard 10G 2x cards (Brocade HBAs)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:24:ff:a2:a7:00", ATTR{type}=="1", KERNEL=="eth*", NAME="net10"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:24:ff:a2:ab:00", ATTR{type}=="1", KERNEL=="eth*", NAME="net11"
```
## Using nmcli
Remember: Tab-completion is your friend\! Additionally, the `nmcli` command will allow you to shorten the commands to a minimal level as long as it's unique; for example these are all functionally equivalent to get the same information:
```
nmcli connection show eno1
nmcli con show eno1
nmcli c s eno1
```
### Show connections
```
nmcli connection show
nmcli connection show eno1
```
### Configure an IP address
```
nmcli connection modify eno1 ipv4.addresses 192.168.0.5/24
```
### Configure an IP address with default gateway
```
nmcli connection modify eno1 ipv4.addresses 192.168.0.5/24 \
ipv4.gateway 192.168.0.1 ipv4.never-default no
```
### Configure an additional IP address
```
nmcli connection modify eno1 +ipv4.addresses 192.168.0.6
```
### Add a static route
These will be saved in the route-\<interface\> file in `/etc/sysconfig/network-scripts/` :
```
nmcli connection modify eno1 +ipv4.routes '192.168.111.0/24 192.168.99.1'
```
### Modify DNS servers
```
# replace - notice no "+"
nmcli connection modify eno1 ipv4.dns '172.20.20.8,172.20.20.9'
# add - notice "+"
nmcli connection modify eno1 +ipv4.dns '192.168.100.123'
# remove - notice "-"
nmcli connection modify eno1 -ipv4.dns '192.168.100.123'
# also the search domains
nmcli connection modify eno1 +ipv4.dns-search 'mydomain.com'
```
### Activate changes
Logically in the same area as "ifup" and "ifdown" which also work, as these commands are NetworkManager aware in RHEL7.
```
nmcli connection up eno1
```
### Reload changes
Used when manually editing the configuration file and you need to update the daemon's cache info:
```
nmcli connection reload eno2
```
### Delete an interface
This will erase the config file entirely, make a backup first if needed\!
```
nmcli connection delete eno2
```
### Create new interface
Note the use of `ipv4.never-default yes ipv4.ignore-auto-dns yes` – this prevents this interface from ever being the default route, and prevents it from updating DNS in `/etc/resolv.conf`. These would be set to "no" for a primary interface where you do want to specify a DNS, gateway and make it the default.
```
# add the primary connection itself -- the IPv4 parameters are not allowed with "add"
nmcli con add autoconnect yes type ethernet con-name eno2 ifname eno2
# now configure the specifics for this interface
nmcli con mod eno2 ipv4.method manual \
ipv4.addresses 10.128.161.98/27 \
ipv4.routes "10.191.192.0/18 10.128.161.97" \
ipv4.never-default yes ipv4.ignore-auto-dns yes ipv6.method link-local
```
## Well Formed Configuration
Based on the Red Hat default methodology a few items are different that previous versions.
### /etc/hostname
The hostname is now part of standard systemd configuration and is managed via `hostnamectl` command; it is no longer configured in the networking config files.
### /etc/resolv.conf
This acts as in the traditional manner, however it's content is managed by the NetworkManager infrastructure. Specify the DNS information in the primary interface config and let NetworkManager update this file as needed, or alter the interface configuration properties as outlined above to disable NetworkManager from updating it for traditional static behaviour.
### /etc/sysconfig/network
By default this file should be empty - but it is possible to specify `GATEWAY` and `GATEWAYDEV` in this file as before.
> Any gateway values present in this file will automatically get added as the gateway for new interfaces created (eno2, etc.) - leave it blank for best results.
### /etc/sysconfig/static-routes
By default this file should be empty - but it is possible to specify routes as before. It is recommended to instead use `route-eth?` files instead.
### /etc/sysconfig/network-scripts/ifcfg-eno1
An example of a `nmcli` based well-formed example of this config file; note the `DOMAIN`, `DEFROUTE` and others.
```
/etc/sysconfig/network-scripts/ifcfg-eno1
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=eno1
UUID=90363eef-bd9c-4b1e-86d6-b6893f9aa04d
DEVICE=eno1
ONBOOT=yes
IPADDR=172.20.20.135
PREFIX=22
GATEWAY=172.20.20.1
DNS1=172.20.20.8
DNS2=172.20.20.9
DOMAIN=mydomain.com
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_PRIVACY=no
```
The interface has other properties seen via `nmcli` con show eno1 style commands:
```
\# nmcli con show eno1
connection.id: eno1
connection.uuid: 90363eef-bd9c-4b1e-86d6-b6893f9aa04d
connection.interface-name: eno1
connection.type: 802-3-ethernet
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1431380830
connection.read-only: no
connection.permissions:
connection.zone: --
connection.master: --
connection.slave-type: --
connection.secondaries:
connection.gateway-ping-timeout: 0
802-3-ethernet.port: --
802-3-ethernet.speed: 0
802-3-ethernet.duplex: --
802-3-ethernet.auto-negotiate: yes
802-3-ethernet.mac-address: --
802-3-ethernet.cloned-mac-address: --
802-3-ethernet.mac-address-blacklist:
802-3-ethernet.mtu: auto
802-3-ethernet.s390-subchannels:
802-3-ethernet.s390-nettype: --
802-3-ethernet.s390-options:
ipv4.method: manual
ipv4.dns: 172.20.20.8,172.20.20.9
ipv4.dns-search: mydomain.com
ipv4.addresses: 172.20.20.135/22
ipv4.gateway: 172.20.20.1
ipv4.routes:
ipv4.route-metric: -1
ipv4.ignore-auto-routes: no
ipv4.ignore-auto-dns: no
ipv4.dhcp-client-id: --
ipv4.dhcp-send-hostname: yes
ipv4.dhcp-hostname: --
ipv4.never-default: no
ipv4.may-fail: yes
ipv6.method: auto
ipv6.dns:
ipv6.dns-search:
ipv6.addresses:
ipv6.gateway: --
ipv6.routes:
ipv6.route-metric: -1
ipv6.ignore-auto-routes: no
ipv6.ignore-auto-dns: no
ipv6.never-default: no
ipv6.may-fail: yes
ipv6.ip6-privacy: 0 (disabled)
ipv6.dhcp-send-hostname: yes
ipv6.dhcp-hostname: --
GENERAL.NAME: eno1
GENERAL.UUID: 90363eef-bd9c-4b1e-86d6-b6893f9aa04d
GENERAL.DEVICES: eno1
GENERAL.STATE: activated
GENERAL.DEFAULT: yes
GENERAL.DEFAULT6: no
GENERAL.VPN: no
GENERAL.ZONE: --
GENERAL.DBUS-PATH: /org/freedesktop/NetworkManager/ActiveConnection/0
GENERAL.CON-PATH: /org/freedesktop/NetworkManager/Settings/0
GENERAL.SPEC-OBJECT: /
GENERAL.MASTER-PATH: --
IP4.ADDRESS[1]: 172.20.20.135/22
IP4.GATEWAY: 172.20.20.1
IP4.DNS[1]: 172.20.20.8
IP4.DNS[2]: 172.20.20.9
IP6.ADDRESS[1]: fe80::3a63:bbff:fe40:f708/64
IP6.GATEWAY:
```
### /etc/sysconfig/network-scripts/ifcfg-eno2
A secondary interface that doesn't route; for example the commands to create an interface:
```
nmcli con add autoconnect yes type ethernet con-name eno2 ifname eno2
nmcli con mod eno2 ipv4.method manual \
ipv4.addresses 10.128.161.98/27 \
ipv4.routes "10.191.192.0/18 10.128.161.97" \
ipv4.never-default yes ipv4.ignore-auto-dns yes ipv6.method link-local
```
...results in this `ifcfg-eno2`:
```
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
NAME=eno2
UUID=a92aedd0-3dae-4648-85c8-1e4aac835146
DEVICE=eno2
ONBOOT=yes
DOMAIN=mydomain.com
IPADDR=10.128.161.98
PREFIX=27
IPV6_PRIVACY=no
```
...and the full `nmcli con show eno2`:
```
connection.id: eno2
connection.uuid: a92aedd0-3dae-4648-85c8-1e4aac835146
connection.interface-name: eno2
connection.type: 802-3-ethernet
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1431381130
connection.read-only: no
connection.permissions:
connection.zone: --
connection.master: --
connection.slave-type: --
connection.secondaries:
connection.gateway-ping-timeout: 0
802-3-ethernet.port: --
802-3-ethernet.speed: 0
802-3-ethernet.duplex: --
802-3-ethernet.auto-negotiate: yes
802-3-ethernet.mac-address: --
802-3-ethernet.cloned-mac-address: --
802-3-ethernet.mac-address-blacklist:
802-3-ethernet.mtu: auto
802-3-ethernet.s390-subchannels:
802-3-ethernet.s390-nettype: --
802-3-ethernet.s390-options:
ipv4.method: manual
ipv4.dns:
ipv4.dns-search: mydomain.com
ipv4.addresses: 10.128.161.98/27
ipv4.gateway: --
ipv4.routes: { ip = 10.191.192.0/18, nh = 10.128.161.97 }
ipv4.route-metric: -1
ipv4.ignore-auto-routes: no
ipv4.ignore-auto-dns: yes
ipv4.dhcp-client-id: --
ipv4.dhcp-send-hostname: yes
ipv4.dhcp-hostname: --
ipv4.never-default: yes
ipv4.may-fail: yes
ipv6.method: link-local
ipv6.dns:
ipv6.dns-search:
ipv6.addresses:
ipv6.gateway: --
ipv6.routes:
ipv6.route-metric: -1
ipv6.ignore-auto-routes: yes
ipv6.ignore-auto-dns: yes
ipv6.never-default: yes
ipv6.may-fail: yes
ipv6.ip6-privacy: 0 (disabled)
ipv6.dhcp-send-hostname: yes
ipv6.dhcp-hostname: --
GENERAL.NAME: eno2
GENERAL.UUID: a92aedd0-3dae-4648-85c8-1e4aac835146
GENERAL.DEVICES: eno2
GENERAL.STATE: activated
GENERAL.DEFAULT: no
GENERAL.DEFAULT6: no
GENERAL.VPN: no
GENERAL.ZONE: --
GENERAL.DBUS-PATH: /org/freedesktop/NetworkManager/ActiveConnection/4
GENERAL.CON-PATH: /org/freedesktop/NetworkManager/Settings/5
GENERAL.SPEC-OBJECT: /
GENERAL.MASTER-PATH: --
IP4.ADDRESS[1]: 10.128.161.98/27
IP4.GATEWAY:
IP4.ROUTE[1]: dst = 10.191.192.0/18, nh = 10.128.161.97, mt = 100
IP6.ADDRESS[1]: fe80::3a63:bbff:fe40:f709/64
IP6.GATEWAY:
```
### /etc/sysconfig/network-scripts/route-eno2
This file accepts the old style and new style at the same time which is unique for a config file.
If using `nmcli` with the `NetworkManager.service` systemd unit it will have a format like so:
```
ADDRESS0=10.191.192.0
NETMASK0=255.255.192.0
GATEWAY0=10.128.161.97
```
...however it has a traditional format for route via the network SysV script:
```
10.191.192.0/18 via 10.128.161.97
```
The `nmcli` method is preferred for full compatibility.
## Bonding Configuration
Given two interfaces on the same VLAN - eno1 and eno2 used here - the information can be pulled from the running eno1 and a bond0 build from it using a small amount of scripting; this is a proof of concept.
```
#!/bin/bash
#
# Create a bond interface on RHEL7 / CentOS7 using pure nmcli methods
# on a fresh installation (or vanilla networking) that was built using
# NetworkManager. IPv4 only.
#
# Required: first parameter passed is the fully configured interface with
# IP(s)/CIDR, Gateway and DNS; these values will be extracted and applied
# to the new bond0 and set as a default route
#
# BACK UP YOUR CONFIGS FIRST
if [ $# -ne 3 ]; then
echo "Usage: $0 <bond name> <1st slave> <2nd slave>"
echo "Example: $0 bond0 eno1 eno2"
exit 1
fi
BOND=$1
SLAVE1=$2
SLAVE2=$3
# Build the basic bond in HA mode (most common)
nmcli con add autoconnect yes type bond con-name ${BOND} ifname ${BOND} \
mode active-backup miimon 100
# Set a high priority to keep gateway as the default route
nmcli con mod ${BOND} connection.autoconnect-priority 99 \
ipv6.method link-local
# Pull existing IPv4
IPV4=($(nmcli -f \
ipv4.method,ipv4.addresses,ipv4.gateway,ipv4.dns,ipv4.dns-search \
con show ${SLAVE1} | awk '{print $2}'))
# Configure the bond
nmcli con mod ${BOND} ipv4.method ${IPV4[0]} \
ipv4.addresses ${IPV4[1]} ipv4.gateway ${IPV4[2]} \
ipv4.dns ${IPV4[3]} ipv4.dns-search ${IPV4[4]} \
ipv4.never-default no ipv4.ignore-auto-dns no
# Remove then add the interfaces as slaves
nmcli con del ${SLAVE1}
nmcli con add autoconnect yes type bond-slave \
con-name ${SLAVE1} ifname ${SLAVE1} master ${BOND}
nmcli con del ${SLAVE2}
nmcli con add autoconnect yes type bond-slave \
con-name ${SLAVE2} ifname ${SLAVE2} master ${BOND}
exit 0
```
## References
- <https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/>
- <https://github.com/sivel/bonding/issues/12>
- <https://mail.gnome.org/archives/networkmanager-list/2014-July/msg00080.html>
- <http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/>
- <http://cgit.freedesktop.org/systemd/systemd/tree/src/udev/udev-builtin-net_id.c>
- <https://en.wikipedia.org/wiki/Consistent_Network_Device_Naming>
- <https://domsch.com/blog/?p=455>
- <https://fedoraproject.org/wiki/Features/ConsistentNetworkDeviceNaming>
- <https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/appe-Consistent_Network_Device_Naming.html>

679
md/scsi_sense_data.md Normal file
View file

@ -0,0 +1,679 @@
# SCSI Sense Data
## Contents
- [Response Codes](#response-codes)
- [SCSI Sense Key](#scsi-sense-key)
- [ASC and ASCQ](#asc-and-ascq)
- [References](#references)
- [Source Credit](#source-credit)
## Response Codes
### 0x70 and 0x71 sense data format
| byte / bit | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
| ---------- | ------------------------------- | ------------------------------- | ------------------------------- | ------------------------------- | ------------------------------- | ------------------------------- | ------------------------------- | ------------------------------- |
| 0 | Valid | Response code (0x70 or 0x71) | Response code (0x70 or 0x71) | Response code (0x70 or 0x71) | Response code (0x70 or 0x71) | Response code (0x70 or 0x71) | Response code (0x70 or 0x71) | Response code (0x70 or 0x71) |
| 1 | Segment number | Segment number | Segment number | Segment number | Segment number | Segment number | Segment number | Segment number |
| 2 | Filemark | EOM | ILI | Reserved | Sense key | Sense key | Sense key | Sense key |
| 3 - 6 | Information | Information | Information | Information | Information | Information | Information | Information |
| 7 | Additional sense length | Additional sense length | Additional sense length | Additional sense length | Additional sense length | Additional sense length | Additional sense length | Additional sense length |
| 8 - 11 | Command-specific information | Command-specific information | Command-specific information | Command-specific information | Command-specific information | Command-specific information | Command-specific information | Command-specific information |
| 12 | Additional sense code | Additional sense code | Additional sense code | Additional sense code | Additional sense code | Additional sense code | Additional sense code | Additional sense code |
| 13 | Additional sense code qualifier | Additional sense code qualifier | Additional sense code qualifier | Additional sense code qualifier | Additional sense code qualifier | Additional sense code qualifier | Additional sense code qualifier | Additional sense code qualifier |
| 14 | Field replaceable unit code | Field replaceable unit code | Field replaceable unit code | Field replaceable unit code | Field replaceable unit code | Field replaceable unit code | Field replaceable unit code | Field replaceable unit code |
| 15 - 17 | Sense-key specific | Sense-key specific | Sense-key specific | Sense-key specific | Sense-key specific | Sense-key specific | Sense-key specific | Sense-key specific |
| 18 - *n* | Additional sense bytes | Additional sense bytes | Additional sense bytes | Additional sense bytes | Additional sense bytes | Additional sense bytes | Additional sense bytes | Additional sense bytes |
## SCSI Sense Key
| Sense Key | Short Description | Long Description |
| --------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0x00 | NO SENSE | Indicates that there is no specific sense key information to be reported. This may occur for a successful command or for a command that receives CHECK CONDITION status because one of the FILEMARK, EOM, or ILI bits is set to one. |
| 0x01 | RECOVERED ERROR | Indicates that the command completed successfully, with some recovery action performed by the device server. Details may be determined by examining the additional sense bytes and the INFORMATION field. When multiple recovered errors occur during one command, the choice of which error to report (e.g., first, last, most severe) is vendor specific. |
| 0x02 | NOT READY | Indicates that the logical unit is not accessible. Operator intervention may be required to correct this condition. |
| 0x03 | MEDIUM ERROR | Indicates that the command terminated with a non-recovered error condition that may have been caused by a flaw in the medium or an error in the recorded data. This sense key may also be returned if the device server is unable to distinguish between a flaw in the medium and a specific hardware failure (i.e., sense key 4h). |
| 0x04 | HARDWARE ERROR | Indicates that the device server detected a non-recoverable hardware failure (e.g., controller failure, device failure, or parity error) while performing the command or during a self test. |
| 0x05 | ILLEGAL REQUEST | Indicates that: (1) The command was addressed to an incorrect logical unit number (see SAM-4); (2) The command had an invalid task attribute (see SAM-4); (3) The command was addressed to a logical unit whose current configuration prohibits processing the command; (4) There was an illegal parameter in the CDB; or (5) There was an illegal parameter in the additional parameters supplied as data for some commands (e.g., PERSISTENT RESERVE OUT). If the device server detects an invalid parameter in the CDB, it shall terminate the command without altering the medium. If the device server detects an invalid parameter in the additional parameters supplied as data, the device server may have already altered the medium. |
| 0x06 | UNIT ATTENTION | Indicates that a unit attention condition has been established (e.g., the removable medium may have been changed, a logical unit reset occurred). See SAM-4. |
| 0x07 | DATA PROTECT | Indicates that a command that reads or writes the medium was attempted on a block that is protected. The read or write operation is not performed. |
| 0x08 | BLANK CHECK | Indicates that a write-once device or a sequential-access device encountered blank medium or format-defined end-of-data indication while reading or that a write-once device encountered a non-blank medium while writing. |
| 0x09 | VENDOR SPECIFIC | This sense key is available for reporting vendor specific conditions. |
| 0x0a | COPY ABORTED | Indicates an EXTENDED COPY command was aborted due to an error condition on the source device, the destination device, or both (see 6.3.3). |
| 0x0b | ABORTED COMMAND | Indicates that the device server aborted the command. The application client may be able to recover by trying the command again. |
| 0x0c | obsolete | |
| 0x0d | VOLUME OVERFLOW | Indicates that a buffered SCSI device has reached the end-of-partition and data may remain in the buffer that has not been written to the medium. One or more RECOVER BUFFERED DATA command(s) may be issued to read the unwritten data from the buffer. (See SSC-2.) |
| 0x0e | MISCOMPARE | Indicates that the source data did not match the data read from the medium. |
| 0x0f | reserved | |
## ASC and ASCQ
| ASC | ASCQ | D | T | L | P | W | R | O | M | A | E | B | K | V | F | Description |
| ---- | ---- |:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| --------------------------------------------------------------- |
| 0x00 | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | NO ADDITIONAL SENSE INFORMATION |
| 0x00 | 0x01 | | T | | | | | | | | | | | | | FILEMARK DETECTED |
| 0x00 | 0x02 | | T | | | | | | | | | | | | | END-OF-PARTITION/MEDIUM DETECTED |
| 0x00 | 0x03 | | T | | | | | | | | | | | | | SETMARK DETECTED |
| 0x00 | 0x04 | | T | | | | | | | | | | | | | BEGINNING-OF-PARTITION/MEDIUM DETECTED |
| 0x00 | 0x05 | | T | L | | | | | | | | | | | | END-OF-DATA DETECTED |
| 0x00 | 0x06 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | I/O PROCESS TERMINATED |
| 0x00 | 0x11 | | | | | | R | | | | | | | | | AUDIO PLAY OPERATION IN PROGRESS |
| 0x00 | 0x12 | | | | | | R | | | | | | | | | AUDIO PLAY OPERATION PAUSED |
| 0x00 | 0x13 | | | | | | R | | | | | | | | | AUDIO PLAY OPERATION SUCCESSFULLY COMPLETED |
| 0x00 | 0x14 | | | | | | R | | | | | | | | | AUDIO PLAY OPERATION STOPPED DUE TO ERROR |
| 0x00 | 0x15 | | | | | | R | | | | | | | | | NO CURRENT AUDIO STATUS TO RETURN |
| 0x00 | 0x16 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | OPERATION IN PROGRESS |
| 0x00 | 0x17 | D | T | L | | W | R | O | M | A | E | B | K | V | F | CLEANING REQUESTED |
| 0x00 | 0x18 | | T | | | | | | | | | | | | | ERASE OPERATION IN PROGRESS |
| 0x00 | 0x19 | | T | | | | | | | | | | | | | LOCATE OPERATION IN PROGRESS |
| 0x00 | 0x1A | | T | | | | | | | | | | | | | REWIND OPERATION IN PROGRESS |
| 0x00 | 0x1B | | T | | | | | | | | | | | | | SET CAPACITY OPERATION IN PROGRESS |
| 0x00 | 0x1C | | T | | | | | | | | | | | | | VERIFY OPERATION IN PROGRESS |
| 0x00 | 0x1D | D | T | | | | | | | | | B | | | | ATA PASS THROUGH INFORMATION AVAILABLE |
| 0x01 | 0x00 | D | | | | W | | O | | | | B | K | | | NO INDEX/SECTOR SIGNAL |
| 0x02 | 0x00 | D | | | | W | R | O | M | | | B | K | | | NO SEEK COMPLETE |
| 0x03 | 0x00 | D | T | L | | W | | O | | | | B | K | | | PERIPHERAL DEVICE WRITE FAULT |
| 0x03 | 0x01 | | T | | | | | | | | | | | | | NO WRITE CURRENT |
| 0x03 | 0x02 | | T | | | | | | | | | | | | | EXCESSIVE WRITE ERRORS |
| 0x04 | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT NOT READY, CAUSE NOT REPORTABLE |
| 0x04 | 0x01 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT IS IN PROCESS OF BECOMING READY |
| 0x04 | 0x02 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT NOT READY, INITIALIZING COMMAND REQUIRED |
| 0x04 | 0x03 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT NOT READY, MANUAL INTERVENTION REQUIRED |
| 0x04 | 0x04 | D | T | L | | | R | O | | | | B | | | | LOGICAL UNIT NOT READY, FORMAT IN PROGRESS |
| 0x04 | 0x05 | D | T | | | W | | O | M | A | | B | K | | | LOGICAL UNIT NOT READY, REBUILD IN PROGRESS |
| 0x04 | 0x06 | D | T | | | W | | O | M | A | | B | K | | | LOGICAL UNIT NOT READY, RECALCULATION IN PROGRESS |
| 0x04 | 0x07 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT NOT READY, OPERATION IN PROGRESS |
| 0x04 | 0x08 | | | | | | R | | | | | | | | | LOGICAL UNIT NOT READY, LONG WRITE IN PROGRESS |
| 0x04 | 0x09 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT NOT READY, SELF-TEST IN PROGRESS |
| 0x04 | 0x0A | D | T | L | P | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT NOT ACCESSIBLE, ASYMMETRIC ACCESS STATE TRANSITION |
| 0x04 | 0x0B | D | T | L | P | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT NOT ACCESSIBLE, TARGET PORT IN STANDBY STATE |
| 0x04 | 0x0C | D | T | L | P | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT NOT ACCESSIBLE, TARGET PORT IN UNAVAILABLE STATE |
| 0x04 | 0x10 | D | T | | | W | R | O | M | | | B | | | | LOGICAL UNIT NOT READY, AUXILIARY MEMORY NOT ACCESSIBLE |
| 0x04 | 0x11 | D | T | | | W | R | O | M | A | E | B | | V | F | LOGICAL UNIT NOT READY, NOTIFY (ENABLE SPINUP) REQUIRED |
| 0x04 | 0x12 | | | | | | | | M | | | | | V | | LOGICAL UNIT NOT READY, OFFLINE |
| 0x05 | 0x00 | D | T | L | | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT DOES NOT RESPOND TO SELECTION |
| 0x06 | 0x00 | D | | | | W | R | O | M | | | B | K | | | NO REFERENCE POSITION FOUND |
| 0x07 | 0x00 | D | T | L | | W | R | O | M | | | B | K | | | MULTIPLE PERIPHERAL DEVICES SELECTED |
| 0x08 | 0x00 | D | T | L | | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT COMMUNICATION FAILURE |
| 0x08 | 0x01 | D | T | L | | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT COMMUNICATION TIME-OUT |
| 0x08 | 0x02 | D | T | L | | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT COMMUNICATION PARITY ERROR |
| 0x08 | 0x03 | D | T | | | | R | O | M | | | B | K | | | LOGICAL UNIT COMMUNICATION CRC ERROR (ULTRA-DMA/32) |
| 0x08 | 0x04 | D | T | L | P | W | R | O | | | | | K | | | UNREACHABLE COPY TARGET |
| 0x09 | 0x00 | D | T | | | W | R | O | | | | B | | | | TRACK FOLLOWING ERROR |
| 0x09 | 0x01 | | | | | W | R | O | | | | | K | | | TRACKING SERVO FAILURE |
| 0x09 | 0x02 | | | | | W | R | O | | | | | K | | | FOCUS SERVO FAILURE |
| 0x09 | 0x03 | | | | | W | R | O | | | | | | | | SPINDLE SERVO FAILURE |
| 0x09 | 0x04 | D | T | | | W | R | O | | | | B | | | | HEAD SELECT FAULT |
| 0x0A | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | ERROR LOG OVERFLOW |
| 0x0B | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | WARNING |
| 0x0B | 0x01 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | WARNING - SPECIFIED TEMPERATURE EXCEEDED |
| 0x0B | 0x02 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | WARNING - ENCLOSURE DEGRADED |
| 0x0B | 0x03 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | WARNING - BACKGROUND SELF-TEST FAILED |
| 0x0B | 0x04 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | WARNING - BACKGROUND PRE-SCAN DETECTED MEDIUM ERROR |
| 0x0B | 0x05 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | WARNING - BACKGROUND MEDIUM SCAN DETECTED MEDIUM ERROR |
| 0x0C | 0x00 | | T | | | | R | | | | | | | | | WRITE ERROR |
| 0x0C | 0x01 | | | | | | | | | | | | K | | | WRITE ERROR - RECOVERED WITH AUTO REALLOCATION |
| 0x0C | 0x02 | D | | | | W | | O | | | | B | K | | | WRITE ERROR - AUTO REALLOCATION FAILED |
| 0x0C | 0x03 | D | | | | W | | O | | | | B | K | | | WRITE ERROR - RECOMMEND REASSIGNMENT |
| 0x0C | 0x04 | D | T | | | W | | O | | | | B | | | | COMPRESSION CHECK MISCOMPARE ERROR |
| 0x0C | 0x05 | D | T | | | W | | O | | | | B | | | | DATA EXPANSION OCCURRED DURING COMPRESSION |
| 0x0C | 0x06 | D | T | | | W | | O | | | | B | | | | BLOCK NOT COMPRESSIBLE |
| 0x0C | 0x07 | | | | | | R | | | | | | | | | WRITE ERROR - RECOVERY NEEDED |
| 0x0C | 0x08 | | | | | | R | | | | | | | | | WRITE ERROR - RECOVERY FAILED |
| 0x0C | 0x09 | | | | | | R | | | | | | | | | WRITE ERROR - LOSS OF STREAMING |
| 0x0C | 0x0A | | | | | | R | | | | | | | | | WRITE ERROR - PADDING BLOCKS ADDED |
| 0x0C | 0x0B | D | T | | | W | R | O | M | | | B | | | | AUXILIARY MEMORY WRITE ERROR |
| 0x0C | 0x0C | D | T | L | P | W | R | O | M | A | E | B | K | V | F | WRITE ERROR - UNEXPECTED UNSOLICITED DATA |
| 0x0C | 0x0D | D | T | L | P | W | R | O | M | A | E | B | K | V | F | WRITE ERROR - NOT ENOUGH UNSOLICITED DATA |
| 0x0C | 0x0F | | | | | | R | | | | | | | | | DEFECTS IN ERROR WINDOW |
| 0x0D | 0x00 | D | T | L | P | W | R | O | | A | | | K | | | ERROR DETECTED BY THIRD PARTY TEMPORARY INITIATOR |
| 0x0D | 0x01 | D | T | L | P | W | R | O | | A | | | K | | | THIRD PARTY DEVICE FAILURE |
| 0x0D | 0x02 | D | T | L | P | W | R | O | | A | | | K | | | COPY TARGET DEVICE NOT REACHABLE |
| 0x0D | 0x03 | D | T | L | P | W | R | O | | A | | | K | | | INCORRECT COPY TARGET DEVICE TYPE |
| 0x0D | 0x04 | D | T | L | P | W | R | O | | A | | | K | | | COPY TARGET DEVICE DATA UNDERRUN |
| 0x0D | 0x05 | D | T | L | P | W | R | O | | A | | | K | | | COPY TARGET DEVICE DATA OVERRUN |
| 0x0E | 0x00 | D | T | | P | W | R | O | M | A | E | B | K | | F | INVALID INFORMATION UNIT |
| 0x0E | 0x01 | D | T | | P | W | R | O | M | A | E | B | K | | F | INFORMATION UNIT TOO SHORT |
| 0x0E | 0x02 | D | T | | P | W | R | O | M | A | E | B | K | | F | INFORMATION UNIT TOO LONG |
| 0x0E | 0x03 | D | T | | P | | R | | M | A | E | B | K | | F | INVALID FIELD IN COMMAND INFORMATION UNIT |
| 0x0F | 0x00 | | | | | | | | | | | | | | | |
| 0x10 | 0x00 | D | | | | W | | O | | | | B | K | | | ID CRC OR ECC ERROR |
| 0x10 | 0x01 | D | T | | | W | | O | | | | | | | | LOGICAL BLOCK GUARD CHECK FAILED |
| 0x10 | 0x02 | D | T | | | W | | O | | | | | | | | LOGICAL BLOCK APPLICATION TAG CHECK FAILED |
| 0x10 | 0x03 | D | T | | | W | | O | | | | | | | | LOGICAL BLOCK REFERENCE TAG CHECK FAILED |
| 0x11 | 0x00 | D | T | | | W | R | O | | | | B | K | | | UNRECOVERED READ ERROR |
| 0x11 | 0x01 | D | T | | | W | R | O | | | | B | K | | | READ RETRIES EXHAUSTED |
| 0x11 | 0x02 | D | T | | | W | R | O | | | | B | K | | | ERROR TOO LONG TO CORRECT |
| 0x11 | 0x03 | D | T | | | W | | O | | | | B | K | | | MULTIPLE READ ERRORS |
| 0x11 | 0x04 | D | | | | W | | O | | | | B | K | | | UNRECOVERED READ ERROR - AUTO REALLOCATE FAILED |
| 0x11 | 0x05 | | | | | W | R | O | | | | B | | | | L-EC UNCORRECTABLE ERROR |
| 0x11 | 0x06 | | | | | W | R | O | | | | B | | | | CIRC UNRECOVERED ERROR |
| 0x11 | 0x07 | | | | | W | | O | | | | B | | | | DATA RE-SYNCHRONIZATION ERROR |
| 0x11 | 0x08 | | T | | | | | | | | | | | | | INCOMPLETE BLOCK READ |
| 0x11 | 0x09 | | T | | | | | | | | | | | | | NO GAP FOUND |
| 0x11 | 0x0A | D | T | | | | | O | | | | B | K | | | MISCORRECTED ERROR |
| 0x11 | 0x0B | D | | | | W | | O | | | | B | K | | | UNRECOVERED READ ERROR - RECOMMEND REASSIGNMENT |
| 0x11 | 0x0C | D | | | | W | | O | | | | B | K | | | UNRECOVERED READ ERROR - RECOMMEND REWRITE THE DATA |
| 0x11 | 0x0D | D | T | | | W | R | O | | | | B | | | | DE-COMPRESSION CRC ERROR |
| 0x11 | 0x0E | D | T | | | W | R | O | | | | B | | | | CANNOT DECOMPRESS USING DECLARED ALGORITHM |
| 0x11 | 0x0F | | | | | | R | | | | | | | | | ERROR READING UPC/EAN NUMBER |
| 0x11 | 0x10 | | | | | | R | | | | | | | | | ERROR READING ISRC NUMBER |
| 0x11 | 0x11 | | | | | | R | | | | | | | | | READ ERROR - LOSS OF STREAMING |
| 0x11 | 0x12 | D | T | | | W | R | O | M | | | B | | | | AUXILIARY MEMORY READ ERROR |
| 0x11 | 0x13 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | READ ERROR - FAILED RETRANSMISSION REQUEST |
| 0x11 | 0x14 | D | | | | | | | | | | | | | | READ ERROR - LBA MARKED BAD BY APPLICATION CLIENT |
| 0x12 | 0x00 | D | | | | W | | O | | | | B | K | | | ADDRESS MARK NOT FOUND FOR ID FIELD |
| 0x13 | 0x00 | D | | | | W | | O | | | | B | K | | | ADDRESS MARK NOT FOUND FOR DATA FIELD |
| 0x14 | 0x00 | D | T | L | | W | R | O | | | | B | K | | | RECORDED ENTITY NOT FOUND |
| 0x14 | 0x01 | D | T | | | W | R | O | | | | B | K | | | RECORD NOT FOUND |
| 0x14 | 0x02 | | T | | | | | | | | | | | | | FILEMARK OR SETMARK NOT FOUND |
| 0x14 | 0x03 | | T | | | | | | | | | | | | | END-OF-DATA NOT FOUND |
| 0x14 | 0x04 | | T | | | | | | | | | | | | | BLOCK SEQUENCE ERROR |
| 0x14 | 0x05 | D | T | | | W | | O | | | | B | K | | | RECORD NOT FOUND - RECOMMEND REASSIGNMENT |
| 0x14 | 0x06 | D | T | | | W | | O | | | | B | K | | | RECORD NOT FOUND - DATA AUTO-REALLOCATED |
| 0x14 | 0x07 | | T | | | | | | | | | | | | | LOCATE OPERATION FAILURE |
| 0x15 | 0x00 | D | T | L | | W | R | O | M | | | B | K | | | RANDOM POSITIONING ERROR |
| 0x15 | 0x01 | D | T | L | | W | R | O | M | | | B | K | | | MECHANICAL POSITIONING ERROR |
| 0x15 | 0x02 | D | T | | | W | R | O | | | | B | K | | | POSITIONING ERROR DETECTED BY READ OF MEDIUM |
| 0x16 | 0x00 | D | | | | W | | O | | | | B | K | | | DATA SYNCHRONIZATION MARK ERROR |
| 0x16 | 0x01 | D | | | | W | | O | | | | B | K | | | DATA SYNC ERROR - DATA REWRITTEN |
| 0x16 | 0x02 | D | | | | W | | O | | | | B | K | | | DATA SYNC ERROR - RECOMMEND REWRITE |
| 0x16 | 0x03 | D | | | | W | | O | | | | B | K | | | DATA SYNC ERROR - DATA AUTO-REALLOCATED |
| 0x16 | 0x04 | D | | | | W | | O | | | | B | K | | | DATA SYNC ERROR - RECOMMEND REASSIGNMENT |
| 0x17 | 0x00 | D | T | | | W | R | O | | | | B | K | | | RECOVERED DATA WITH NO ERROR CORRECTION APPLIED |
| 0x17 | 0x01 | D | T | | | W | R | O | | | | B | K | | | RECOVERED DATA WITH RETRIES |
| 0x17 | 0x02 | D | T | | | W | R | O | | | | B | K | | | RECOVERED DATA WITH POSITIVE HEAD OFFSET |
| 0x17 | 0x03 | D | T | | | W | R | O | | | | B | K | | | RECOVERED DATA WITH NEGATIVE HEAD OFFSET |
| 0x17 | 0x04 | | | | | W | R | O | | | | B | | | | RECOVERED DATA WITH RETRIES AND/OR CIRC APPLIED |
| 0x17 | 0x05 | D | | | | W | R | O | | | | B | K | | | RECOVERED DATA USING PREVIOUS SECTOR ID |
| 0x17 | 0x06 | D | | | | W | | O | | | | B | K | | | RECOVERED DATA WITHOUT ECC - DATA AUTO-REALLOCATED |
| 0x17 | 0x07 | D | | | | W | R | O | | | | B | K | | | RECOVERED DATA WITHOUT ECC - RECOMMEND REASSIGNMENT |
| 0x17 | 0x08 | D | | | | W | R | O | | | | B | K | | | RECOVERED DATA WITHOUT ECC - RECOMMEND REWRITE |
| 0x17 | 0x09 | D | | | | W | R | O | | | | B | K | | | RECOVERED DATA WITHOUT ECC - DATA REWRITTEN |
| 0x18 | 0x00 | D | T | | | W | R | O | | | | B | K | | | RECOVERED DATA WITH ERROR CORRECTION APPLIED |
| 0x18 | 0x01 | D | | | | W | R | O | | | | B | K | | | RECOVERED DATA WITH ERROR CORR. & RETRIES APPLIED |
| 0x18 | 0x02 | D | | | | W | R | O | | | | B | K | | | RECOVERED DATA - DATA AUTO-REALLOCATED |
| 0x18 | 0x03 | | | | | | R | | | | | | | | | RECOVERED DATA WITH CIRC |
| 0x18 | 0x04 | | | | | | R | | | | | | | | | RECOVERED DATA WITH L-EC |
| 0x18 | 0x05 | D | | | | W | R | O | | | | B | K | | | RECOVERED DATA - RECOMMEND REASSIGNMENT |
| 0x18 | 0x06 | D | | | | W | R | O | | | | B | K | | | RECOVERED DATA - RECOMMEND REWRITE |
| 0x18 | 0x07 | D | | | | W | | O | | | | B | K | | | RECOVERED DATA WITH ECC - DATA REWRITTEN |
| 0x18 | 0x08 | | | | | | R | | | | | | | | | RECOVERED DATA WITH LINKING |
| 0x19 | 0x00 | D | | | | | | O | | | | | K | | | DEFECT LIST ERROR |
| 0x19 | 0x01 | D | | | | | | O | | | | | K | | | DEFECT LIST NOT AVAILABLE |
| 0x19 | 0x02 | D | | | | | | O | | | | | K | | | DEFECT LIST ERROR IN PRIMARY LIST |
| 0x19 | 0x03 | D | | | | | | O | | | | | K | | | DEFECT LIST ERROR IN GROWN LIST |
| 0x1A | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | PARAMETER LIST LENGTH ERROR |
| 0x1B | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | SYNCHRONOUS DATA TRANSFER ERROR |
| 0x1C | 0x00 | D | | | | | | O | | | | B | K | | | DEFECT LIST NOT FOUND |
| 0x1C | 0x01 | D | | | | | | O | | | | B | K | | | PRIMARY DEFECT LIST NOT FOUND |
| 0x1C | 0x02 | D | | | | | | O | | | | B | K | | | GROWN DEFECT LIST NOT FOUND |
| 0x1D | 0x00 | D | T | | | W | R | O | | | | B | K | | | MISCOMPARE DURING VERIFY OPERATION |
| 0x1E | 0x00 | D | | | | W | | O | | | | B | K | | | RECOVERED ID WITH ECC CORRECTION |
| 0x1F | 0x00 | D | | | | | | O | | | | | K | | | PARTIAL DEFECT LIST TRANSFER |
| 0x20 | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | INVALID COMMAND OPERATION CODE |
| 0x20 | 0x01 | D | T | | P | W | R | O | M | A | E | B | K | | | ACCESS DENIED - INITIATOR PENDING-ENROLLED |
| 0x20 | 0x02 | D | T | | P | W | R | O | M | A | E | B | K | | | ACCESS DENIED - NO ACCESS RIGHTS |
| 0x20 | 0x03 | D | T | | P | W | R | O | M | A | E | B | K | | | ACCESS DENIED - INVALID MGMT ID KEY |
| 0x20 | 0x04 | | T | | | | | | | | | | | | | ILLEGAL COMMAND WHILE IN WRITE CAPABLE STATE |
| 0x20 | 0x05 | | T | | | | | | | | | | | | | Obsolete |
| 0x20 | 0x06 | | T | | | | | | | | | | | | | ILLEGAL COMMAND WHILE IN EXPLICIT ADDRESS MODE |
| 0x20 | 0x07 | | T | | | | | | | | | | | | | ILLEGAL COMMAND WHILE IN IMPLICIT ADDRESS MODE |
| 0x20 | 0x08 | D | T | | P | W | R | O | M | A | E | B | K | | | ACCESS DENIED - ENROLLMENT CONFLICT |
| 0x20 | 0x09 | D | T | | P | W | R | O | M | A | E | B | K | | | ACCESS DENIED - INVALID LU IDENTIFIER |
| 0x20 | 0x0A | D | T | | P | W | R | O | M | A | E | B | K | | | ACCESS DENIED - INVALID PROXY TOKEN |
| 0x20 | 0x0B | D | T | | P | W | R | O | M | A | E | B | K | | | ACCESS DENIED - ACL LUN CONFLICT |
| 0x21 | 0x00 | D | T | | | W | R | O | M | | | B | K | | | LOGICAL BLOCK ADDRESS OUT OF RANGE |
| 0x21 | 0x01 | D | T | | | W | R | O | M | | | B | K | | | INVALID ELEMENT ADDRESS |
| 0x21 | 0x02 | | | | | | R | | | | | | | | | INVALID ADDRESS FOR WRITE |
| 0x21 | 0x03 | | | | | | R | | | | | | | | | INVALID WRITE CROSSING LAYER JUMP |
| 0x22 | 0x00 | D | | | | | | | | | | | | | | ILLEGAL FUNCTION (USE 20 00, 24 00, OR 26 00) |
| 0x23 | 0x00 | | | | | | | | | | | | | | | |
| 0x24 | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | INVALID FIELD IN CDB |
| 0x24 | 0x01 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | CDB DECRYPTION ERROR |
| 0x24 | 0x02 | | T | | | | | | | | | | | | | Obsolete |
| 0x24 | 0x03 | | T | | | | | | | | | | | | | Obsolete |
| 0x24 | 0x04 | | | | | | | | | | | | | | F | SECURITY AUDIT VALUE FROZEN |
| 0x24 | 0x05 | | | | | | | | | | | | | | F | SECURITY WORKING KEY FROZEN |
| 0x24 | 0x06 | | | | | | | | | | | | | | F | NONCE NOT UNIQUE |
| 0x24 | 0x07 | | | | | | | | | | | | | | F | NONCE TIMESTAMP OUT OF RANGE |
| 0x25 | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT NOT SUPPORTED |
| 0x26 | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | INVALID FIELD IN PARAMETER LIST |
| 0x26 | 0x01 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | PARAMETER NOT SUPPORTED |
| 0x26 | 0x02 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | PARAMETER VALUE INVALID |
| 0x26 | 0x03 | D | T | L | P | W | R | O | M | A | E | | K | | | THRESHOLD PARAMETERS NOT SUPPORTED |
| 0x26 | 0x04 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | INVALID RELEASE OF PERSISTENT RESERVATION |
| 0x26 | 0x05 | D | T | L | P | W | R | O | M | A | | B | K | | | DATA DECRYPTION ERROR |
| 0x26 | 0x06 | D | T | L | P | W | R | O | | | | | K | | | TOO MANY TARGET DESCRIPTORS |
| 0x26 | 0x07 | D | T | L | P | W | R | O | | | | | K | | | UNSUPPORTED TARGET DESCRIPTOR TYPE CODE |
| 0x26 | 0x08 | D | T | L | P | W | R | O | | | | | K | | | TOO MANY SEGMENT DESCRIPTORS |
| 0x26 | 0x09 | D | T | L | P | W | R | O | | | | | K | | | UNSUPPORTED SEGMENT DESCRIPTOR TYPE CODE |
| 0x26 | 0x0A | D | T | L | P | W | R | O | | | | | K | | | UNEXPECTED INEXACT SEGMENT |
| 0x26 | 0x0B | D | T | L | P | W | R | O | | | | | K | | | INLINE DATA LENGTH EXCEEDED |
| 0x26 | 0x0C | D | T | L | P | W | R | O | | | | | K | | | INVALID OPERATION FOR COPY SOURCE OR DESTINATION |
| 0x26 | 0x0D | D | T | L | P | W | R | O | | | | | K | | | COPY SEGMENT GRANULARITY VIOLATION |
| 0x26 | 0x0E | D | T | | P | W | R | O | M | A | E | B | K | | | INVALID PARAMETER WHILE PORT IS ENABLED |
| 0x26 | 0x0F | | | | | | | | | | | | | | F | INVALID DATA-OUT BUFFER INTEGRITY CHECK VALUE |
| 0x26 | 0x10 | | T | | | | | | | | | | | | | DATA DECRYPTION KEY FAIL LIMIT REACHED |
| 0x26 | 0x11 | | T | | | | | | | | | | | | | INCOMPLETE KEY-ASSOCIATED DATA SET |
| 0x26 | 0x12 | | T | | | | | | | | | | | | | VENDOR SPECIFIC KEY REFERENCE NOT FOUND |
| 0x27 | 0x00 | D | T | | | W | R | O | | | | B | K | | | WRITE PROTECTED |
| 0x27 | 0x01 | D | T | | | W | R | O | | | | B | K | | | HARDWARE WRITE PROTECTED |
| 0x27 | 0x02 | D | T | | | W | R | O | | | | B | K | | | LOGICAL UNIT SOFTWARE WRITE PROTECTED |
| 0x27 | 0x03 | | T | | | | R | | | | | | | | | ASSOCIATED WRITE PROTECT |
| 0x27 | 0x04 | | T | | | | R | | | | | | | | | PERSISTENT WRITE PROTECT |
| 0x27 | 0x05 | | T | | | | R | | | | | | | | | PERMANENT WRITE PROTECT |
| 0x27 | 0x06 | | | | | | R | | | | | | | | | CONDITIONAL WRITE PROTECT |
| 0x28 | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | NOT READY TO READY CHANGE, MEDIUM MAY HAVE CHANGED |
| 0x28 | 0x01 | D | T | | | W | R | O | M | | | B | | | | IMPORT OR EXPORT ELEMENT ACCESSED |
| 0x28 | 0x02 | | | | | | R | | | | | | | | | FORMAT-LAYER MAY HAVE CHANGED |
| 0x29 | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | POWER ON, RESET, OR BUS DEVICE RESET OCCURRED |
| 0x29 | 0x01 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | POWER ON OCCURRED |
| 0x29 | 0x02 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | SCSI BUS RESET OCCURRED |
| 0x29 | 0x03 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | BUS DEVICE RESET FUNCTION OCCURRED |
| 0x29 | 0x04 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | DEVICE INTERNAL RESET |
| 0x29 | 0x05 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | TRANSCEIVER MODE CHANGED TO SINGLE-ENDED |
| 0x29 | 0x06 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | TRANSCEIVER MODE CHANGED TO LVD |
| 0x29 | 0x07 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | I\_T NEXUS LOSS OCCURRED |
| 0x2A | 0x00 | D | T | L | | W | R | O | M | A | E | B | K | V | F | PARAMETERS CHANGED |
| 0x2A | 0x01 | D | T | L | | W | R | O | M | A | E | B | K | V | F | MODE PARAMETERS CHANGED |
| 0x2A | 0x02 | D | T | L | | W | R | O | M | A | E | | K | | | LOG PARAMETERS CHANGED |
| 0x2A | 0x03 | D | T | L | P | W | R | O | M | A | E | | K | | | RESERVATIONS PREEMPTED |
| 0x2A | 0x04 | D | T | L | P | W | R | O | M | A | E | | | | | RESERVATIONS RELEASED |
| 0x2A | 0x05 | D | T | L | P | W | R | O | M | A | E | | | | | REGISTRATIONS PREEMPTED |
| 0x2A | 0x06 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | ASYMMETRIC ACCESS STATE CHANGED |
| 0x2A | 0x07 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | IMPLICIT ASYMMETRIC ACCESS STATE TRANSITION FAILED |
| 0x2A | 0x08 | D | T | | | W | R | O | M | A | E | B | K | V | F | PRIORITY CHANGED |
| 0x2A | 0x09 | D | | | | | | | | | | | | | | CAPACITY DATA HAS CHANGED |
| 0x2A | 0x10 | D | T | | | | | | M | | E | | | V | | TIMESTAMP CHANGED |
| 0x2A | 0x11 | | T | | | | | | | | | | | | | DATA ENCRYPTION PARAMETERS CHANGED BY ANOTHER I\_T NEXUS |
| 0x2A | 0x12 | | T | | | | | | | | | | | | | DATA ENCRYPTION PARAMETERS CHANGED BY VENDOR SPECIFIC EVENT |
| 0x2A | 0x13 | | T | | | | | | | | | | | | | DATA ENCRYPTION KEY INSTANCE COUNTER HAS CHANGED |
| 0x2B | 0x00 | D | T | L | P | W | R | O | | | | | K | | | COPY CANNOT EXECUTE SINCE HOST CANNOT DISCONNECT |
| 0x2C | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | COMMAND SEQUENCE ERROR |
| 0x2C | 0x01 | | | | | | | | | | | | | | | TOO MANY WINDOWS SPECIFIED |
| 0x2C | 0x02 | | | | | | | | | | | | | | | INVALID COMBINATION OF WINDOWS SPECIFIED |
| 0x2C | 0x03 | | | | | | R | | | | | | | | | CURRENT PROGRAM AREA IS NOT EMPTY |
| 0x2C | 0x04 | | | | | | R | | | | | | | | | CURRENT PROGRAM AREA IS EMPTY |
| 0x2C | 0x05 | | | | | | | | | | | B | | | | ILLEGAL POWER CONDITION REQUEST |
| 0x2C | 0x06 | | | | | | R | | | | | | | | | PERSISTENT PREVENT CONFLICT |
| 0x2C | 0x07 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | PREVIOUS BUSY STATUS |
| 0x2C | 0x08 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | PREVIOUS TASK SET FULL STATUS |
| 0x2C | 0x09 | D | T | L | P | W | R | O | M | | E | B | K | V | F | PREVIOUS RESERVATION CONFLICT STATUS |
| 0x2C | 0x0A | | | | | | | | | | | | | | F | PARTITION OR COLLECTION CONTAINS USER OBJECTS |
| 0x2C | 0x0B | | T | | | | | | | | | | | | | NOT RESERVED |
| 0x2D | 0x00 | | T | | | | | | | | | | | | | OVERWRITE ERROR ON UPDATE IN PLACE |
| 0x2E | 0x00 | | | | | | R | | | | | | | | | INSUFFICIENT TIME FOR OPERATION |
| 0x2F | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | COMMANDS CLEARED BY ANOTHER INITIATOR |
| 0x2F | 0x01 | D | | | | | | | | | | | | | | COMMANDS CLEARED BY POWER LOSS NOTIFICATION |
| 0x2F | 0x02 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | COMMANDS CLEARED BY DEVICE SERVER |
| 0x30 | 0x00 | D | T | | | W | R | O | M | | | B | K | | | INCOMPATIBLE MEDIUM INSTALLED |
| 0x30 | 0x01 | D | T | | | W | R | O | | | | B | K | | | CANNOT READ MEDIUM - UNKNOWN FORMAT |
| 0x30 | 0x02 | D | T | | | W | R | O | | | | B | K | | | CANNOT READ MEDIUM - INCOMPATIBLE FORMAT |
| 0x30 | 0x03 | D | T | | | | R | | | | | | K | | | CLEANING CARTRIDGE INSTALLED |
| 0x30 | 0x04 | D | T | | | W | R | O | | | | B | K | | | CANNOT WRITE MEDIUM - UNKNOWN FORMAT |
| 0x30 | 0x05 | D | T | | | W | R | O | | | | B | K | | | CANNOT WRITE MEDIUM - INCOMPATIBLE FORMAT |
| 0x30 | 0x06 | D | T | | | W | R | O | | | | B | | | | CANNOT FORMAT MEDIUM - INCOMPATIBLE MEDIUM |
| 0x30 | 0x07 | D | T | L | | W | R | O | M | A | E | B | K | V | F | CLEANING FAILURE |
| 0x30 | 0x08 | | | | | | R | | | | | | | | | CANNOT WRITE - APPLICATION CODE MISMATCH |
| 0x30 | 0x09 | | | | | | R | | | | | | | | | CURRENT SESSION NOT FIXATED FOR APPEND |
| 0x30 | 0x0A | D | T | | | W | R | O | M | A | E | B | K | | | CLEANING REQUEST REJECTED |
| 0x30 | 0x0C | | T | | | | | | | | | | | | | WORM MEDIUM - OVERWRITE ATTEMPTED |
| 0x30 | 0x0D | | T | | | | | | | | | | | | | WORM MEDIUM - INTEGRITY CHECK |
| 0x30 | 0x10 | | | | | | R | | | | | | | | | MEDIUM NOT FORMATTED |
| 0x31 | 0x00 | D | T | | | W | R | O | | | | B | K | | | MEDIUM FORMAT CORRUPTED |
| 0x31 | 0x01 | D | | L | | | R | O | | | | B | | | | FORMAT COMMAND FAILED |
| 0x31 | 0x02 | | | | | | R | | | | | | | | | ZONED FORMATTING FAILED DUE TO SPARE LINKING |
| 0x32 | 0x00 | D | | | | W | | O | | | | B | K | | | NO DEFECT SPARE LOCATION AVAILABLE |
| 0x32 | 0x01 | D | | | | W | | O | | | | B | K | | | DEFECT LIST UPDATE FAILURE |
| 0x33 | 0x00 | | T | | | | | | | | | | | | | TAPE LENGTH ERROR |
| 0x34 | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | ENCLOSURE FAILURE |
| 0x35 | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | ENCLOSURE SERVICES FAILURE |
| 0x35 | 0x01 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | UNSUPPORTED ENCLOSURE FUNCTION |
| 0x35 | 0x02 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | ENCLOSURE SERVICES UNAVAILABLE |
| 0x35 | 0x03 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | ENCLOSURE SERVICES TRANSFER FAILURE |
| 0x35 | 0x04 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | ENCLOSURE SERVICES TRANSFER REFUSED |
| 0x35 | 0x05 | D | T | L | | W | R | O | M | A | E | B | K | V | F | ENCLOSURE SERVICES CHECKSUM ERROR |
| 0x36 | 0x00 | | | L | | | | | | | | | | | | RIBBON, INK, OR TONER FAILURE |
| 0x37 | 0x00 | D | T | L | | W | R | O | M | A | E | B | K | V | F | ROUNDED PARAMETER |
| 0x38 | 0x00 | | | | | | | | | | | B | | | | EVENT STATUS NOTIFICATION |
| 0x38 | 0x02 | | | | | | | | | | | B | | | | ESN - POWER MANAGEMENT CLASS EVENT |
| 0x38 | 0x04 | | | | | | | | | | | B | | | | ESN - MEDIA CLASS EVENT |
| 0x38 | 0x06 | | | | | | | | | | | B | | | | ESN - DEVICE BUSY CLASS EVENT |
| 0x39 | 0x00 | D | T | L | | W | R | O | M | A | E | | K | | | SAVING PARAMETERS NOT SUPPORTED |
| 0x3A | 0x00 | D | T | L | | W | R | O | M | | | B | K | | | MEDIUM NOT PRESENT |
| 0x3A | 0x01 | D | T | | | W | R | O | M | | | B | K | | | MEDIUM NOT PRESENT - TRAY CLOSED |
| 0x3A | 0x02 | D | T | | | W | R | O | M | | | B | K | | | MEDIUM NOT PRESENT - TRAY OPEN |
| 0x3A | 0x03 | D | T | | | W | R | O | M | | | B | | | | MEDIUM NOT PRESENT - LOADABLE |
| 0x3A | 0x04 | D | T | | | W | R | O | M | | | B | | | | MEDIUM NOT PRESENT - MEDIUM AUXILIARY MEMORY ACCESSIBLE |
| 0x3B | 0x00 | | T | L | | | | | | | | | | | | SEQUENTIAL POSITIONING ERROR |
| 0x3B | 0x01 | | T | | | | | | | | | | | | | TAPE POSITION ERROR AT BEGINNING-OF-MEDIUM |
| 0x3B | 0x02 | | T | | | | | | | | | | | | | TAPE POSITION ERROR AT END-OF-MEDIUM |
| 0x3B | 0x03 | | | L | | | | | | | | | | | | TAPE OR ELECTRONIC VERTICAL FORMS UNIT NOT READY |
| 0x3B | 0x04 | | | L | | | | | | | | | | | | SLEW FAILURE |
| 0x3B | 0x05 | | | L | | | | | | | | | | | | PAPER JAM |
| 0x3B | 0x06 | | | L | | | | | | | | | | | | FAILED TO SENSE TOP-OF-FORM |
| 0x3B | 0x07 | | | L | | | | | | | | | | | | FAILED TO SENSE BOTTOM-OF-FORM |
| 0x3B | 0x08 | | T | | | | | | | | | | | | | REPOSITION ERROR |
| 0x3B | 0x09 | | | | | | | | | | | | | | | READ PAST END OF MEDIUM |
| 0x3B | 0x0A | | | | | | | | | | | | | | | READ PAST BEGINNING OF MEDIUM |
| 0x3B | 0x0B | | | | | | | | | | | | | | | POSITION PAST END OF MEDIUM |
| 0x3B | 0x0C | | T | | | | | | | | | | | | | POSITION PAST BEGINNING OF MEDIUM |
| 0x3B | 0x0D | D | T | | | W | R | O | M | | | B | K | | | MEDIUM DESTINATION ELEMENT FULL |
| 0x3B | 0x0E | D | T | | | W | R | O | M | | | B | K | | | MEDIUM SOURCE ELEMENT EMPTY |
| 0x3B | 0x0F | | | | | | R | | | | | | | | | END OF MEDIUM REACHED |
| 0x3B | 0x11 | D | T | | | W | R | O | M | | | B | K | | | MEDIUM MAGAZINE NOT ACCESSIBLE |
| 0x3B | 0x12 | D | T | | | W | R | O | M | | | B | K | | | MEDIUM MAGAZINE REMOVED |
| 0x3B | 0x13 | D | T | | | W | R | O | M | | | B | K | | | MEDIUM MAGAZINE INSERTED |
| 0x3B | 0x14 | D | T | | | W | R | O | M | | | B | K | | | MEDIUM MAGAZINE LOCKED |
| 0x3B | 0x15 | D | T | | | W | R | O | M | | | B | K | | | MEDIUM MAGAZINE UNLOCKED |
| 0x3B | 0x16 | | | | | | R | | | | | | | | | MECHANICAL POSITIONING OR CHANGER ERROR |
| 0x3B | 0x17 | | | | | | | | | | | | | | F | READ PAST END OF USER OBJECT |
| 0x3C | 0x00 | | | | | | | | | | | | | | | |
| 0x3D | 0x00 | D | T | L | P | W | R | O | M | A | E | | K | | | INVALID BITS IN IDENTIFY MESSAGE |
| 0x3E | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT HAS NOT SELF-CONFIGURED YET |
| 0x3E | 0x01 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT FAILURE |
| 0x3E | 0x02 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | TIMEOUT ON LOGICAL UNIT |
| 0x3E | 0x03 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT FAILED SELF-TEST |
| 0x3E | 0x04 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT UNABLE TO UPDATE SELF-TEST LOG |
| 0x3F | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | TARGET OPERATING CONDITIONS HAVE CHANGED |
| 0x3F | 0x01 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | MICROCODE HAS BEEN CHANGED |
| 0x3F | 0x02 | D | T | L | P | W | R | O | M | | | B | K | | | CHANGED OPERATING DEFINITION |
| 0x3F | 0x03 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | INQUIRY DATA HAS CHANGED |
| 0x3F | 0x04 | D | T | | | W | R | O | M | A | E | B | K | | | COMPONENT DEVICE ATTACHED |
| 0x3F | 0x05 | D | T | | | W | R | O | M | A | E | B | K | | | DEVICE IDENTIFIER CHANGED |
| 0x3F | 0x06 | D | T | | | W | R | O | M | A | E | B | | | | REDUNDANCY GROUP CREATED OR MODIFIED |
| 0x3F | 0x07 | D | T | | | W | R | O | M | A | E | B | | | | REDUNDANCY GROUP DELETED |
| 0x3F | 0x08 | D | T | | | W | R | O | M | A | E | B | | | | SPARE CREATED OR MODIFIED |
| 0x3F | 0x09 | D | T | | | W | R | O | M | A | E | B | | | | SPARE DELETED |
| 0x3F | 0x0A | D | T | | | W | R | O | M | A | E | B | K | | | VOLUME SET CREATED OR MODIFIED |
| 0x3F | 0x0B | D | T | | | W | R | O | M | A | E | B | K | | | VOLUME SET DELETED |
| 0x3F | 0x0C | D | T | | | W | R | O | M | A | E | B | K | | | VOLUME SET DEASSIGNED |
| 0x3F | 0x0D | D | T | | | W | R | O | M | A | E | B | K | | | VOLUME SET REASSIGNED |
| 0x3F | 0x0E | D | T | L | P | W | R | O | M | A | E | | | | | REPORTED LUNS DATA HAS CHANGED |
| 0x3F | 0x0F | D | T | L | P | W | R | O | M | A | E | B | K | V | F | ECHO BUFFER OVERWRITTEN |
| 0x3F | 0x10 | D | T | | | W | R | O | M | | | B | | | | MEDIUM LOADABLE |
| 0x3F | 0x11 | D | T | | | W | R | O | M | | | B | | | | MEDIUM AUXILIARY MEMORY ACCESSIBLE |
| 0x3F | 0x12 | D | T | L | P | W | R | | M | A | E | B | K | | F | iSCSI IP ADDRESS ADDED |
| 0x3F | 0x13 | D | T | L | P | W | R | | M | A | E | B | K | | F | iSCSI IP ADDRESS REMOVED |
| 0x3F | 0x14 | D | T | L | P | W | R | | M | A | E | B | K | | F | iSCSI IP ADDRESS CHANGED |
| 0x40 | 0x00 | D | | | | | | | | | | | | | | RAM FAILURE (SHOULD USE 40 NN) |
| 0x40 | 0xNN | D | T | L | P | W | R | O | M | A | E | B | K | V | F | DIAGNOSTIC FAILURE ON COMPONENT NN (80h-FFh) |
| 0x41 | 0x00 | D | | | | | | | | | | | | | | DATA PATH FAILURE (SHOULD USE 40 NN) |
| 0x42 | 0x00 | D | | | | | | | | | | | | | | POWER-ON OR SELF-TEST FAILURE (SHOULD USE 40 NN) |
| 0x43 | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | MESSAGE ERROR |
| 0x44 | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | INTERNAL TARGET FAILURE |
| 0x44 | 0x71 | D | T | | | | | | | | | B | | | | ATA DEVICE FAILED SET FEATURES |
| 0x45 | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | SELECT OR RESELECT FAILURE |
| 0x46 | 0x00 | D | T | L | P | W | R | O | M | | | B | K | | | UNSUCCESSFUL SOFT RESET |
| 0x47 | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | SCSI PARITY ERROR |
| 0x47 | 0x01 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | DATA PHASE CRC ERROR DETECTED |
| 0x47 | 0x02 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | SCSI PARITY ERROR DETECTED DURING ST DATA PHASE |
| 0x47 | 0x03 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | INFORMATION UNIT iuCRC ERROR DETECTED |
| 0x47 | 0x04 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | ASYNCHRONOUS INFORMATION PROTECTION ERROR DETECTED |
| 0x47 | 0x05 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | PROTOCOL SERVICE CRC ERROR |
| 0x47 | 0x06 | D | T | | | | | | M | A | E | B | K | V | F | PHY TEST FUNCTION IN PROGRESS |
| 0x47 | 0x7F | D | T | | P | W | R | O | M | A | E | B | K | | | SOME COMMANDS CLEARED BY ISCSI PROTOCOL EVENT |
| 0x48 | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | INITIATOR DETECTED ERROR MESSAGE RECEIVED |
| 0x49 | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | INVALID MESSAGE ERROR |
| 0x4A | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | COMMAND PHASE ERROR |
| 0x4B | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | DATA PHASE ERROR |
| 0x4B | 0x01 | D | T | | P | W | R | O | M | A | E | B | K | | | INVALID TARGET PORT TRANSFER TAG RECEIVED |
| 0x4B | 0x02 | D | T | | P | W | R | O | M | A | E | B | K | | | TOO MUCH WRITE DATA |
| 0x4B | 0x03 | D | T | | P | W | R | O | M | A | E | B | K | | | ACK/NAK TIMEOUT |
| 0x4B | 0x04 | D | T | | P | W | R | O | M | A | E | B | K | | | NAK RECEIVED |
| 0x4B | 0x05 | D | T | | P | W | R | O | M | A | E | B | K | | | DATA OFFSET ERROR |
| 0x4B | 0x06 | D | T | | P | W | R | O | M | A | E | B | K | | | INITIATOR RESPONSE TIMEOUT |
| 0x4C | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | LOGICAL UNIT FAILED SELF-CONFIGURATION |
| 0x4D | 0xNN | D | T | L | P | W | R | O | M | A | E | B | K | V | F | TAGGED OVERLAPPED COMMANDS (NN = TASK TAG) |
| 0x4E | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | OVERLAPPED COMMANDS ATTEMPTED |
| 0x4F | 0x00 | | | | | | | | | | | | | | | |
| 0x50 | 0x00 | | T | | | | | | | | | | | | | WRITE APPEND ERROR |
| 0x50 | 0x01 | | T | | | | | | | | | | | | | WRITE APPEND POSITION ERROR |
| 0x50 | 0x02 | | T | | | | | | | | | | | | | POSITION ERROR RELATED TO TIMING |
| 0x51 | 0x00 | | T | | | | R | O | | | | | | | | ERASE FAILURE |
| 0x51 | 0x01 | | | | | | R | | | | | | | | | ERASE FAILURE - INCOMPLETE ERASE OPERATION DETECTED |
| 0x52 | 0x00 | | T | | | | | | | | | | | | | CARTRIDGE FAULT |
| 0x53 | 0x00 | D | T | L | | W | R | O | M | | | B | K | | | MEDIA LOAD OR EJECT FAILED |
| 0x53 | 0x01 | | T | | | | | | | | | | | | | UNLOAD TAPE FAILURE |
| 0x53 | 0x02 | D | T | | | W | R | O | M | | | B | K | | | MEDIUM REMOVAL PREVENTED |
| 0x53 | 0x03 | | | | | | | | M | | | | | | | MEDIUM REMOVAL PREVENTED BY DATA TRANSFER ELEMENT |
| 0x53 | 0x04 | | T | | | | | | | | | | | | | MEDIUM THREAD OR UNTHREAD FAILURE |
| 0x54 | 0x00 | | | | P | | | | | | | | | | | SCSI TO HOST SYSTEM INTERFACE FAILURE |
| 0x55 | 0x00 | | | | P | | | | | | | | | | | SYSTEM RESOURCE FAILURE |
| 0x55 | 0x01 | D | | | | | | O | | | | B | K | | | SYSTEM BUFFER FULL |
| 0x55 | 0x02 | D | T | L | P | W | R | O | M | A | E | | K | | | INSUFFICIENT RESERVATION RESOURCES |
| 0x55 | 0x03 | D | T | L | P | W | R | O | M | A | E | | K | | | INSUFFICIENT RESOURCES |
| 0x55 | 0x04 | D | T | L | P | W | R | O | M | A | E | | K | | | INSUFFICIENT REGISTRATION RESOURCES |
| 0x55 | 0x05 | D | T | | P | W | R | O | M | A | E | B | K | | | INSUFFICIENT ACCESS CONTROL RESOURCES |
| 0x55 | 0x06 | D | T | | | W | R | O | M | | | B | | | | AUXILIARY MEMORY OUT OF SPACE |
| 0x55 | 0x07 | | | | | | | | | | | | | | F | QUOTA ERROR |
| 0x55 | 0x08 | | T | | | | | | | | | | | | | MAXIMUM NUMBER OF SUPPLEMENTAL DECRYPTION KEYS EXCEEDED |
| 0x56 | 0x00 | | | | | | | | | | | | | | | |
| 0x57 | 0x00 | | | | | | R | | | | | | | | | UNABLE TO RECOVER TABLE-OF-CONTENTS |
| 0x58 | 0x00 | | | | | | | O | | | | | | | | GENERATION DOES NOT EXIST |
| 0x59 | 0x00 | | | | | | | O | | | | | | | | UPDATED BLOCK READ |
| 0x5A | 0x00 | D | T | L | P | W | R | O | M | | | B | K | | | OPERATOR REQUEST OR STATE CHANGE INPUT |
| 0x5A | 0x01 | D | T | | | W | R | O | M | | | B | K | | | OPERATOR MEDIUM REMOVAL REQUEST |
| 0x5A | 0x02 | D | T | | | W | R | O | | A | | B | K | | | OPERATOR SELECTED WRITE PROTECT |
| 0x5A | 0x03 | D | T | | | W | R | O | | A | | B | K | | | OPERATOR SELECTED WRITE PERMIT |
| 0x5B | 0x00 | D | T | L | P | W | R | O | M | | | | K | | | LOG EXCEPTION |
| 0x5B | 0x01 | D | T | L | P | W | R | O | M | | | | K | | | THRESHOLD CONDITION MET |
| 0x5B | 0x02 | D | T | L | P | W | R | O | M | | | | K | | | LOG COUNTER AT MAXIMUM |
| 0x5B | 0x03 | D | T | L | P | W | R | O | M | | | | K | | | LOG LIST CODES EXHAUSTED |
| 0x5C | 0x00 | D | | | | | | O | | | | | | | | RPL STATUS CHANGE |
| 0x5C | 0x01 | D | | | | | | O | | | | | | | | SPINDLES SYNCHRONIZED |
| 0x5C | 0x02 | D | | | | | | O | | | | | | | | SPINDLES NOT SYNCHRONIZED |
| 0x5D | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | FAILURE PREDICTION THRESHOLD EXCEEDED |
| 0x5D | 0x01 | | | | | | R | | | | | B | | | | MEDIA FAILURE PREDICTION THRESHOLD EXCEEDED |
| 0x5D | 0x02 | | | | | | R | | | | | | | | | LOGICAL UNIT FAILURE PREDICTION THRESHOLD EXCEEDED |
| 0x5D | 0x03 | | | | | | R | | | | | | | | | SPARE AREA EXHAUSTION PREDICTION THRESHOLD EXCEEDED |
| 0x5D | 0x10 | D | | | | | | | | | | B | | | | HARDWARE IMPENDING FAILURE GENERAL HARD DRIVE FAILURE |
| 0x5D | 0x11 | D | | | | | | | | | | B | | | | HARDWARE IMPENDING FAILURE DRIVE ERROR RATE TOO HIGH |
| 0x5D | 0x12 | D | | | | | | | | | | B | | | | HARDWARE IMPENDING FAILURE DATA ERROR RATE TOO HIGH |
| 0x5D | 0x13 | D | | | | | | | | | | B | | | | HARDWARE IMPENDING FAILURE SEEK ERROR RATE TOO HIGH |
| 0x5D | 0x14 | D | | | | | | | | | | B | | | | HARDWARE IMPENDING FAILURE TOO MANY BLOCK REASSIGNS |
| 0x5D | 0x15 | D | | | | | | | | | | B | | | | HARDWARE IMPENDING FAILURE ACCESS TIMES TOO HIGH |
| 0x5D | 0x16 | D | | | | | | | | | | B | | | | HARDWARE IMPENDING FAILURE START UNIT TIMES TOO HIGH |
| 0x5D | 0x17 | D | | | | | | | | | | B | | | | HARDWARE IMPENDING FAILURE CHANNEL PARAMETRICS |
| 0x5D | 0x18 | D | | | | | | | | | | B | | | | HARDWARE IMPENDING FAILURE CONTROLLER DETECTED |
| 0x5D | 0x19 | D | | | | | | | | | | B | | | | HARDWARE IMPENDING FAILURE THROUGHPUT PERFORMANCE |
| 0x5D | 0x1A | D | | | | | | | | | | B | | | | HARDWARE IMPENDING FAILURE SEEK TIME PERFORMANCE |
| 0x5D | 0x1B | D | | | | | | | | | | B | | | | HARDWARE IMPENDING FAILURE SPIN-UP RETRY COUNT |
| 0x5D | 0x1C | D | | | | | | | | | | B | | | | HARDWARE IMPENDING FAILURE DRIVE CALIBRATION RETRY COUNT |
| 0x5D | 0x20 | D | | | | | | | | | | B | | | | CONTROLLER IMPENDING FAILURE GENERAL HARD DRIVE FAILURE |
| 0x5D | 0x21 | D | | | | | | | | | | B | | | | CONTROLLER IMPENDING FAILURE DRIVE ERROR RATE TOO HIGH |
| 0x5D | 0x22 | D | | | | | | | | | | B | | | | CONTROLLER IMPENDING FAILURE DATA ERROR RATE TOO HIGH |
| 0x5D | 0x23 | D | | | | | | | | | | B | | | | CONTROLLER IMPENDING FAILURE SEEK ERROR RATE TOO HIGH |
| 0x5D | 0x24 | D | | | | | | | | | | B | | | | CONTROLLER IMPENDING FAILURE TOO MANY BLOCK REASSIGNS |
| 0x5D | 0x25 | D | | | | | | | | | | B | | | | CONTROLLER IMPENDING FAILURE ACCESS TIMES TOO HIGH |
| 0x5D | 0x26 | D | | | | | | | | | | B | | | | CONTROLLER IMPENDING FAILURE START UNIT TIMES TOO HIGH |
| 0x5D | 0x27 | D | | | | | | | | | | B | | | | CONTROLLER IMPENDING FAILURE CHANNEL PARAMETRICS |
| 0x5D | 0x28 | D | | | | | | | | | | B | | | | CONTROLLER IMPENDING FAILURE CONTROLLER DETECTED |
| 0x5D | 0x29 | D | | | | | | | | | | B | | | | CONTROLLER IMPENDING FAILURE THROUGHPUT PERFORMANCE |
| 0x5D | 0x2A | D | | | | | | | | | | B | | | | CONTROLLER IMPENDING FAILURE SEEK TIME PERFORMANCE |
| 0x5D | 0x2B | D | | | | | | | | | | B | | | | CONTROLLER IMPENDING FAILURE SPIN-UP RETRY COUNT |
| 0x5D | 0x2C | D | | | | | | | | | | B | | | | CONTROLLER IMPENDING FAILURE DRIVE CALIBRATION RETRY COUNT |
| 0x5D | 0x30 | D | | | | | | | | | | B | | | | DATA CHANNEL IMPENDING FAILURE GENERAL HARD DRIVE FAILURE |
| 0x5D | 0x31 | D | | | | | | | | | | B | | | | DATA CHANNEL IMPENDING FAILURE DRIVE ERROR RATE TOO HIGH |
| 0x5D | 0x32 | D | | | | | | | | | | B | | | | DATA CHANNEL IMPENDING FAILURE DATA ERROR RATE TOO HIGH |
| 0x5D | 0x33 | D | | | | | | | | | | B | | | | DATA CHANNEL IMPENDING FAILURE SEEK ERROR RATE TOO HIGH |
| 0x5D | 0x34 | D | | | | | | | | | | B | | | | DATA CHANNEL IMPENDING FAILURE TOO MANY BLOCK REASSIGNS |
| 0x5D | 0x35 | D | | | | | | | | | | B | | | | DATA CHANNEL IMPENDING FAILURE ACCESS TIMES TOO HIGH |
| 0x5D | 0x36 | D | | | | | | | | | | B | | | | DATA CHANNEL IMPENDING FAILURE START UNIT TIMES TOO HIGH |
| 0x5D | 0x37 | D | | | | | | | | | | B | | | | DATA CHANNEL IMPENDING FAILURE CHANNEL PARAMETRICS |
| 0x5D | 0x38 | D | | | | | | | | | | B | | | | DATA CHANNEL IMPENDING FAILURE CONTROLLER DETECTED |
| 0x5D | 0x39 | D | | | | | | | | | | B | | | | DATA CHANNEL IMPENDING FAILURE THROUGHPUT PERFORMANCE |
| 0x5D | 0x3A | D | | | | | | | | | | B | | | | DATA CHANNEL IMPENDING FAILURE SEEK TIME PERFORMANCE |
| 0x5D | 0x3B | D | | | | | | | | | | B | | | | DATA CHANNEL IMPENDING FAILURE SPIN-UP RETRY COUNT |
| 0x5D | 0x3C | D | | | | | | | | | | B | | | | DATA CHANNEL IMPENDING FAILURE DRIVE CALIBRATION RETRY COUNT |
| 0x5D | 0x40 | D | | | | | | | | | | B | | | | SERVO IMPENDING FAILURE GENERAL HARD DRIVE FAILURE |
| 0x5D | 0x41 | D | | | | | | | | | | B | | | | SERVO IMPENDING FAILURE DRIVE ERROR RATE TOO HIGH |
| 0x5D | 0x42 | D | | | | | | | | | | B | | | | SERVO IMPENDING FAILURE DATA ERROR RATE TOO HIGH |
| 0x5D | 0x43 | D | | | | | | | | | | B | | | | SERVO IMPENDING FAILURE SEEK ERROR RATE TOO HIGH |
| 0x5D | 0x44 | D | | | | | | | | | | B | | | | SERVO IMPENDING FAILURE TOO MANY BLOCK REASSIGNS |
| 0x5D | 0x45 | D | | | | | | | | | | B | | | | SERVO IMPENDING FAILURE ACCESS TIMES TOO HIGH |
| 0x5D | 0x46 | D | | | | | | | | | | B | | | | SERVO IMPENDING FAILURE START UNIT TIMES TOO HIGH |
| 0x5D | 0x47 | D | | | | | | | | | | B | | | | SERVO IMPENDING FAILURE CHANNEL PARAMETRICS |
| 0x5D | 0x48 | D | | | | | | | | | | B | | | | SERVO IMPENDING FAILURE CONTROLLER DETECTED |
| 0x5D | 0x49 | D | | | | | | | | | | B | | | | SERVO IMPENDING FAILURE THROUGHPUT PERFORMANCE |
| 0x5D | 0x4A | D | | | | | | | | | | B | | | | SERVO IMPENDING FAILURE SEEK TIME PERFORMANCE |
| 0x5D | 0x4B | D | | | | | | | | | | B | | | | SERVO IMPENDING FAILURE SPIN-UP RETRY COUNT |
| 0x5D | 0x4C | D | | | | | | | | | | B | | | | SERVO IMPENDING FAILURE DRIVE CALIBRATION RETRY COUNT |
| 0x5D | 0x50 | D | | | | | | | | | | B | | | | SPINDLE IMPENDING FAILURE GENERAL HARD DRIVE FAILURE |
| 0x5D | 0x51 | D | | | | | | | | | | B | | | | SPINDLE IMPENDING FAILURE DRIVE ERROR RATE TOO HIGH |
| 0x5D | 0x52 | D | | | | | | | | | | B | | | | SPINDLE IMPENDING FAILURE DATA ERROR RATE TOO HIGH |
| 0x5D | 0x53 | D | | | | | | | | | | B | | | | SPINDLE IMPENDING FAILURE SEEK ERROR RATE TOO HIGH |
| 0x5D | 0x54 | D | | | | | | | | | | B | | | | SPINDLE IMPENDING FAILURE TOO MANY BLOCK REASSIGNS |
| 0x5D | 0x55 | D | | | | | | | | | | B | | | | SPINDLE IMPENDING FAILURE ACCESS TIMES TOO HIGH |
| 0x5D | 0x56 | D | | | | | | | | | | B | | | | SPINDLE IMPENDING FAILURE START UNIT TIMES TOO HIGH |
| 0x5D | 0x57 | D | | | | | | | | | | B | | | | SPINDLE IMPENDING FAILURE CHANNEL PARAMETRICS |
| 0x5D | 0x58 | D | | | | | | | | | | B | | | | SPINDLE IMPENDING FAILURE CONTROLLER DETECTED |
| 0x5D | 0x59 | D | | | | | | | | | | B | | | | SPINDLE IMPENDING FAILURE THROUGHPUT PERFORMANCE |
| 0x5D | 0x5A | D | | | | | | | | | | B | | | | SPINDLE IMPENDING FAILURE SEEK TIME PERFORMANCE |
| 0x5D | 0x5B | D | | | | | | | | | | B | | | | SPINDLE IMPENDING FAILURE SPIN-UP RETRY COUNT |
| 0x5D | 0x5C | D | | | | | | | | | | B | | | | SPINDLE IMPENDING FAILURE DRIVE CALIBRATION RETRY COUNT |
| 0x5D | 0x60 | D | | | | | | | | | | B | | | | FIRMWARE IMPENDING FAILURE GENERAL HARD DRIVE FAILURE |
| 0x5D | 0x61 | D | | | | | | | | | | B | | | | FIRMWARE IMPENDING FAILURE DRIVE ERROR RATE TOO HIGH |
| 0x5D | 0x62 | D | | | | | | | | | | B | | | | FIRMWARE IMPENDING FAILURE DATA ERROR RATE TOO HIGH |
| 0x5D | 0x63 | D | | | | | | | | | | B | | | | FIRMWARE IMPENDING FAILURE SEEK ERROR RATE TOO HIGH |
| 0x5D | 0x64 | D | | | | | | | | | | B | | | | FIRMWARE IMPENDING FAILURE TOO MANY BLOCK REASSIGNS |
| 0x5D | 0x65 | D | | | | | | | | | | B | | | | FIRMWARE IMPENDING FAILURE ACCESS TIMES TOO HIGH |
| 0x5D | 0x66 | D | | | | | | | | | | B | | | | FIRMWARE IMPENDING FAILURE START UNIT TIMES TOO HIGH |
| 0x5D | 0x67 | D | | | | | | | | | | B | | | | FIRMWARE IMPENDING FAILURE CHANNEL PARAMETRICS |
| 0x5D | 0x68 | D | | | | | | | | | | B | | | | FIRMWARE IMPENDING FAILURE CONTROLLER DETECTED |
| 0x5D | 0x69 | D | | | | | | | | | | B | | | | FIRMWARE IMPENDING FAILURE THROUGHPUT PERFORMANCE |
| 0x5D | 0x6A | D | | | | | | | | | | B | | | | FIRMWARE IMPENDING FAILURE SEEK TIME PERFORMANCE |
| 0x5D | 0x6B | D | | | | | | | | | | B | | | | FIRMWARE IMPENDING FAILURE SPIN-UP RETRY COUNT |
| 0x5D | 0x6C | D | | | | | | | | | | B | | | | FIRMWARE IMPENDING FAILURE DRIVE CALIBRATION RETRY COUNT |
| 0x5D | 0xFF | D | T | L | P | W | R | O | M | A | E | B | K | V | F | FAILURE PREDICTION THRESHOLD EXCEEDED (FALSE) |
| 0x5E | 0x00 | D | T | L | P | W | R | O | | A | | | K | | | LOW POWER CONDITION ON |
| 0x5E | 0x01 | D | T | L | P | W | R | O | | A | | | K | | | IDLE CONDITION ACTIVATED BY TIMER |
| 0x5E | 0x02 | D | T | L | P | W | R | O | | A | | | K | | | STANDBY CONDITION ACTIVATED BY TIMER |
| 0x5E | 0x03 | D | T | L | P | W | R | O | | A | | | K | | | IDLE CONDITION ACTIVATED BY COMMAND |
| 0x5E | 0x04 | D | T | L | P | W | R | O | | A | | | K | | | STANDBY CONDITION ACTIVATED BY COMMAND |
| 0x5E | 0x41 | | | | | | | | | | | B | | | | POWER STATE CHANGE TO ACTIVE |
| 0x5E | 0x42 | | | | | | | | | | | B | | | | POWER STATE CHANGE TO IDLE |
| 0x5E | 0x43 | | | | | | | | | | | B | | | | POWER STATE CHANGE TO STANDBY |
| 0x5E | 0x45 | | | | | | | | | | | B | | | | POWER STATE CHANGE TO SLEEP |
| 0x5E | 0x47 | | | | | | | | | | | B | K | | | POWER STATE CHANGE TO DEVICE CONTROL |
| 0x5F | 0x00 | | | | | | | | | | | | | | | |
| 0x60 | 0x00 | | | | | | | | | | | | | | | LAMP FAILURE |
| 0x61 | 0x00 | | | | | | | | | | | | | | | VIDEO ACQUISITION ERROR |
| 0x61 | 0x01 | | | | | | | | | | | | | | | UNABLE TO ACQUIRE VIDEO |
| 0x61 | 0x02 | | | | | | | | | | | | | | | OUT OF FOCUS |
| 0x62 | 0x00 | | | | | | | | | | | | | | | SCAN HEAD POSITIONING ERROR |
| 0x63 | 0x00 | | | | | | R | | | | | | | | | END OF USER AREA ENCOUNTERED ON THIS TRACK |
| 0x63 | 0x01 | | | | | | R | | | | | | | | | PACKET DOES NOT FIT IN AVAILABLE SPACE |
| 0x64 | 0x00 | | | | | | R | | | | | | | | | ILLEGAL MODE FOR THIS TRACK |
| 0x64 | 0x01 | | | | | | R | | | | | | | | | INVALID PACKET SIZE |
| 0x65 | 0x00 | D | T | L | P | W | R | O | M | A | E | B | K | V | F | VOLTAGE FAULT |
| 0x66 | 0x00 | | | | | | | | | | | | | | | AUTOMATIC DOCUMENT FEEDER COVER UP |
| 0x66 | 0x01 | | | | | | | | | | | | | | | AUTOMATIC DOCUMENT FEEDER LIFT UP |
| 0x66 | 0x02 | | | | | | | | | | | | | | | DOCUMENT JAM IN AUTOMATIC DOCUMENT FEEDER |
| 0x66 | 0x03 | | | | | | | | | | | | | | | DOCUMENT MISS FEED AUTOMATIC IN DOCUMENT FEEDER |
| 0x67 | 0x00 | | | | | | | | | A | | | | | | CONFIGURATION FAILURE |
| 0x67 | 0x01 | | | | | | | | | A | | | | | | CONFIGURATION OF INCAPABLE LOGICAL UNITS FAILED |
| 0x67 | 0x02 | | | | | | | | | A | | | | | | ADD LOGICAL UNIT FAILED |
| 0x67 | 0x03 | | | | | | | | | A | | | | | | MODIFICATION OF LOGICAL UNIT FAILED |
| 0x67 | 0x04 | | | | | | | | | A | | | | | | EXCHANGE OF LOGICAL UNIT FAILED |
| 0x67 | 0x05 | | | | | | | | | A | | | | | | REMOVE OF LOGICAL UNIT FAILED |
| 0x67 | 0x06 | | | | | | | | | A | | | | | | ATTACHMENT OF LOGICAL UNIT FAILED |
| 0x67 | 0x07 | | | | | | | | | A | | | | | | CREATION OF LOGICAL UNIT FAILED |
| 0x67 | 0x08 | | | | | | | | | A | | | | | | ASSIGN FAILURE OCCURRED |
| 0x67 | 0x09 | | | | | | | | | A | | | | | | MULTIPLY ASSIGNED LOGICAL UNIT |
| 0x67 | 0x0A | D | T | L | P | W | R | O | M | A | E | B | K | V | F | SET TARGET PORT GROUPS COMMAND FAILED |
| 0x67 | 0x0B | D | T | | | | | | | | | B | | | | ATA DEVICE FEATURE NOT ENABLED |
| 0x68 | 0x00 | | | | | | | | | A | | | | | | LOGICAL UNIT NOT CONFIGURED |
| 0x69 | 0x00 | | | | | | | | | A | | | | | | DATA LOSS ON LOGICAL UNIT |
| 0x69 | 0x01 | | | | | | | | | A | | | | | | MULTIPLE LOGICAL UNIT FAILURES |
| 0x69 | 0x02 | | | | | | | | | A | | | | | | PARITY/DATA MISMATCH |
| 0x6A | 0x00 | | | | | | | | | A | | | | | | INFORMATIONAL, REFER TO LOG |
| 0x6B | 0x00 | | | | | | | | | A | | | | | | STATE CHANGE HAS OCCURRED |
| 0x6B | 0x01 | | | | | | | | | A | | | | | | REDUNDANCY LEVEL GOT BETTER |
| 0x6B | 0x02 | | | | | | | | | A | | | | | | REDUNDANCY LEVEL GOT WORSE |
| 0x6C | 0x00 | | | | | | | | | A | | | | | | REBUILD FAILURE OCCURRED |
| 0x6D | 0x00 | | | | | | | | | A | | | | | | RECALCULATE FAILURE OCCURRED |
| 0x6E | 0x00 | | | | | | | | | A | | | | | | COMMAND TO LOGICAL UNIT FAILED |
| 0x6F | 0x00 | | | | | | R | | | | | | | | | COPY PROTECTION KEY EXCHANGE FAILURE - AUTHENTICATION FAILURE |
| 0x6F | 0x01 | | | | | | R | | | | | | | | | COPY PROTECTION KEY EXCHANGE FAILURE - KEY NOT PRESENT |
| 0x6F | 0x02 | | | | | | R | | | | | | | | | COPY PROTECTION KEY EXCHANGE FAILURE - KEY NOT ESTABLISHED |
| 0x6F | 0x03 | | | | | | R | | | | | | | | | READ OF SCRAMBLED SECTOR WITHOUT AUTHENTICATION |
| 0x6F | 0x04 | | | | | | R | | | | | | | | | MEDIA REGION CODE IS MISMATCHED TO LOGICAL UNIT REGION |
| 0x6F | 0x05 | | | | | | R | | | | | | | | | DRIVE REGION MUST BE PERMANENT/REGION RESET COUNT ERROR |
| 0x6F | 0x06 | | | | | | R | | | | | | | | | INSUFFICIENT BLOCK COUNT FOR BINDING NONCE RECORDING |
| 0x6F | 0x07 | | | | | | R | | | | | | | | | CONFLICT IN BINDING NONCE RECORDING |
| 0x70 | 0xNN | | T | | | | | | | | | | | | | DECOMPRESSION EXCEPTION SHORT ALGORITHM ID OF NN |
| 0x71 | 0x00 | | T | | | | | | | | | | | | | DECOMPRESSION EXCEPTION LONG ALGORITHM ID |
| 0x72 | 0x00 | | | | | | R | | | | | | | | | SESSION FIXATION ERROR |
| 0x72 | 0x01 | | | | | | R | | | | | | | | | SESSION FIXATION ERROR WRITING LEAD-IN |
| 0x72 | 0x02 | | | | | | R | | | | | | | | | SESSION FIXATION ERROR WRITING LEAD-OUT |
| 0x72 | 0x03 | | | | | | R | | | | | | | | | SESSION FIXATION ERROR - INCOMPLETE TRACK IN SESSION |
| 0x72 | 0x04 | | | | | | R | | | | | | | | | EMPTY OR PARTIALLY WRITTEN RESERVED TRACK |
| 0x72 | 0x05 | | | | | | R | | | | | | | | | NO MORE TRACK RESERVATIONS ALLOWED |
| 0x72 | 0x06 | | | | | | R | | | | | | | | | RMZ EXTENSION IS NOT ALLOWED |
| 0x72 | 0x07 | | | | | | R | | | | | | | | | NO MORE TEST ZONE EXTENSIONS ARE ALLOWED |
| 0x73 | 0x00 | | | | | | R | | | | | | | | | CD CONTROL ERROR |
| 0x73 | 0x01 | | | | | | R | | | | | | | | | POWER CALIBRATION AREA ALMOST FULL |
| 0x73 | 0x02 | | | | | | R | | | | | | | | | POWER CALIBRATION AREA IS FULL |
| 0x73 | 0x03 | | | | | | R | | | | | | | | | POWER CALIBRATION AREA ERROR |
| 0x73 | 0x04 | | | | | | R | | | | | | | | | PROGRAM MEMORY AREA UPDATE FAILURE |
| 0x73 | 0x05 | | | | | | R | | | | | | | | | PROGRAM MEMORY AREA IS FULL |
| 0x73 | 0x06 | | | | | | R | | | | | | | | | RMA/PMA IS ALMOST FULL |
| 0x73 | 0x10 | | | | | | R | | | | | | | | | CURRENT POWER CALIBRATION AREA ALMOST FULL |
| 0x73 | 0x11 | | | | | | R | | | | | | | | | CURRENT POWER CALIBRATION AREA IS FULL |
| 0x73 | 0x17 | | | | | | R | | | | | | | | | RDZ IS FULL |
| 0x74 | 0x00 | | T | | | | | | | | | | | | | SECURITY ERROR |
| 0x74 | 0x01 | | T | | | | | | | | | | | | | UNABLE TO DECRYPT DATA |
| 0x74 | 0x02 | | T | | | | | | | | | | | | | UNENCRYPTED DATA ENCOUNTERED WHILE DECRYPTING |
| 0x74 | 0x03 | | T | | | | | | | | | | | | | INCORRECT DATA ENCRYPTION KEY |
| 0x74 | 0x04 | | T | | | | | | | | | | | | | CRYPTOGRAPHIC INTEGRITY VALIDATION FAILED |
| 0x74 | 0x05 | | T | | | | | | | | | | | | | ERROR DECRYPTING DATA |
| 0x74 | 0x71 | D | T | | | | R | | M | | E | | | V | | LOGICAL UNIT ACCESS NOT AUTHORIZED |
| 0x75 | 0x00 | | | | | | | | | | | | | | | |
| 0x76 | 0x00 | | | | | | | | | | | | | | | |
| 0x77 | 0x00 | | | | | | | | | | | | | | | |
| 0x78 | 0x00 | | | | | | | | | | | | | | | |
| 0x79 | 0x00 | | | | | | | | | | | | | | | |
| 0x7A | 0x00 | | | | | | | | | | | | | | | |
| 0x7B | 0x00 | | | | | | | | | | | | | | | |
| 0x7C | 0x00 | | | | | | | | | | | | | | | |
| 0x7D | 0x00 | | | | | | | | | | | | | | | |
| 0x7E | 0x00 | | | | | | | | | | | | | | | |
| 0x7F | 0x00 | | | | | | | | | | | | | | | |
### Device legend
| Key | Description |
| --- | ---------------------------------------- |
| D | DIRECT ACCESS DEVICE (SBC-2) |
| T | SEQUENTIAL ACCESS DEVICE (SSC) |
| L | PRINTER DEVICE (SSC) |
| P | PROCESSOR DEVICE (SPC) |
| W | WRITE ONCE READ MULTIPLE DEVICE (SBC-2) |
| R | CD DEVICE (MMC) |
| O | OPTICAL MEMORY DEVICE (SBC-2) |
| M | MEDIA CHANGER DEVICE (SMC) |
| A | STORAGE ARRAY DEVICE (SCC) |
| E | ENCLOSURE SERVICES DEVICE (SES) |
| B | SIMPLIFIED DIRECT-ACCESS DEVICE (RBC) |
| K | OPTICAL CARD READER/WRITER DEVICE (OCRW) |
| V | AUTOMATION/DRIVE INTERFACE (ADC) |
| F | OBJECT-BASED STORAGE (OSD) |
## References
- <http://en.wikipedia.org/wiki/Key_Code_Qualifier>
- <http://www.t10.org/ftp/t10/drafts/spc4/spc4r07a.pdf>
- <http://www.t10.org/lists/2asc.htm>
- <http://www.t10.org/lists/1spc-lst.htm>
## Source Credit
The information was gleaned from [SCSI Primary Commands-4 (SPC-4, draft)](http://www.t10.org/ftp/t10/drafts/spc4/spc4r07a.pdf), available online. The ASC/ASCQ table was generated from the ASCII list available at [t10.org](http://www.t10.org/lists/2asc.htm) by stixpjr@gNOSPAMl.com on 2006-11-14. This copy is edited for wiki presentation.

392
md/smtp_relay.md Normal file
View file

@ -0,0 +1,392 @@
# SMTP Relay
## Contents
- [Overview](#overview)
- [Requirements](#requirements)
- [Conventions Used](#conventions-used)
- [SSL Certificates](#ssl-certificates)
- [Software Installation](#software-installation)
- [Outbound Configuration](#outbound-configuration)
- [Basic Outbound Testing](#basic-outbound-testing)
- [Inbound Configuration](#inbound-configuration)
- [Local Relay Testing](#local-relay-testing)
- [Allowing Remote Access](#allowing-remote-access)
- [Configuring mailx](#configuring-mailx)
## Overview
When outbound network connections to ports 25 (_smtp_), 587 (_submission_), 465 (_smtps_) are blocked (as with AT\&T U-verse), sending email from a system can be troublesome; however it's likely that other outbound ports are open which can be used instead. This article will cover setting up an inbound SSL authenticated-only smtpd instance on port 443, which then relays email upstream to a standard mail provider also using SSL with the outbound connection requiring authentication.
```
any client software -> this.server:443 w/SSL and auth -> upstream normal SMTP w/SSL and auth
```
## Requirements
- [CentOS 7](http://centos.org/)
- [Postfix MTA](http://www.postfix.org/)
- [Cyrus SASL](http://asg.web.cmu.edu/sasl/)
- [OpenSSL](http://www.openssl.org/)
- [Heirloom mailx](http://heirloom.sourceforge.net/mailx.html)
- [iptables](http://en.wikipedia.org/wiki/Iptables)
Note: any modern Linux distribution should work, however package installation names and config file locations may vary slightly. CentOS 7 ships with Postfix 2.10.x, Cyrus SASL 2.1.x and OpenSSL 1.0.2k.
## Conventions Used
- **mx.mydomain.org** - the name of the DNS listed server being implemented here
- **supercoolusername**/**supercoolpassword** - the login created for _mx.mydomain.org_ herein
- **smtp.mailgun.org** - we will use [Mailgun](http://www.mailgun.com/) as the upstream hop, any upstream standard SMTP will work
- **postmaster@mydomain.org**/**mgdomainpass** - the login credentials to the upstream Mailgun server
- **port 443** - port 443/tcp is normally used for HTTPS and tends to be open in all firewalls, etc.
Note: any port can be used depending on needs, 443 was chosen as it's already open in most firewalls and other network blocking situations. This would mean you cannot share this SMTP relay server with a true HTTPS website, however. Using port 60 is a good alternate choice if port 443 is not possible.
## SSL Certificates
Both inbound to the relay server and outbound to the upstream host connections will be implemented - obtain an SSL certificate with the name of your preferred domain. This can be self-signed, a free one from [CAcert](https://cacert.org), [StartCom SSL](https://www.startssl.com/), etc. - it's the same type of SSL key/certificate used in Apache webservers.
You will typically need the Intermediate CA certificates for your SSL cert as appropriate and the Intermediate for the upstream SMTP server. Intermediate CA certificates are generally used to prevent man-in-the-middle attacks on SSL connections and provide a higher security posture.
## Software Installation
Install the required software:
```
yum -y install postfix cyrus-sasl cyrus-sasl-lib cyrus-sasl-plain cyrus-sasl-md5 openssl mailx iptables-services
```
## Outbound Configuration
Configuring the outbound SMTP from the server first is recommended; it's easier to test locally from the server before setting up the inbound SMTP daemon. By configuring and testing this portion first we confirm any future failures should not be related to dispatching outbound email to the upstream connection.
First, obtain the upstream Intermediate CA certificate(s) and save them concatenated to a single PEM in `/etc/postfix/`. The Mailgun services herein use Geotrust Global CA so we'll grab that:
```
cd /etc/postfix/
wget https://www.geotrust.com/resources/root_certificates/certificates/GeoTrust_Global_CA.pem
```
Next, create a plain text file `/etc/postfix/sasl_passwd`that contains the upstream authentication username/password info. Secure it from prying eyes and then use `postmap` to create a Postfix-friendly database file out of it.
> For Mailgun this information is listed separately for each one of your configured domains, **not** your primary account login; the username is typically _postmaster@domain.name_ in form.
```
echo 'smtp.mailgun.org postmaster@mydomain.org:mgdomainpass' >> /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd
chmod 0640 /etc/postfix/sasl_passwd*
chown root:postfix /etc/postfix/sasl_passwd*
```
Now add the configuration to Postfix for the outbound connection - these are the `smtp_*` settings (think "postfix as a client"):
```
/etc/postfix/main.cf
myhostname = mx.mydomain.org
mynetworks_style = host
relayhost = [smtp.mailgun.org]:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_tls_security_level = may
smtp_tls_note_starttls_offer = yes
smtp_tls_CAfile = /etc/postfix/GeoTrust_Global_CA.pem
```
Lastly ensure Postfix is enabled at boot and start it:
```
systemctl enable --now postfix.service
```
> If this is the first time configuring Postfix on a new installation, be sure and set the other needed settings such as `inet_interfaces`, `mynetworks_style` and similar configuration options. By default the config is locked to localhost and cannot receive mail from the outside world.
## Basic Outbound Testing
At this point you should be able to send an email using `mail` which will connect to the local Postfix instance on port 25 and relay your outbound email:
```
mail -s "Test subject" myemail@gmail.com
```
Examining `/var/log/maillog` should show the email being accepted by the upstream host for delivery like so:
```
/var/log/maillog
Oct 4 18:00:53 myserver postfix/smtp[949]: 8C3F4421E2: to=<myemail@gmail.com>, relay=smtp.mailgun.org[50.56.21.178]:587, delay=0.95, delays=0.35/0.11/0.28/0.21, dsn=2.0.0, status=sent (250 Great success)
```
Before proceeding further, ensure that this basic functionality is working.
## Inbound Configuration
This part is a little more involved, as precautions need to be taken to ensure that SSL is mandatory and a username/password is required for anything. Only persons with these credentials can send email, even to local server users.
First, save the SSL key and certificate for this server together in a single PEM file; additionally you'll need the Intermediate CA from where the certificate was obtained. StartCom SSL is used in this example.
> The key must be decrypted (without a password) - you may have to manually decrypt the key first, like so: `openssl rsa -in` _encrypted.key_ `-out` _decrypted.key_
```
cd /etc/postfix/
wget -O StartCom_CA.pem http://www.startssl.com/certs/sub.class1.server.ca.pem
cat /path/to/server.key >> /etc/postfix/mydomain-org.pem
cat /path/to/server.crt >> /etc/postfix/mydomain-org.pem
chmod 0640 /etc/postfix/mydomain-org.pem
chown root:postfix /etc/postfix/mydomain-org.pem
```
Next, create a SASL database file with the user/pass combo of your choosing to use for logging in -- the `-c` flag is to create a new databases, so if you're adding a second, third, etc. user drop that from the command. The `-u foo` option is the _realm_ that is used for authentication domain later, it can be anything but we'll use our DNS name _mx.mydomain.org_ to make it easy.
```
saslpasswd2 -c -f /etc/postfix/sasldb2 -u mx.mydomain.org -a smtpauth supercoolusername
chmod 0640 /etc/postfix/sasldb2
chown root:postfix /etc/postfix/sasldb2
```
Update the Postfix SASL config file to use this database file instead of the default which is [PAM](http://en.wikipedia.org/wiki/Pluggable_authentication_module) -- we're not going to create system users at all. This also means there is no need to run the `saslauthd` daemon, the SASL library dynamically linked into Postfix (`/usr/libexec/postfix/smtpd` -\> `/usr/lib64/libsasl2.so.2`) will read this config and use the DB directly as needed.
```
/etc/sasl2/smtpd.conf
pwcheck_method: auxprop
auxprop_plugin: sasldb
sasldb_path: /etc/postfix/sasldb2
mech_list: plain login
log_level: 1
```
Add the configuration of the `smtpd_*` settings in Postfix to accept the incoming SSL connections, again specifying parameters to lock it down for security. Note that the `smtpd_sasl_local_domain` setting matched the _realm_ named used above in the `saslpasswd2` command.
```
/etc/postfix/main.cf
smtpd_tls_cert_file = /etc/postfix/mydomain-org.pem
smtpd_tls_key_file = /etc/postfix/mydomain-org.pem
smtpd_tls_CAfile = /etc/postfix/StartCom_CA.pem
smtpd_tls_loglevel = 0
smtpd_tls_mandatory_ciphers = medium
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3
smtpd_sasl_type = cyrus
smtpd_sasl_local_domain = mx.mydomain.org
smtpd_sasl_path = smtpd
smtpd_sasl_security_options = noanonymous
broken_sasl_auth_clients = yes
```
The last major step is to define the listening daemon on a port; the `/etc/postfix/master.cf` file usually contains a commented out example for **smtps** (port 465), this definition should be inserted right at that spot. Because we want the standard localhost:25 connection to still work for basic server things, we're explicitly passing some of the "force SSL, force login" settings here as commandline options only on this port. The **443** is the port to use, change as you see fit.
```
/etc/postfix/master.cf
443 inet n - n - - smtpd
-o smtpd_sasl_auth_enable=yes
-o smtpd_tls_security_level=encrypt
-o smtpd_tls_auth_only=yes
-o smtpd_enforce_tls=yes
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
-o smtpd_recipient_restrictions=permit_sasl_authenticated,reject_unauth_destination
-o milter_macro_daemon_name=ORIGINATING
```
Finally, restart Postfix:
```
systemctl restart postfix
```
## Local Relay Testing
Before configuring iptables to allow traffic in, testing should be performed. First you need to generate a base64 encoded version of the username and password to be used as a SMTP client, this is required for manual testing and is part of the PLAIN style SMTP specification.
```
echo -ne '\000supercoolusername\000supercoolpassword' | openssl base64
```
This should give a string of alphanumeric characters and maybe a few symbols that is used in the `AUTH PLAIN` command typed below. then try testing the process locally using the `openssl` tool. Things you type are denoted with `### TYPE THIS ###` inline - for more help on how to manually "talk SMTP" hit up Google.
```
# openssl s_client -quiet -starttls smtp -connect localhost:443
250 DSN
EHLO example.com ### TYPE THIS ###
250-mx.mydomain.org
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
AUTH PLAIN XXXXXXXXXXXXXXXXXXXX ### TYPE THIS ###
235 2.7.0 Authentication successful
MAIL FROM: someone@server.local ### TYPE THIS ###
250 2.1.0 Ok
RCPT TO: myemail@gmail.com ### TYPE THIS ###
250 2.1.5 Ok
QUIT ### TYPE THIS ###
221 2.0.0 Bye
```
The above is actually testing multiple things all at once; TLS (SSL) is working, authentication is working and relaying as an authenticated user is working.
Next, test that **not** using SSL is blocked:
```
# telnet localhost 443
Trying ::1...
Connected to localhost.
Escape character is '^]'.
220 mx.mydomain.org ESMTP Postfix
EHLO example.com
250-mx.mydomain.org
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
AUTH PLAIN XXXXXXXXXXXXXXXXXXXX
530 5.7.0 Must issue a STARTTLS command first
QUIT
221 2.0.0 Bye
```
Finally, test that you cannot try sending mail without authentication - the `RCPT TO:` should be tested against local server accounts (i.e. root@localhost) and relayed accounts (i.e. myemail@gmail.com). In all cases the **554** error should happen:
```
# openssl s_client -quiet -starttls smtp -connect localhost:443
250 DSN
EHLO example.com
250-mx.mydomain.org
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
MAIL FROM: someone@server.local
250 2.1.0 Ok
RCPT TO: myemail@gmail.com
554 5.7.1 <localhost[::1]>: Client host rejected: Access denied
QUIT
221 2.0.0 Bye
```
All of these tests ensure that it's working when SSL and authentication is used, but denied in all other cases. We specifically did not enable the local server to send email **on this port** without SSL and authentication, however localhost:25 should still work for that.
## Allowing Remote Access
As a last step, open the iptables firewall to allow port 443 usage.
> Do not open port 25 unless your server actually receives normal email from the Internet
If you already use iptables, the rule might be added like so:
```
-A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
```
If this is a brand new setup, or iptables wasn't used here's a more full-fledged example that allows 22 (SSH), 80 (HTTP) and 443 which is pretty average for a server (note: I recommend [fail2ban](http://www.fail2ban.org/wiki/index.php/Main_Page) if opening SSH):
```
/etc/sysconfig/iptables
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
/etc/sysconfig/ip6tables
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p ipv6-icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -d fe80::/64 -p udp -m udp --dport 546 -m state --state NEW -j ACCEPT
-A INPUT -j REJECT --reject-with icmp6-adm-prohibited
-A FORWARD -j REJECT --reject-with icmp6-adm-prohibited
COMMIT
```
Once this is open, perform the same testing as the Local section above, except use the DNS name of the server to connect.
For completeness sake because we love security and stopping the bad guys, here's a simple `fail2ban` configuration that will monitor the SSH port for brute force attacks and null route the evildoers:
```
/etc/fail2ban/jail.local
[DEFAULT]
ignoreip = 127.0.0.1/8
bantime = 600
findtime = 600
maxretry = 3
[sshd]
enabled = true
maxretry = 5
usedns = no
```
## Configuring mailx
One of the most common ways to send email from cron jobs, scripts and other assorted commands is using the `mail` (_mailx_) command. To configure the client systems to use this relay, a `~/.mailrc` configuration needs to be set up like so:
```
~/.mailrc
account myrelay {
set from=myname@mylaptop.local
set smtp-hostname=mylaptop.local
set smtp=smtp://supercoolusername:supercoolpassword@mx.mydomain.org:443
set smtp-use-starttls
set smtp-auth=plain
set ssl-verify=warn
set v15-compat
}
```
Then the usage is with the `-A myrelay` commandline option:
```
mail -A myrelay -s "Test email" myemail@gmail.com
```
Using other clients should work in a normal fashion with their built in configuration options, just choose `STARTTLS` in the connection options (not **direct SSL** or similar). The use of direct SSL (_smtps_) to the server is not used herein; `STARTTLS` is more universally compatible.

186
md/stunnel_setup.md Normal file
View file

@ -0,0 +1,186 @@
# Stunnel Setup
## Contents
- [Overview](#overview)
- [Conventions](#conventions)
- [Installation](#installation)
- [RHEL / CentOS 7 Server](#rhel--centos-7-as-server)
- [Ubuntu 14 as Client](#ubuntu-14-as-client)
- [Server Configuration](#server-configuration)
- [Client Configuration](#client-configuration)
- [Testing](#testing)
- [References](#references)
## Overview
[Stunnel](https://www.stunnel.org) is a SSL proxy designed to add TLS encryption to existing clients and servers without changes to the daemon's themselves. One (or more) endpoint is run in server mode, the other endpoint is run in client mode. The daemon software connects to a localhost port, the connection is proxied over the SSL tunnel, then handed to the server localhost port as defined.
### Conventions
In this article, a traditional MariaDB replication configuration will be used to exemplify use as compatible version 5.5.x is available on both distributions. One server is located in one area of the USA, the second server in another USA region, and standard public IPv4 networking to connect the two servers. Two different Linux distributions will be used to verify the technology is agnostic.
- s1 = CentOS 7 as server, IP 1.2.3.4
- s2 = Ubuntu 14 as client, IP 9.8.7.6
Actual public IPs would be used in implementation as appropriate. This article will not cover setting up MariaDB replication as it's a standard, by the book process however two notes:
1. On the MariaDB master `GRANT REPLICATION SLAVE ON ...` stanza for the user, use `<user>@localhost` not the actual IP of the remote slave like you would normally.
2. For both master and slave MariaDB instances, implement `bind-address = 127.0.0.1` to lock the daemons to localhost.
3. Remember to use `MASTER_HOST='localhost'` in your `CHANGE MASTER TO ...` stanza on the slave to match the user created in note 1
MariaDB traffic will travel over the stunnel proxy, so they should not listen on the public IPs for security best practices.
## Installation
The `stunnel` package may be a part of the base distribution or it may be required to use a third party repository such as EPEL or a PPA to obtain. In these examples with CentOS 7 and Ubuntu 14 the package is readily available for both in the base repositories.
### RHEL / CentOS 7 as Server
First install the base package:
```
yum install stunnel
```
Next, create a user and directories to run the software - on this platform the RPM package does not create the user or directories:
```
# the user and directory for immediate use:
useradd -r -m -d /var/run/stunnel -s /bin/false stunnel
# the tmpfiles.d configuration to recreate the directory on reboot:
echo "d /var/run/stunnel 0770 stunnel stunnel -" > /etc/tmpfiles.d/stunnel.conf
```
Lastly, create the [systemd](systemd_mechanics.md) unit file to run stunnel as a service:
```
cat << XYZZY > /etc/systemd/system/stunnel.service
[Unit]
Description=SSL tunnel for network daemons
After=syslog.target
[Service]
ExecStart=/usr/bin/stunnel
Type=forking
[Install]
WantedBy=multi-user.target
XYZZY
```
### Ubuntu 14 as Client
First install the base package:
```
apt-get update && apt-get install stunnel
```
Next, update /etc/default/stunnel4 to enable it at boot:
```
sed -i -e 's/^ENABLED=0/ENABLED=1/' /etc/default/stunnel4
```
Ubuntu creates the `stunnel4` user and group, and `/var/run/stunnel4` directory as part of the package.
## Server Configuration
First, create a basic self-signed certificate to use on the server; if a real SSL cert is available from a certificate authority it can be used, however a self-signed cert works for the basic point-to-point setup.
```
openssl req -new -newkey rsa:2048 -days 3650 \
-nodes -x509 -sha256 \
-subj '/CN=127.0.0.1/O=localhost/C=US' \
-keyout /etc/stunnel/stunnel.pem \
-out /etc/stunnel/stunnel.pem
```
Next, create the stunnel server oriented config file; in our example we're using MariaDB so we'll choose the ports accordingly to have stunnel accept the connection on the public IP port 3307, then pass the connection to the localhost port 3306:
```
/etc/stunnel/stunnel.conf
chroot = /var/run/stunnel
setuid = stunnel
setgid = stunnel
pid = /stunnel.pid
fips = no
[mysql]
client = no
accept = 1.2.3.4:3307
connect = 127.0.0.1:3306
cert = /etc/stunnel/stunnel.pem
key = /etc/stunnel/stunnel.pem
# stunnel 4.53 (Ubuntu 14) only supports TSLv1 not TLSv1.2
# stunnel 4.56 (CentOS 7) supports both TLSv1 and TSLv1.2
sslVersion = TLSv1
```
Last, as appropriate open up an iptables/firewalld/ufw rule that allows the client to connect on port 3307; a very basic `iptables` rule with no port restrictions would look like:
```
-A INPUT -p tcp -m tcp -s 9.8.7.6 --dport 3307 -j ACCEPT
```
Tailor the ACL on your firewall(s) as needed to meet your desired security posture. Assuming MariaDB is up and running, start and enable the stunnel service:
```
systemctl start stunnel
```
## Client Configuration
The client does not require a SSL certificate; create the client oriented config file that accepts a connection on local port 3307 and talks to the remote stunnel on 3307:
```
/etc/stunnel/stunnel.conf
chroot = /var/run/stunnel4
setuid = stunnel4
setgid = stunnel4
pid = /stunnel.pid
[mysql]
client = yes
accept = 127.0.0.1:3307
connect = 1.2.3.4:3307
# stunnel 4.53 (Ubuntu 14) only supports TSLv1 not TLSv1.2
# stunnel 4.56 (CentOS 7) supports both TLSv1 and TSLv1.2
sslVersion = TLSv1
```
Once again, assuming the basic MariaDB is up and running, start the stunnel service:
```
service stunnel start
```
## Testing
From the Ubuntu client, use the standard mysql command to connect to remote instance and observe the hostname (or some other test of your design) to ensure you're connecting to the remote MariaDB, not local:
```
root@s2:~# mysql -p --host=127.0.0.1 --port=3307 -e "show variables like '%hostname%';"
Enter password:
+---------------+----------+
| Variable_name | Value |
+---------------+----------+
| hostname | s1.local |
+---------------+----------+
```
Given that it connected, you are clear to proceed with configuring normal replication between the two instances.
## References
- <http://www.freedesktop.org/software/systemd/man/tmpfiles.d.html>

529
md/system_backup.md Normal file
View file

@ -0,0 +1,529 @@
# System Backup
## Contents
- [Overview](#overview)
- [Installation](#installation)
- [fcron](#fcron)
- [duplicity / boto / gnupg / duply](#duplicity--boto--gnupg--duply)
- [Arch](#arch)
- [CentOS](#centos)
- [gsutil](#gsutil)
- [Setup](#setup)
- [Google Cloud Storage](#google-cloud-storage)
- [Enable GCS](#enable-gcs)
- [Create a Project](#create-a-project)
- [Interoperable Access](#interoperable-access)
- [Configure gsutil](#configure-gsutil)
- [Create a Bucket](#create-a-bucket)
- [GnuPG](#gnupg)
- [Creating a key](#creating-a-key)
- [Migrating a key](#migrating-a-key)
- [Import Preferences](#import-preferences)
- [Duply](#duply)
- [conf](#conf)
- [exclude](#exclude)
- [Backup](#backup)
- [Cleanup](#cleanup)
- [Scripted Backups](#scripted-backups)
- [Scheduled Backups](#scheduled-backups)
- [Manual Recovery](#manual-recovery)
- [Backend Portability](#backend-portability)
- [References](#references)
## Overview
There are a million and one ways to configure backups, everything is situationally dependent - for our purposes, we have these challenges and needs as a laptop user, cloud server administrator or just a [FOSS](https://en.wikipedia.org/wiki/Free_and_open-source_software) advocate:
- Random uptime, laptops suspend/hibernate frequently
- Slow upload bandwidth on home DSL-style networks or spotty wifi
- Avoid vendor lock-in on both the client and server side
- Portability on the client and server side between distributions
- Reduce server administration - use a cloud files/storage provider
- Include standardized encryption algorithms in the backups
- Ability to tune backups granularly on a per-case basis
- Ability to verify backups and restore, even if it's a manual action (worst case)
Given these principles, the solution being implemented will use:
- [fcron](http://fcron.free.fr/) for running missed crons upon wakeup
- [GnuPG](http://www.gnupg.org/) for security/encryption
- [duplicity](http://www.nongnu.org/duplicity/) as the main engine
- [duply](http://duply.net/) to make using duplicity easier
- [boto](https://github.com/boto/boto) (python-boto) for bucket storage:
- [Google Cloud Storage](https://cloud.google.com/products/cloud-storage/)
- Amazon [S3](http://aws.amazon.com/s3/) or [Glacier](http://aws.amazon.com/glacier/)
More specifically this article is using [Arch Linux](https://www.archlinux.org/) and Google Cloud Storage with [Durable Reduced Availability Storage](https://developers.google.com/storage/docs/durable-reduced-availability) to reduce costs to a trivial (truly - \< $5/mo, see the [pricing](https://developers.google.com/storage/pricing)) amount by excluding things like Downloads and Music. There are better ways to back up data like that, what you're really after are the things that matter (pictures, documents, configs, etc.) that can't be replaced.
All software is available in every distro (within reason), just replace the Arch pacman installs with yum/apt-get/emerge/etc. as appropriate. You may wish to consider using a custom PPA (ppa:duplicity-team/ppa) or similar source if your distribution's mainline version is too old.
## Installation
These actions are typically performed as root or with sudo - adjust as needed.
### fcron
Like `anacron`, `fcron` assumes the computer is not always running and, unlike `anacron`, it can schedule events at intervals shorter than a single day which is useful for systems which suspend/hibernate regularly. For an always-on system like a cloud server there's no real need to replace the standard cron package, however.
When replacing cronie with fcron be aware the spool directory is `/var/spool/fcron` and the `fcrontab` command is used instead of `crontab` to edit the user crontabs. These crontabs are stored in a binary format with the text version next to them as _foo_.orig in the spool directory. Any scripts which manually edit user crontabs may need to be adjusted due to this difference in behavior.
A quick scriptlet which will replace cronie and convert traditional user crontabs to fcron format:
```
systemctl stop cronie; systemctl disable cronie
pacman -Sy; pacman -S fcron
cd /var/spool/cron && (
for ctab in *; do
fcrontab ${ctab} -u ${ctab}
done
)
systemctl start fcron; systemctl enable fcron
```
### duplicity / boto / gnupg / duply
Duplicity has minimal dependencies for a python application, no need to pip install a lot of extra modules. Many other backends are available for use however we are only focusing on GCS and/or S3 here. If a `duply` package is not available in your distribution, it's a single bash script - visit the [website](http://duply.net/) and grab a copy, place it somewhere handy (such as ~/bin/duply) as needed.
> **Duplicity 0.6.22 or newer is required for Google Cloud Storage**
#### Arch
These are in the default repositories - most folks have gnupg already installed:
```
pacman -Sy; pacman -S duplicity python2-boto gnupg --noconfirm
```
Duply is currently in [AUR](https://aur.archlinux.org/) - many folks use [pacaur](https://aur.archlinux.org/packages/pacaur/) (which uses [cower](https://aur.archlinux.org/packages/cower/) for the heavy lifting):
```
pacaur -S duply
```
#### CentOS
These are split between the main repositories and [EPEL](https://fedoraproject.org/wiki/EPEL) repo:
> It's common the CentOS/EPEL repositories are behind in versions - I recommend downloading the SRPMs from EPEL and rebuild new RPMs using the latest versions of the duplicity, duply and python-boto packages.
```
yum -y install duplicity python-boto duply gnupg
```
### gsutil
While not strictly necessary, having `gsutil` installed and ready makes working with unexpected issues easy; for example, while working your your Excludes you may try several times until you get it perfect -- `gsutil` makes it easy to delete en-mass or list bucket contents. Highly recommended - use a `tools/` subdirectory for things like this:
```
[ ! -d ~/tools ] && mkdir ~/tools; cd ~/tools
wget https://storage.googleapis.com/pub/gsutil.tar.gz
tar -zxf gsutil.tar.gz; rm gsutil.tar.gz
# Arch default is python v3, gsutil needs python v2
sed -i.orig '1 s/python/python2/g' gsutil/gsutil
```
## Setup
These actions are all performed as yourself, not root, for a typical backup of a home directory. This is where your usage pattern of a Linux system comes into play -- I personally have a ~/System/ directory where I copy any config change made outside my home directory (i.e. system level). Keep your home directory on a separate partition or Logical Volume and encrypt it with LUKS.
This makes your home directory the source of all evil and allows a copy of the home directory to a new machine, a fresh laptop install, or even moving from one distro to another very contained. It also makes backups trivial - backup your home directory, the rest is disposable and easier re-installed rather than restored.
This same method can apply to a cloud server -- implement a methodology so that all your routine database dumps, git trees and so forth are all parented under a single higher directory instead of scattered about the filesystem. When making edits to system level files such as `/etc/httpd/conf/httpd.conf`, `/etc/php.ini`, `/etc/my.cnf`, cron tasks and so on copy them (or just use symlinks) over to a collected location under one tree for easier backup and restore. This has a side effect of making your whole infrastructure portable to another server with little work.
### Google Cloud Storage
This section is always subject to change as it depends on the Google web links in question - they frequently update and shift things around. So as an overview, our mission is to:
1. Enable Google Cloud Storage for your Google account
2. Set up a Project and attach billing (credit card) to it
3. Enable Interoperable Access and generate Storage Access Keys
4. Configure gsutil for random operations
5. Set up a Bucket for each backup (i.e. laptop)
The first four are one-time only, the last one is repeated for each backupset you configure with duply/duplicity later. The setup generated by this section is then easily copied to another laptop for use on a second backup, etc.
#### Enable GCS
Log into your Google account, then visit <https://cloud.google.com/products/cloud-storage/> and click the "Get Started" or "Go to my console" (or similar) link usually at the top and follow any instructions to get the basics set up. You may have to agree to Terms of Service and all that jazz - do the needful.
#### Create a Project
A Project is the higher level umbrella where you will be billed for usage; as of this writing the URL is at
<https://console.developers.google.com/project> and you click Create Project and choose a meta-level name like **Backups** (not a specific laptop name, e.g.). This will create a `PROJECT ID` displayed, jot that down in your notepad for use later.
On the left of Google's console, click **Billing** to connect the new Project to your credit card -- follow the instructions as appropriate.
#### Interoperable Access
This part can be the most confusing, as Google's webUI seems to be in flux a lot and the exact links change. As of this writing, the way to access the area:
1. Click into your Project from the console
2. On the left, click Storage then Cloud Storage
3. Click Project Dashboard which opens a new tab
You're in a different UI at this point - on the left should be **Google Cloud Storage** with two sub-menus Storage Access and Interoperable Access. Enable Interoperable Access, then generate new Keys and jot down in the notepad both parts of the Key (one is secret). It's possible these two links may work for you:
- [Enable Interoperable Access](https://code.google.com/apis/console#:storage)
- [Generate Interoperable Storage Access Keys](https://code.google.com/apis/console#:storage:legacy)
#### Configure gsutil
Use the `gsutil` tool to generate a default configuration:
```
cd ~/tools/gsutil
./gsutil config -a
```
This creates the `~/.boto` file which has a lot of comments. You need to insert the Project ID and Google keypair from the above steps. If you strip out the comments, here's the required portions - replace AAA, BBB and ZZZ with your data:
```
[Credentials]
gs_access_key_id = AAAAAAAAAAAAAAAAAAAA
gs_secret_access_key = BBBBBBBBBBBBBBBBBBBB
[Boto]
https_validate_certificates = True
[GSUtil]
content_language = en
default_api_version = 1
default_project_id = ZZZZZZZZZZZZ
[OAuth2]
```
#### Create a Bucket
A bucket is where the encrypted tarballs will actually be stored - so you want a Bucket for each backupset (system) you'll configure. A good choice might be the short name of the laptop for example, "mylaptop" will be used here as an example. We'll enable Durable Reduced Availability on the bucket to save money as well. The `gsutil` tool can perform many actions, just run `./gsutil --help` and investigate.
```
cd ~/tools/gsutil
./gsutil mb -c DRA gs://mylaptop/
```
> The bucket name is global to all users, you may get an error if the name chosen is already in use.
### GnuPG
Generate a standard key specifically for use with your backups; because the password will be stored in plaintext in the duply config in your home directory, create a new key and use a unique password and not one of your existing keys. This keypair can then be copied to the other systems for backup encryption with a common key.
#### Creating a key
Create a standard GPG key:
```
GnuPG 2.0 and earlier
echo "pinentry-program /usr/bin/pinentry-curses" >> ~/.gnupg/gpg-agent.conf
GPG_AGENT_INFO=""; gpg --gen-key
GnuPG 2.1 and later
echo "allow-loopback-pinentry" >> ~/.gnupg/gpg-agent.conf
gpg --gen-key
```
> Creating a key requires _entropy_ to be generated by the system. If this is a virtual instance (i.e. Virtualbox guest) consider installing `rng-tools` and starting the `rngd` daemon to provide the required entropy. Using the `haveged` daemon is an alternate option to `rngd` as well.
Check the key is available:
```
$ gpg --list-keys QQQQQQQQ
pub 2048R/QQQQQQQQ 2014-07-18
uid [ultimate] duply <duply@localhost>
sub 2048R/RRRRRRRR 2014-07-18
$ gpg --list-secret-keys QQQQQQQQ
sec 2048R/QQQQQQQQ 2014-07-18
uid duply <duply@localhost>
ssb 2048R/RRRRRRRR 2014-07-18
```
#### Migrating a key
If you are already using a key on another system, it can be exported and imported so that all your backups upstream are encrypted with the same key. First, export the public and private keys on the source and copy them to the new system:
```
gpg --export -a QQQQQQQQ > duply_public.asc
gpg --export-secret-keys -a QQQQQQQQ > duply_secret.asc
scp duply*.asc user@remote:
```
On the new device, import the keys:
```
gpg --import duply_public.asc
gpg --import duply_secret.asc
```
Finally, edit the key and set trust to Ultimate:
```
gpg --edit-key QQQQQQQQ
Command> trust
[...]
5 = I trust ultimately
Your decision? 5
```
#### Import Preferences
During import, if your source GPG secret key is newer than the destination GPG you may get an error about incompatible preferences, and an offer to fix them - choose Yes:
```
$ gpg --import duply_secret.asc
gpg: key QQQQQQQQ: secret key imported
gpg: key QQQQQQQQ: "duply <duply@localhost>" not changed
gpg: WARNING: key QQQQQQQQ contains preferences for unavailable algorithms on these user IDs:
gpg: "duply <duply@localhost>": preference for cipher algorithm 1
gpg: it is strongly suggested that you update your preferences and
gpg: re-distribute this key to avoid potential algorithm mismatch problems
Set preference list to:
Cipher: AES256, AES192, AES, CAST5, 3DES
Digest: SHA256, SHA1, SHA384, SHA512, SHA224
Compression: ZLIB, BZIP2, ZIP, Uncompressed
Features: MDC, Keyserver no-modify
Really update the preferences? (y/N) y
```
For the curious, this usually means the newer version of GPG supports a newer cipher or digest that the older one needs to remove; for example a key on the newer GPG contains the IDEA cipher:
```
source GPG 2.0.26
gpg> showpref
[ultimate] (1). duply <duply@localhost>
Cipher: AES256, AES192, AES, CAST5, 3DES, IDEA
Digest: SHA256, SHA1, SHA384, SHA512, SHA224
Compression: ZLIB, BZIP2, ZIP, Uncompressed
Features: MDC, Keyserver no-modify
destination GPG 2.0.14
Command> showpref
[ultimate] (1). duply <duply@localhost>
Cipher: AES256, AES192, AES, CAST5, 3DES
Digest: SHA256, SHA1, SHA384, SHA512, SHA224
Compression: ZLIB, BZIP2, ZIP, Uncompressed
Features: MDC, Keyserver no-modify
```
### Duply
Generate a default configuration for the backupset - we'll use the same name as the laptop and Bucket, "mylaptop":
```
duply mylaptop create
```
This creates two files that need edited:
```
~/.duply/mylaptop/conf
~/.duply/mylaptop/exclude
```
There are two other files that can be configured, `pre` and `post` that run commands before and after a duply backup. These are not created by default, however might come in handy if you need to mount/umount a filesystem, dump a database, etc. as part of the process. See the duply documentation for further info.
#### conf
Very similar to the gsutil setup, you'll need to configure the GCS data in this file for storing your backups, as well as all the other settings as to what should be backed up, retention periods and so forth. This part is situationally dependent -- I choose to manage my own Full backups manually since they require over 9 hours to upload and I need to disable Suspend on my laptop. Given that, my configuration looks like so (without all the comments):
> The use of `GPG_OPTS='--pinentry-mode loopback'` is required for GnuPG 2.1 and later, along with the above setting in `~/gnupg/gpg-agent.conf` `allow-loopback-pinentry`. Failure to configure these will result in the passphrase not working from an unattended mode.
> With duply 1.10 and above, do not set `TARGET_USER` and `TARGET_PASS` in this config file - they are now configured elsewhere, see below.
```
GPG_KEY='XXXXXXXX'
GPG_PW='YYYYYYYYY'
GPG_OPTS='--pinentry-mode loopback'
TARGET='gs://mylaptop'
TARGET_USER='AAAAAAAAAAAAAAAAAAAA'
TARGET_PASS='BBBBBBBBBBBBBBBBBBBB'
SOURCE='/home/CCCCCC'
FILENAME='.duplicity-ignore'
DUPL_PARAMS="$DUPL_PARAMS --exclude-if-present '$FILENAME'"
MAX_AGE=2M
MAX_FULL_BACKUPS=2
```
...where you're obviously replacing AAA, BBB, CCC, XXX and YYY with your information as created above. This file should be mode 0600 so that only you can read it, as it contains both your GPG key password and GCS access keypair.
#### exclude
Configure the exclude file to ignore things you do not want in the backup - it uses globs (wildcards) to make things a little easier. As an average MATE desktop user with the typical applications, here is a basic exclude file that tends to work as a good starting point:
```
- /home/*/Downloads
- /home/*/Misc
- /home/*/Movies
- /home/*/Music
- /home/*/VirtualBox**
- /home/*/abs
- /home/*/builds
- /home/*/tools/android-sdk-linux
- /home/*/tools/jdk**
- /home/*/.ICEauthority
- /home/*/.Xauthority
- /home/*/.adobe
- /home/*/.android/cache
- /home/*/.cache
- /home/*/.cddb*
- /home/*/.config/**metadata*
- /home/*/.config/*/sessions
- /home/*/.config/*session*
- /home/*/.config/VirtualBox
- /home/*/.config/libreoffice
- /home/*/.config/pulse
- /home/*/.gstreamer*
- /home/*/.hplip
- /home/*/.icons
- /home/*/.java/deployment
- /home/*/.java/fonts
- /home/*/.local/share/gvfs-metadata
- /home/*/.local/share/icons
- /home/*/.macromedia
- /home/*/.mozilla/firefox/Crash**
- /home/*/.mozilla/firefox/*/storage
- /home/*/.purple/icons
- /home/*/.thumbnails
- /home/*/.thunderbird/**.msf
- /home/*/.thunderbird/Crash**
- /home/*/.xsession-errors*
```
Everyone will have a slight variation on this file, adjust as needed. It's a bit difficult to get globbing to work right with dot-files so I tend to just avoid that specific pattern usage.
## Backup
This part is dead simple - just run duply with the name of the backupset; it will detect it's the first time run and trigger a full backup. Be sure to use `screen`, disable suspend/hibernate, etc. if your backup is going to take a really long time:
```
duply mylaptop backup
```
> With duply 1.10 and above, you must first export the environment variables with your Google Cloud Storage API credentials. As such, skip to the "Scripted Backup" section below to see what's needed.
You may wish to increase verbosity and/or add `--dry-run` to the `conf` file the first time to ensure what you think it happening is actually happening. The default conf generated has the options and instructions present to set those up. For normal usage just use the default verbosity level.
From this point, every time you run `duply backup foo` it will detect the full backup and run Incrementals instead; how much and how long it takes depends on your changeset each time. You can force a full or incremental by using `full` or `incr` instead of `backup` as a command as well. The `bkp` action will skip the pre/post files execution.
## Cleanup
Duply has several actions to help keep track of your backups - `verify` to show changed local files since the backup, `status` to list the upstream full and incremental statistics, and various forms of `purge` to flush older backups. Keep in mind that you cannot purge incrementals until a full backup supersedes them, so there's a bit of an art to knowing when you should generate a new full backup.
For a laptop it's probably sufficient to perform a full backup once a month (or less) and roll with the incrementals, unless you have massive amounts of change. For a scenario where you're backing up databases and such it might make sense to perform full backups weekly, since the size of your incrementals will grow rapidly and consume space. After a full backup, purge the incrementals older than that full backup to save space.
```
duply mylaptop status
duply mylaptop full
duply mylaptop purge
duply mylaptop purge --force
```
The purge actions will respect the settings in `conf` as outlined above. From time to time you may wish to use the `cleanup` action that will attempt to find orphaned backup bits and `fetch` to randomly test restoring files from the online backups to ensure everything is working as intended.
## Scripted Backups
Create a small shell script that will be run via fcron to perform the backup, check status and email the results to yourself from the saved logfile. I use a very basic script with `mailx` (the standard commandline `mail`):
```
#!/bin/bash
MAILTO="me@mydomain.com"
LOGDIR="/home/CCCCCC/.logs"
TIMESTAMP=$(date +%Y-%m-%d_%H%M)
MAILSUB="mylaptop backup report: ${TIMESTAMP}"
LOGFILE="${LOGDIR}/duply_${TIMESTAMP}.log"
# Duply 1.10+ requires ENV vars
export GS_ACCESS_KEY_ID='<my API user, the old TARGET_USER in duply>'
export GS_SECRET_ACCESS_KEY='<my secret key, the old TARGET_PASS in duply>'
echo "" >> ${LOGFILE}
duply mylaptop backup 1>>${LOGFILE} 2>&1
duply mylaptop status 1>>${LOGFILE} 2>&1
echo "" >> ${LOGFILE}
cat ${LOGFILE} | mail -s "${MAILSUB}" ${MAILTO}
find "${LOGDIR}" -type f -mtime +30 -delete
exit 0
```
## Scheduled Backups
Insert the above script into your fcrontab using the `%daily` keyword; fcron will run any missed jobs on the next hour after the system is online when `0` is specified in the minutes field and `*` in the hours:
```
%daily,mail(no) 0 * /home/CCCCCC/bin/mylaptopduply.sh
```
See the `fcrontab(5)` man page for more information.
## Manual Recovery
The encrypted backup files are uploaded in chunks of 25M by default (configurable in the conf file of duply backupset); these files can be downloaded using a web browser from GCS, decrypted and untarred manually in the worst case scenario. **The GPG key is still required** so be sure that your `~/.gnupg` keychain is backed up in some fashion not inside your duply/duplicity backups. Assuming the GPG key used to encrypt is still available:
1. Go to the GCS console in a web browser
2. Click the Project, Storage, Cloud Storage, Storage Browser
3. Click into the Bucket, find a file you think might have what you need
4. Download the file to your local system
Once downloaded, decrypt it (you will be prompted for the GPG secret key password) and untar:
```
gpg -d duplicity-inc.20140802T010001Z.to.20140802T170542Z.vol1.difftar.gpg > recover.tar
tar -xf recover.tar
```
This process is really only for recovery in a disaster; as the files upstream are chunked tarballs it's a rather random process to know which backup-file might have the specific file you need. There are manifests and signatures upstream as well, so downloading those first and perusing might help.
```
gpg -d duplicity-inc.20140802T010001Z.to.20140802T170542Z.manifest.gpg > recover.manifest
less recover.manifest
```
It would definitely be quicker to download all the manifests first, decrypt them then just grep the files to find the target; then you can download the backup-file in question. If your `gsutil` is working it can be used instead of a browser:
```
mkdir ~/recover; cd ~/recover
../tools/gsutil/gsutil cp gs://mylaptop/*.manifest.gpg .
for ii in *.gpg; do gpg -d "${ii}" > "${ii%%.gpg}"; done;
grep "some filename" *.manifest
```
It would be easier to set up another Linux instance and use duply/duplicity to recover the data properly (and quicker), but the option here is available to do it all by hand.
## Backend Portability
An extension of the Manual Recovery could be to copy all GCS files down to a local path using `gsutil` (or web browser), copy/upload them to a filesystem or different provider then reconfigure your duply config to use the new backend without losing your existing backups. This might even be used just to download a copy of everything and save onto a USB drive that is kept in a fireproof safe.
Given the duplicity gpg-tarball storage design, your solution is upstream provider independent - the backup files can be ported from one backend to another with a bit of scripting and elbow grease. This could also be leveraged to keep a backup on different providers at the same time or use different backends for short vs. long term storage.
## References
- <http://verb.bz/2013/08/29/backup-to-google-cloud-storage-using-duplicity-0-6-22/>
- <http://www.willdurness.com/back-up-your-vm-to-google-cloud-storage-using-duplicity/>
- <https://wiki.archlinux.org/index.php/Cron#Fcron>
- <https://wiki.archlinux.org/index.php/Duplicity>

389
md/systemd_mechanics.md Normal file
View file

@ -0,0 +1,389 @@
# systemd Mechanics
## Contents
- [Overview](#overview)
- [Configuration](#configuration)
- [Common Tools](#common-tools)
- [Boot Parameters](#boot-parameters)
- [Runlevels/Targets](#runlevelstargets)
- [Manipulating Runlevels](#manipulating-runlevels)
- [Manipulating Services](#manipulating-services)
- [Customizing Services](#customizing-services)
- [Instanced Units (.service, .socket, etc...)](#instanced-units-service-socket-etc)
- [Path Units (.path)](#path-units-path)
- [Mount Units (.mount)](#mount-units-mount)
- [Example Bind Mount - /var/tmp to /tmp](#example-bind-mount---vartmp-to-tmp)
- [Service Resource Limits](#service-resource-limits)
- [Service Types](#service-types)
- [Operational Commands](#operational-commands)
- [Power Management](#power-management)
- [Journal Commands](#journal-commands)
- [Notes](#notes)
- [References](#references)
## Overview
From the [systemd project homepage](http://www.freedesktop.org/wiki/Software/systemd/):
> `systemd` is a system and service manager for Linux, compatible with SysV and LSB init scripts. systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux cgroups, supports snapshotting and restoring of the system state, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic. It can work as a drop-in replacement for sysvinit.
## Configuration
| **File / Directory** | **Function** |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `/etc/hostname` | The host name for the system (yes, it's part of systemd now) |
| `/etc/os-release` | Standardization of the various distribution ID files like /etc/redhat-release and similar |
| `/etc/vconsole.conf` | Configuration of the default keyboard mapping and console font |
| `/etc/locale.conf` | Configuration of the system-wide locale |
| `/etc/machine-id` | Machine ID file, superseding D-Bus' machine ID file. Guaranteed to exist and be valid on a systemd system |
| `/etc/machine-info` | Metadata about a host, like a pretty host name and an icon name; maintained by systemd-hostnamed |
| `/etc/systemd/system.conf` | Read when systemd is run as a system instance |
| `/etc/systemd/user.conf` | Read when systemd is run as a non-system service |
| `/etc/systemd/journald.conf` | Configures the systemd-journald system service |
| `/etc/systemd/logind.conf` | Configures various parameters of the systemd login manager systemd-logind |
| `/etc/modules-load.d/*.conf` | Drop-in directory for kernel modules to statically load at boot |
| `/etc/sysctl.d/\*.conf` | Drop-in directory for kernel sysctl parameters, extending /etc/sysctl.conf |
| `/etc/tmpfiles.d/*.conf` | Drop-in directory for configuration of runtime files that need to be removed/created/cleaned up at boot and during uptime |
| `/etc/binfmt.d/*.conf` | Drop-in directory for registration of additional binary formats for systems like Java, Mono and WINE |
| `/etc/systemd/` | Main configuration directory for customizations |
| `/var/log/journal/` | Location of the systemd-journald persistent log data |
| `/usr/lib/systemd/` | Main directory of distro-provided systemd components |
| `/var/lib/systemd/` | Main directory of runtime systemd components |
## Common Tools
Primary commands used in the day-to-day systemd world:
| **Command** | **Purpose** |
| ---------------- | --------------------------------------------------------------------------------------------- |
| `systemctl` | Used to introspect and control the state of the systemd system and service manager |
| `journalctl` | Query the contents of the systemd journal as written by systemd-journald.service |
| `hostnamectl` | Used to query and change the system hostname and related settings |
| `timedatectl` | Query and change the system clock and its settings |
| `localectl` | Control the system locale and keyboard layout settings |
| `loginctl` | Control the systemd login manager systemd-logind |
| `systemd-cat` | Connect a pipeline or program's output with the journal (like **logger**) |
| `systemd-cgls` | Recursively shows the contents of the selected Linux control group hierarchy in a tree |
| `systemd-cgtop` | Show top control groups by their resource usage; useful when using cgroups to limit resources |
| `systemd-delta` | Identify and compare configuration files in /etc that override default counterparts in /usr |
| `kernel-install` | Add and remove kernel and initramfs images to and from /boot |
## Boot Parameters
On boot **systemd** activates (by default) the target unit `default.target` whose job is to activate services and other units by pulling them in via dependencies. To override the unit to activate, **systemd** parses its own kernel command line arguments via the `systemd.unit=` command line option. This may be used to temporarily boot into a different boot unit. The classic runlevels are replaced as following:
| **Parameter** | **Purpose** |
| -------------------------------- | --------------------------------------------------------------------------------------------------- |
| `systemd.unit=rescue.target` | A special target unit for setting up the base system and a rescue shell (**similar to runlevel 1**) |
| `systemd.unit=emergency.target` | Very similar to passing `init=/bin/sh` but with the option to boot the full system from there |
| `systemd.unit=multi-user.target` | Setting up a non-graphical multi-user system |
| `systemd.unit=graphical.target` | Setting up a graphical login screen |
| **Parameter** | **Type** | **Default** | **Purpose** |
| ------------------------ | ------------- | --------------- | --------------------------------------------------------------------------------------------------- |
| `systemd.unit=` | string | default.target | Overrides the unit to activate on boot. Example: rescue.target or emergency.target |
| `systemd.dump_core=` | boolean | true | If true systemd dumps core when it crashes. Otherwise no core dump is created |
| `systemd.crash_shell=` | boolean | false | If true systemd spawns a shell when it crashes. Otherwise no core dump is created |
| `systemd.crash_chvt=` | int | \-1 | If positive systemd activates the specified virtual terminal when it crashes |
| `systemd.confirm_spawn=` | boolean | false | If true asks for confirmation when spawning processes |
| `systemd.show_status=` | boolean | true | If true shows terse service status updates on the console during bootup |
| `systemd.sysv_console=` | boolean | true | If true output of SysV initscripts will be directed to the console |
| `systemd.log_target=` | string | journal-or-kmsg | Set log target. One of: console, journal, syslog, kmsg, journal-or-kmsg, syslog-or-kmsg, null |
| `systemd.log_level=` | int, constant | info | Set log level. Numerical log level or one of: emerg, alert, crit, err, warning, notice, info, debug |
| `systemd.log_color=` | boolean | true | Highlight important log messages |
| `systemd.log_location=` | boolean | true | Include code location in log messages. This is mostly relevant for debugging purposes |
For details about these special systemd boot units, view the [systemd.special](http://www.freedesktop.org/software/systemd/man/systemd.special.html) man page.
## Runlevels/Targets
Systemd has a concept of _targets_ which serve a similar purpose as runlevels but act a little different. Each _target_ is named instead of numbered and is intended to serve a specific purpose. Some _targets_ are implemented by inheriting all of the services of another _target_ and adding additional services to it. There are systemd _targets_ that mimic the common sysvinit runlevels so you can still switch _targets_ using the familiar `telinit RUNLEVEL` command. The runlevels that are assigned a specific purpose on vanilla systemd-enabled RHEL/Fedora installs; 0, 1, 3, 5, and 6; have a 1:1 mapping with a specific systemd *target*.
| **SysV Runlevel** | **systemd Target** | **Notes** |
| ----------------- | ----------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| 0 | runlevel0.target, poweroff.target | Halt the system. |
| 1, s, single | runlevel1.target, rescue.target | Single user mode. |
| 2, 4 | runlevel2.target, runlevel4.target, multi-user.target | User-defined/Site-specific runlevels. By default, identical to 3. |
| 3 | runlevel3.target, multi-user.target | Multi-user, non-graphical. Users can usually login via multiple consoles or via the network. |
| 5 | runlevel5.target, graphical.target | Multi-user, graphical. Usually has all the services of runlevel 3 plus a graphical login. |
| 6 | runlevel6.target, reboot.target | Reboot |
| emergency | emergency.target | Emergency shell |
## Manipulating Runlevels
| **SysVinit Command** | **systemd Command** | **Notes** |
| ------------------------------- | -------------------------------------------- | --------------------------------------------- |
| `grep initdefault /etc/inittab` | `systemctl list-units --type=target` | Show current runlevel |
| `telinit 3` | `systemctl isolate multi-user.target` | Change to multi-user runlevel |
| `vi /etc/inittab` (initdefault) | `systemctl enable multi-user.target --force` | Set to use multi-user runlevel on next reboot |
## Manipulating Services
Note that multiple service can be specified after the command, so `systemctl start foo.service bar.service baz@foo.service` is a valid command.
| **SysVinit Command** | **systemd Command** | **Notes** |
| ------------------------------ | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `service (foobar) start` | `systemctl start (foobar).service` | Used to start a service (not reboot persistent) |
| `service (foobar) stop` | `systemctl stop (foobar).service` | Used to stop a service (not reboot persistent) |
| `service (foobar) restart` | `systemctl restart (foobar).service` | Used to stop and then start a service |
| `service (foobar) reload` | `systemctl reload (foobar).service` | When supported, reloads the config file without interrupting pending operations. |
| `service (foobar) condrestart` | `systemctl condrestart (foobar).service` | Restarts if the service is already running. |
| `service (foobar) status` | `systemctl status (foobar).service` | Tells whether a service is currently running. |
| `ls /etc/rc.d/init.d/` | `systemctl list-unit-files --type=service` | Used to list the services that can be started or stopped; Used to list all the services and other units |
| `chkconfig (foobar) on` | `systemctl enable (foobar).service` | Turn the service on, for start at next boot, or other trigger. |
| `chkconfig (foobar) off` | `systemctl disable (foobar).service` | Turn the service off for the next reboot, or any other trigger. |
| `chkconfig (foobar)` | `systemctl is-enabled (foobar).service` | Used to check whether a service is configured to start or not in the current environment. |
| `chkconfig --list` | `systemctl list-unit-files --type=service` | Print a table of services that lists which runlevels each is configured on or off |
| `chkconfig (foobar) --list` | `ls /etc/systemd/system/*.wants/(foobar).service` | Used to list what levels this service is configured on or off |
| `chkconfig (foobar) --add` | `systemctl daemon-reload` | Used when you create a new service file or modify any configuration |
> All `/sbin/service` and `/sbin/chkconfig` lines listed above continue to work on Fedora (and likely RHEL7), and will be translated to native equivalents as necessary. The only exception is chkconfig --list.
> In SysVinit, services can define arbitrary commands. Examples would be `service iptables panic`, or `service httpd graceful`. Native systemd services do not have this ability; any service that defines an additional command in this way would need to define some other - service-specific - way to accomplish this task when writing a native systemd service definition. Check the package-specific release notes for any services that may have done this.
## Customizing Services
The best way to customize unit files is to add `/etc/systemd/system/`foobar.service.d`/*.conf` where _foobar.service.d_ is the name of the service you want to customize. If a directory doesn't already exist, create one and add a conf file with the settings you want to override.
- [systemd.unit](http://www.freedesktop.org/software/systemd/man/systemd.unit.html)
- [systemd.directives](http://www.freedesktop.org/software/systemd/man/systemd.directives.html)
**Example**: raise the number of Open Files for MariaDB (previously configured in /etc/security/limits.conf):
```
/etc/systemd/system/mariadb.service.d/limits.conf
[Service]
LimitNOFILE=10000
LimitMEMLOCK=100000
```
**Example**: automatically restart Apache if it dies/killed/etc.:
```
/etc/systemd/system/httpd.service.d/restart.conf
[Service]
Restart=always
RestartSec=30
```
Alternatively, you can copy the distribution provided unit file from /lib/systemd/system to /etc/systemd/system since the latter has higher precedence. If a line starts with .include followed by a file name, the specified file will be parsed at this point.
```
/etc/systemd/system/httpd.service
.include /lib/systemd/system/httpd.service
[Service]
Nice=-5
```
> Don't forget to reload systemd daemon using `systemctl daemon-reload` and `systemctl restart (foobar)` after editing a unit file where _foobar_ is the name of the unit. Also note that you can `systemd-delta` to list the unit files which have been customized and also the precise differences
> Be careful when using `.include` together with directives that can be defined multiple times like `EnvironmentFile`, since we can only add new directives, but we can't remove already defined ones. You have to copy the whole file from /lib/systemd/system to /etc/systemd/system in this case
## Instanced Units (.service, .socket, etc...)
systemd allows you to set up _instanced_ services, or services that you can run multiple times in parallel with a small change to the unit file, such as getty, dhcpcd, sshd, and more. The way you define these services is slightly different from normal service files such as sshd.service or _foobar_.socket. Instead of _name_._type_, you would call the file _name_@._type_, and this would become an instanced unit file. The internals of the unit file are also somewhat different:
```
/usr/lib/systemd/system/dhcpcd@.service
[Unit]
Description=dhcpcd on %I
Wants=network.target
Before=network.target
BindsTo=sys-subsystem-net-devices-%i.device
After=sys-subsystem-net-devices-%i.device
[Service]
Type=forking
PIDFile=/run/dhcpcd-%I.pid
ExecStart=/usr/bin/dhcpcd -q -w %I
ExecStop=/usr/bin/dhcpcd -x %I
[Install]
WantedBy=multi-user.target
# # TO run the service you would use:
# systemctl start dhcpcd@eth1.service
# # enabling would also be the same, unit@instance.type
# systemctl enable dhcpcd@eth1.service
```
These commands would start or enable the dhcpcd service with the instance 'eth1'. All of the %i and %I variables would be replaced with the instance, in this case eth1. This allows you to have a generic service file for dhcpcd or other units, instead of writing a new service file for each instance of the daemon.
The %i and %I are two of the different ways to specify the instance name in a service file. You can see more here: [systemd.unit](http://www.freedesktop.org/software/systemd/man/systemd.unit.html#Specifiers)
## Path Units (.path)
systemd includes many types of unit files. The .path unit file allows systemd to watch a path with inotify and execute a service according to the .path file. You can find a full manual here: [systemd.path](http://www.freedesktop.org/software/systemd/man/systemd.path)
## Mount Units (.mount)
Most mounts are specified in `/etc/fstab` and the `systemd-generator` creates the mount units dynamically upon boot, however there are cases where it does not work as expected or needed. One of those types is a bind mount - a bind mount needs to have a unique mount file created for it outside of fstab. See the upstream [systemd.mount](https://www.freedesktop.org/software/systemd/man/systemd.mount.html) documentation for full details on all possible options and features that can be used.
### Example Bind Mount - /var/tmp to /tmp
One of the common bind mounts to create is for `/var/tmp` to `/tmp` as it is a line item in the [CIS Hardening Guidelines](https://benchmarks.cisecurity.org/downloads/benchmarks/) as a requirement. The name of the mount unit file **must be the name of the destination** with the leading slash removed, and all internal slashes converted to hyphens; it is placed in the expected runtime location if created by hand (`/etc/systemd/system`) and not delivered via a package (`/usr/lib/systemd/system`).
```
/etc/systemd/system/var-tmp.mount
# https://www.freedesktop.org/software/systemd/man/systemd.mount.html
# -> Automatic Dependencies
[Unit]
DefaultDependencies=yes
[Mount]
What=/tmp
Where=/var/tmp
Type=none
Options=bind
[Install]
WantedBy=local-fs.target
```
The unit must be enabled as per any other unit type:
```
# systemctl daemon-reload
# systemctl enable var-tmp.mount
```
With this particular bind mount, if anything is currently using `/var/tmp` those services must be stopped first to activate (`systemctl start ...`) the unit. It is possible a full reboot is needed depending on which services are using the directory.
## Service Resource Limits
The integration of cgroups into systemd allows us to limit resource usage for a service; the limiting of CPU, Memory and Block I/O are the primary concern to most system administrators. These limits are a per-cgroup limit, not per-process, so for Apache it would only allow the whole cgroup to use 1G of memory, no matter how many tiny processes it spawns.
By default all processes get an even value of CPU time (1024); by raising this number per-service more CPU shares are given over those at the default 1024 level:
```
/etc/systemd/system/httpd.service
.include /lib/systemd/system/httpd.service
[Service]
CPUShares=1500
```
Memory is limited in the exact same way; this setting understands K, M, G, T suffixes (base 1024)
```
/etc/systemd/system/httpd.service
.include /lib/systemd/system/httpd.service
[Service]
MemoryLimit=1G
```
Block I/O is _weighted_; default weight is 1000, valid range is from 10 to 1000 and can be used by service, by block device or by named directory:
```
/etc/systemd/system/httpd.service
.include /lib/systemd/system/httpd.service
[Service]
BlockIOWeight=500
BlockIOWeight=/dev/disk/by-id/dm-name-vg_local-lv_home 250
BlockIOWeight=/var/www/html 750
```
## Service Types
| **Type** | **Definition** |
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| simple | (default) _systemd_ considers the service to be started up immediately. The process must not fork. Do not use this type if other services need to be ordered on this service, unless it is socket activated. |
| forking | _systemd_ considers the service started up once the process forks and the parent has exited. Specify `PIDFile=` as well so systemd can keep track of the main process |
| oneshot | Useful for scripts that do a single job and then exit. Set `RemainAfterExit=yes` as well so that systemd still considers the service as active after the process has exited |
| notify | Identical to `Type=simple`, but with the stipulation that the daemon will send a signal to systemd when it is ready |
| dbus | The service is considered ready when the specified BusName appears on DBus's system bus |
## Operational Commands
| **Command** | **Purpose** |
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `systemctl --failed` | List failed units only |
| `systemctl list-jobs` | Look for jobs "running" (boot waits for completion) and "waiting" (executed only after those which are "running" are completed) |
| `systemctl list-units -t service --all` | List all available services and their current status |
| `systemctl list-units -t service` | Show all active services |
| `systemctl status` _foobar_.service | Examine the current runtime status of a service |
| `systemctl list-units -t target --all` | Show all available targets. |
| `systemctl list-units -t target` | Show all active targets |
| `systemctl show -p "Wants"` multi-user.target | See which services a target pulls in |
| `systemctl kill` httpd.service | Send the *kill* (SIGTERM) signal to all processes of a service |
| `systemctl kill -s SIGKILL` httpd.service | Send the specified signal to all processes of a service |
| `systemctl kill -s HUP --kill-who=main` httpd.service | Send the HUP signal to only the parent process of the service |
| `systemd-analyze blame` | List systemd unit initialization times at boot |
| `systemd --test --system --unit=`multi-user.target | Examine what gets started when when booted into a specific target |
## Power Management
| **Command** | **Purpose** |
| ------------------------ | ----------------------------------------------------------- |
| `systemctl reboot` | Shut down and reboot the system |
| `systemctl poweroff` | Shut down and power-off the system |
| `systemctl suspend` | Suspend the system |
| `systemctl hibernate` | Put the system into hibernation |
| `systemctl hybrid-sleep` | Put the system into hybrid-sleep state (or suspend-to-both) |
## Journal Commands
Compatibility with classic syslog implementations is provided via a socket `/run/systemd/journal/syslog`, to which all messages are forwarded. To make the syslog daemon work with the journal, it has to bind to this socket instead of `/dev/log`.
| **Command** | **Purpose** |
| ------------------------------------------------ | --------------------------------------------------------------------------------- |
| `journalctl -b -0` | Show all messages from the current boot (-1 = previous, -2 = two boots ago, etc.) |
| `journalctl -b -p err` | Show only priority ERROR level boot log messages |
| `journalctl --since=yesterday` | Show only logs since yesterday if rebooting is seldom (servers) |
| `journalctl -f` | Follow the journal like `tail -f /var/log/messages` |
| `journalctl /usr/sbin/httpd` | Show all messages by a specific binary |
| `journalctl /usr/sbin/vpnc /usr/sbin/dhclient` | Show messages interleaved between two binaries |
| `journalctl _PID=1234` | Show all messages by a specific process ID |
| `journalctl -u httpd` | Show all messages by a specific unit |
| `journalctl -u httpd --since=00:00 --until=9:30` | Show all messages for a unit and timeframe |
| `journalctl _TRANSPORT=kernel` | Show kernel ring buffer |
| `journalctl /dev/sdc` | Show messages related to a specific device |
## Notes
- RHEL7 will not have kdbus, which is going to make `system --user` require `$DISPLAY` variable because of dbus. kdbus is something that has been added very recently to the kernel and systemd.
## References
- <http://www.freedesktop.org/software/systemd/man/bootup.html>
- <http://en.wikipedia.org/wiki/Systemd>
- <https://en.wikipedia.org/wiki/Cgroups>
- <http://www.freedesktop.org/wiki/Software/systemd/FrequentlyAskedQuestions/>
- <http://www.freedesktop.org/wiki/Software/systemd/TipsAndTricks/>
- <http://www.freedesktop.org/wiki/Software/systemd/Debugging/>
- <http://www.freedesktop.org/software/systemd/man/>
- <http://0pointer.de/blog/projects/systemd-docs.html>
- <http://0pointer.de/blog/projects/journalctl.html>
- <https://www.kernel.org/doc/Documentation/cgroups/cgroups.txt>
- <https://fedoraproject.org/wiki/SysVinit_to_Systemd_Cheatsheet>
- <https://fedoraproject.org/wiki/How_to_debug_Systemd_problems>
- <https://fedoraproject.org/wiki/Systemd>
- <https://wiki.debian.org/systemd>
- <https://wiki.ubuntu.com/systemd>
- <https://wiki.archlinux.org/index.php/Systemd>

1634
md/tomcat_configuration.md Normal file
View file

File diff suppressed because it is too large Load diff

287
md/tomcat_logging.md Normal file
View file

@ -0,0 +1,287 @@
# Tomcat Logging
## Contents
- [Default Logging in Tomcat 5.5](#default-logging-in-tomcat-55)
- [Problem with growing log size](#problem-with-growing-log-size)
- [Solving with Apache's rotatelogs](#solving-with-apaches-rotatelogs)
- [Log rotation script example for rotatelogs](#log-rotation-script-example-for-rotatelogs)
- [Solving with logrotate.d](#solving-with-logrotated)
- [Using log4j with Tomcat 5.5](#using-log4j-with-tomcat-55)
- [Install needed libraries](#install-needed-libraries)
- [Configure logging.properties](#configure-loggingproperties)
- [References](#references)
Articles in this series
- [Tomcat Mechanics](tomcat_mechanics.md)
- [Tomcat Configuration](tomcat_configuration.md)
- [Tomcat Packaging](tomcat_packaging.md)
- **Tomcat Logging**
## Default Logging in Tomcat 5.5
### Problem with growing log size
The core logging file catalina.out with Tomcat grows and has no builtin log rotation without switching all of Tomcat to log4j. This file can grow to enormous sizes and use up disk space when it could be rotated automatically. The core startup script bin/catalina.sh does a very simple redirect of STDOUT and STDERR to the file catalina.out. This causes the file descriptor to be held open and makes it difficult to get this large file rotated without restarting Tomcat as a whole.
### Solving with Apache's rotatelogs
There are a few variations of a solution; at its core, the idea is to pipe the logging to a program which can rotate the file on it's own. In most cases the Apache software is also installed on the server which includes the 'rotatelogs' app: /usr/bin/rotatelogs by default on RHEL/CentOS RPM installs, otherwise in \<apache\>/bin/rotatelogs with a make install, etc.
Once you have rotatelogs on the server, a very minor change is made to the \<tomcat\>/bin/catalina.sh to pipe the logging instead; this is identical to Apache use. In Tomcat 5/6, the change looks like this:
```
org.apache.catalina.startup.Bootstrap "$@" start \
>> "$CATALINA_BASE"/logs/catalina.out 2>&1 &
org.apache.catalina.startup.Bootstrap "$@" start \
|/usr/bin/rotatelogs "$CATALINA_BASE"/logs/catalina.out.%Y%m%d-%H%M 200M >> /dev/null 2>&1 &
```
The implementation in each version of Tomcat will differ slightly as the script changes from release to release; it's important to note there are two places to change the code as the 'start' block has two ways that Tomcat can be started based on the security model. The '200M' is completely customizable, it can be time based or size based with most piped logging apps like rotatelogs.
### Log rotation script example for rotatelogs
You still need separate logic to remove old logs. The rotatelogs format keeps the catalina log files small but does nothing to purge or compress old ones so you'll need to set up a bash script in cron that does a 'find ... -exec rm ...' or similar, based on desire.
```
#!/bin/bash
# archivecatalina.sh
# v1.0
## SETTINGS MUST BE CHANGED TO USE THIS SCRIPT ##
# This script will operate on the already rotated catalina.out logfiles
# generated by the changing catalina.sh to use a piped logging method
# such as:
#
# org.apache.catalina.startup.Bootstrap "$@" start \
# |/usr/bin/rotatelogs "$CATALINA_BASE"/logs/catalina.out.%Y%m%d-%H%M 200M >> /dev/null 2>&1 &
# Run this from crontab nightly at an off hour,
# as gzip can suck up some serious CPU on huge logs:
# 0 2 * * * /opt/sbin/archivecatalina.sh
########## SETTINGS BEGIN ##########
# ARCHIVE_METHOD settings
# 0 = unset (default)
# 1 = compress in place
# 2 = move to another location
# 3 = compress and move
ARCHIVE_METHOD=0
# DURATION_KEEP setting
# <n> = number of days prior to keep around
DURATION_KEEP=28
# CATALINA_LOGS setting
# path to source catalina.out.* files
CATALINA_LOGS=/usr/local/tomcat/logs
# CATALINA_ARCH setting
# path to store older logs based on ARCHIVE_METHOD=2 or 3
# NOTE: this directory is NOT autocleaned on purpose!!
CATALINA_ARCH=/mnt/server/arclogs
# DURATION_WORK setting
# <n> = number of days prior to start working (default 1 day)
DURATION_WORK=1
########## SETTINGS END ##########
# What's our log name
LOG_BASE="catalina.out"
# Errorlevels
ERR_ERROR=33
ERR_PANIC=66
# Default RedHat program locations
MV=/bin/mv
DT=/bin/date
RM=/bin/rm
ID=/usr/bin/id
GZ=/usr/bin/gzip
# Are we root?
USER=`$ID -u`
if [ "X$USER" != "X0" ]; then
echo "PANIC: Only root can run this script!"
exit $ERR_PANIC
fi
# Sanity checks
if [ $ARCHIVE_METHOD -eq 0 ]; then
echo "PANIC: ARCHIVE_METHOD is set to 0, please configure the script."
exit $ERR_PANIC
fi
if [ $DURATION_WORK -lt 1 ]; then
echo "PANIC: DURATION_WORK is set to less than one day."
exit $ERR_PANIC
fi
if [ $DURATION_KEEP -le $DURATION_WORK ]; then
echo "PANIC: DURATION_KEEP is less than or equal to DURATION_WORK."
exit $ERR_PANIC
fi
# Does the source path exist?
if [ ! -x $CATALINA_LOGS ]; then
echo "ERROR: $CATALINA_LOGS doesn't exist!"
exit $ERR_ERROR
fi
# Backup dir exists/writable?
if [ $ARCHIVE_METHOD -eq 2 ] || [ $ARCHIVE_METHOD -eq 3 ]; then
if [ ! -x $CATALINA_ARCH ]; then
echo "ERROR: $CATALINA_ARCH doesn't exist or isn't writable!"
exit $ERR_ERROR
fi
fi
# generate fromat stamps
DATE_WORK=`$DT --date "now - $DURATION_WORK days" +"%Y%m%d"`
DATE_KEEP=`$DT --date "now - $DURATION_KEEP days" +"%Y%m%d"`
# log actions below depends on formatting looking like:
# /some/path/catalina.out.20100520-1345 (YYYYMMDD-HHMM)
# (.gz is accounted for)
# ARCHIVE_METHOD=1
compress_keep (){
for logfile in $CATALINA_LOGS/$LOG_BASE.*; do
# if it ends in gz it's already compressed
if [ ${logfile:(-3):3} == ".gz" ]; then
continue
else
# grab the %Y%m%d out of the name
DTS=${logfile:(-13):8}
if [ $DTS -ge $DATE_KEEP ] && [ $DTS -le $DATE_WORK ]; then
# compress in place
$GZ $logfile
fi
fi
done
}
# ARCHIVE_METHOD=2
move_only (){
for logfile in $CATALINA_LOGS/$LOG_BASE.*; do
# strip off .gz, we don't care if it's compressed
logtemp = ${logfile%.gz}
DTS=${logtmp:(-13):8}
if [ $DTS -ge $DATE_KEEP ] && [ $DTS -le $DATE_WORK ]; then
$MV -f $logfile $CATALINA_ARCH/
fi
done
}
# ARCHIVE_METHOD=3
compress_move (){
for logfile in $CATALINA_LOGS/$LOG_BASE.*; do
logtemp = ${logfile%.gz}
DTS=${logtmp:(-13):8}
if [ $DTS -ge $DATE_KEEP ] && [ $DTS -le $DATE_WORK ]; then
# already compressed
if [ ${logfile:(-3):3} == ".gz" ]; then
$MV -f $logfile $CATALINA_ARCH/
else
$GZ -c $logfile >> $CATALINA_ARCH/$logfile.gz
$RM -f $logfile
fi
fi
done
}
# cleanup older than DURATION_KEEP
cleanup_older (){
for logfile in $CATALINA_LOGS/$LOG_BASE.*; do
logtemp = ${logfile%.gz}
DTS=${logtmp:(-13):8}
if [ $DTS -lt $DATE_KEEP ]; then
$RM -f $logfile
fi
done
}
# the main case
case $ARCHIVE_METHOD in
1)
compress_keep
;;
2)
move_only
;;
3)
compress_move
;;
*)
echo "PANIC: Invalid ARCHIVE_METHOD settting: $ARCHIVE_METHOD"
exit $ERR_PANIC
esac
# do a final cleanup
cleanup_older
exit 0
```
### Solving with logrotate.d
Another way that works just as well:
- <http://java.dzone.com/articles/how-rotate-tomcat-catalinaout>
It's important to note that "copytruncate" verb used, it causes the original file to be truncated in place so it's FD; the only real downside to this method is that the copy process can chew up serious I/O if it's a really chatty Tomcat creating huge files. Using a rotatelogs method can save a bit of strain in this situation.
## Using log4j with Tomcat 5.5
Often when debugging a strange problem the default logging in Tomcat 5 does not go into enough detail to help pinpoint the problem. Adding log4j to the Tomcat instance can help track down problems, as well as provide proper log rotation to null out the growing catalina.out issue.
### Install needed libraries
- Download [Log4J](http://logging.apache.org/log4j/1.2/download.html) (v1.2 or later), unpack the tarball and place the log4j-\<version\>.jar in $CATALINA\_HOME/common/lib directory. In a RHEL based RPM install, this is usually /usr/share/tomcat/common/lib.
- Download [Commons Logging](http://commons.apache.org/logging/download_logging.cgi) and place the commons-logging-x.y.z.jar (not commons-logging-api-x.y.z.jar) in $CATALINA\_HOME/common/lib with the log4j jar added above.
### Configure logging.properties
Create a file called log4j.properties with the following content and save it into common/classes.
```
log4j.rootLogger=DEBUG, R
log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=${catalina.home}/logs/tomcat.log
log4j.appender.R.MaxFileSize=10MB
log4j.appender.R.MaxBackupIndex=10
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n
```
Start (or restart) Tomcat and log4j will now be creating a debug logfile for examination. This is just a basic example of a log4j setup, there are a lot of options that can be configured for use.
## References
**Log4J API Reference (FileAppender)**
- <http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/FileAppender.html>
- <http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/RollingFileAppender.html>
- <http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html>
- <http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/varia/ExternallyRolledFileAppender.html>
**Tomcat Logging**
- <http://tomcat.apache.org/tomcat-5.5-doc/logging.html>
- <http://tomcat.apache.org/tomcat-6.0-doc/logging.html>
- <http://tomcat.apache.org/tomcat-7.0-doc/logging.html>
- <http://tomcat.apache.org/tomcat-8.0-doc/logging.html>
**General Resources**
- <http://wiki.apache.org/tomcat/FAQ/Logging>
- <http://wiki.apache.org/logging-log4j/>
- <http://commons.apache.org/logging/>

132
md/tomcat_mechanics.md Normal file
View file

@ -0,0 +1,132 @@
# Tomcat Mechanics
## Contents
- [Common Variables](#common-variables)
- [Configuration Files](#configuration-files)
- [Filesystem Locations](#filesystem-locations)
- [Common](#common)
- [Tomcat 6 and above](#tomcat-6-and-above)
- [Tomcat 5 and below](#tomcat-5-and-below)
- [Operational Scripts](#operational-scripts)
- [Classpath Usage](#classpath-usage)
- [References](#references)
Articles in this series
- **Tomcat Mechanics**
- [Tomcat Configuration](tomcat_configuration.md)
- [Tomcat Packaging](tomcat_packaging.md)
- [Tomcat Logging](tomcat_logging.md)
## Common Variables
Installing Tomcat generically (non distribution packaging) is straight forward; download the appropriate archive and unpack it. Once unpacked there are several environment variables commonly referred to by most Tomcat admins:
| **Variable** | **Default** | **Description** |
| ----------------- | ---------------------------------- | ----------------------------------------------------------------------------------- |
| `CATALINA_HOME` | (none) | The main directory that represents the unpacked archive |
| `CATALINA_BASE` | `$CATALINA_HOME` | (Optional) Base directory for resolving dynamic portions of a Catalina installation |
| `CATALINA_OUT` | `$CATALINA_BASE/logs/catalina.out` | (Optional) Full path to a file where stdout and stderr will be redirected |
| `CATALINA_OPTS` | (none) | (Optional) Java runtime options used when the "start" command is executed |
| `CATALINA_TMPDIR` | `$CATALINA_BASE/temp` | (Optional) Temporary directory the JVM should use (java.io.tmpdir) |
| `JAVA_HOME` | (none) | Points at the Java Development Kit installation |
| `JRE_HOME` | `$JAVA_HOME` | Points at the Java Runtime Environment installation |
| `JAVA_OPTS` | (none) | (Optional) Java runtime options used when the "start" or "stop" command is executed |
## Configuration Files
Selected configuration files of the most interest:
| **File** | **Function** |
| -------------------------- | ----------------------------------------------------------------------------------- |
| `conf/server.xml` | The main configuration file where most items are configured |
| `conf/logging.properties` | Configuration for the logging infrastructure (`juli`, `log4j`, etc.) |
| `conf/web.xml` | Common to all web application deployments (.jsp servlet mappings, mime types, etc.) |
| `conf/catalina.policy` | Controls security policies when Java is started with `-security` |
| `conf/catalina.properties` | Controls class loading order and behaviour |
| `conf/context.xml` | Contexts that will be loaded for every web application (sets `WEB-INF/web.xml` up) |
| `conf/tomcat-users.xml` | User roles that can be used with various internal security classes |
## Filesystem Locations
### Common
Common subdirectories within the unpacked Tomcat directory for all versions:
| **Directory** | **Purpose** |
| ------------- | ---------------------------------------------------------------------------- |
| `bin/` | Executable scripts such as `catalina.sh`, `startup.sh` and `shutdown.sh` |
| `conf/` | Configuration files such as `server.xml`, `web.xml` and `logging.properties` |
| `logs/` | Default directory for logfiles |
| `temp/` | Various temporary files as needed (`java.io.tmpdir`) |
| `webapps/` | Location for .war (web archive) or unpacked web applications |
| `work/` | Location where .jsp files are pre-compiled |
### Tomcat 6 and above
| **Directory** | **Purpose** |
| ------------- | --------------------------------------------------------------------------- |
| `lib/` | Fundamental .jar (java archive) files for Tomcat operation common directory |
### Tomcat 5 and below
| **Directory** | **Purpose** |
| ------------------ | ------------------------------------------------------------------------- |
| `common/` | Classes visible to both internal Tomcat and all web applications |
| `common/lib/` | Classes and resources in .jar format |
| `common/il8n/` | Translation .jar files |
| `common/classes/` | Unpacked classes and resources |
| `common/endorsed/` | Deployment specific .jar files |
| `server/` | Classes visible to internal Tomcat only (Catalina) |
| `server/lib/` | Classes and resources in .jar format |
| `server/classes/` | Unpacked classes and resources |
| `server/webapps/` | Core web applications for management such as `host-manager` and `manager` |
| `shared/` | Classes visible to all web applications |
| `shared/classes/` | Unpacked classes and resources |
| `shared/libs/` | Classes and resources in .jar format |
## Operational Scripts
Some of the commonly used scripts in `$CATALINA_HOME/bin`:
| **Script** | **Function** |
| --------------- | ------------------------------------------------------------------------------- |
| `catalina.sh` | The main script for starting/stopping Tomcat |
| `startup.sh` | Resolves symlinks and launches `catalina.sh start $@` |
| `shutdown.sh` | Resolves symlinks and launches `catalina.sh stop $@` |
| `version.sh` | Resolves symlinks and launches `catalina.sh version $@` |
| `daemon.sh` | `stop`|`run`|`version` (_run_ stays attached to console) |
| `configtest.sh` | Resolves symlinks and launches `catalina.sh configtest $@` (Tomcat 7 and above) |
## Classpath Usage
The script `catalina.sh` overwrites the standard `$CLASSPATH` environment variable common to java use; the CLASSPATH is first nulled then constructed manually. It loads scripts and files in this order common to all versions:
1. `$CATALINA_BASE/bin/setenv.sh`
2. `$CATALINA_HOME/bin/setclasspath.sh`
3. `$CATALINA_HOME/bin/bootstrap.jar`
From this point onward the classes are loaded internally within Tomcat's java code and varies depending on version; see the **Classloaders** section of the **References** for links to each version's documentation.
## References
**Classloaders**
- <http://tomcat.apache.org/tomcat-5.5-doc/class-loader-howto.html>
- <http://tomcat.apache.org/tomcat-6.0-doc/class-loader-howto.html>
- <http://tomcat.apache.org/tomcat-7.0-doc/class-loader-howto.html>
- <http://tomcat.apache.org/tomcat-8.0-doc/class-loader-howto.html>
**Resources**
- <http://archive.apache.org/dist/tomcat/>
- <http://tomcat.apache.org/whichversion.html>
- <http://tomcat.apache.org/migration.html>
- <http://wiki.apache.org/tomcat/FAQ>

315
md/tomcat_packaging.md Normal file
View file

@ -0,0 +1,315 @@
# Tomcat Packaging
## Contents
- [RHEL/CentOS Tomcat 7](#rhelcentos-tomcat-7)
- [Installation Layout](#installation-layout)
- [Common Packages](#common-packages)
- [RHEL/CentOS Tomcat 6](#rhelcentos-tomcat-6)
- [Installation Layout](#installation-layout-1)
- [Common Packages](#common-packages-1)
- [RHEL/CentOS Tomcat 5](#rhelcentos-tomcat-5)
- [Installation Layout](#installation-layout-2)
- [Common Packages](#common-packages-2)
- [Ubuntu Tomcat 7](#ubuntu-tomcat-7)
- [Installation Layout](#installation-layout-3)
- [Common Packages](#common-packages-3)
- [Ubuntu Tomcat 6](#ubuntu-tomcat-6)
- [Installation Layout](#installation-layout-4)
- [Common Packages](#common-packages-4)
Articles in this series
- [Tomcat Mechanics](tomcat_mechanics.md)
- [Tomcat Configuration](tomcat_configuration.md)
- **Tomcat Packaging**
- [Tomcat Logging](tomcat_logging.md)
## RHEL/CentOS Tomcat 7
### Installation Layout
RHEL/CentOS 7 are based on systemd; a 'yum install tomcat' (note no _7_ in the names) will install the systemd service unit. This unit, `/usr/lib/systemd/system/tomcat.service`, calls the shell script `/usr/sbin/tomcat-sysd` to mimic previous behaviour and read /etc/sysconfig/tomcat. An additional package, `tomcat-systemv`, can be installed to provide `/etc/init.d/tomcat` which works the same as RHEL/CentOS 5/6.
Unlike a traditional unpacked archive, the systemd unit and SysVinit script will read two files for variables; the first is a static-named file `/etc/tomcat/tomcat.conf` for defaults, then a dynamic file `/etc/sysconfig/tomcat`. The file in sysconfig that will be read is the same name as the script; this allows copying of the `/etc/init.d/` (or cloning the systemd unit) and `/etc/sysconfig/` files to a new name to run a 2nd, 3rd, etc. instance of Tomcat.
| **Item** | **Purpose** |
| -------------------------------- | ---------------------------------------------------------------------------------------------- |
| `/usr/lib/systemd/system/tomcat` | systemd unit used to start/stop Tomcat |
| `/etc/init.d/tomcat` | SysVinit script used to start/stop Tomcat |
| `/etc/tomcat/tomcat.conf` | System-wide configuration file for tomcat services |
| `/etc/sysconfig/tomcat` | Service-specific configuration file for tomcat; exact name loaded matches the unit/init script |
| `/usr/sbin/tomcat` | Script that somewhat emulates the traditional `catalina.sh` for starting/stopping Tomcat |
| `/var/run/tomcat.pid` | PID of the running tomcat java service |
| `/etc/logrotate.d/tomcat` | Logrotate configuration for `catalina.out` |
| `/var/log/tomcat/` | Logging directory |
| `/usr/share/tomcat/` | Main directory that mimics a traditional unpacked archive |
| `/usr/share/tomcat/bin` | Contains only the `bootstrap.jar` and `tomcat-juli.jar` files |
| `/usr/share/tomcat/conf` | Symlink to `/etc/tomcat` |
| `/usr/share/tomcat/lib` | Symlink to `/usr/share/java/tomcat` |
| `/usr/share/tomcat/logs` | Symlink to `/var/log/tomcat` |
| `/usr/share/tomcat/temp` | Symlink to `/var/cache/tomcat/temp` |
| `/usr/share/tomcat/webapps` | Symlink to `/var/lib/tomcat/webapps` |
| `/usr/share/tomcat/work` | Symlink to `/var/cache/tomcat/work` |
| `/usr/share/java/` | Main directory containing various .jar (java archive) files |
| `/usr/share/java/tomcat/` | Contains classes and resources for Tomcat only |
### Common Packages
The core packages installed when `yum install tomcat` takes place; others may be installed as dependencies depending on what's already installed on the system.
| **Package** | **Summary** |
| ---------------------------- | --------------------------------------------------------------------------- |
| `apache-commons-collections` | Provides new interfaces, implementations and utilities for Java Collections |
| `apache-commons-daemon` | Defines API to support an alternative invocation mechanism |
| `apache-commons-dbcp` | Apache Commons DataBase Pooling Package |
| `apache-commons-logging` | Apache Commons Logging |
| `apache-commons-pool` | Apache Commons Pool Package |
| `avalon-framework` | Java components interfaces |
| `avalon-logkit` | Java logging toolkit |
| `ecj` | Eclipse Compiler for Java |
| `geronimo-jms` | J2EE JMS v1.1 API |
| `geronimo-jta` | J2EE JTA v1.1 API |
| `javamail` | Java Mail API |
| `javapackages-tools` | Macros and scripts for Java packaging support |
| `jline` | Java library for reading and editing user input in console applications |
| `log4j` | Java logging package |
| `python-javapackages` | Module for handling various files for Java packaging |
| `tomcat` | Apache Servlet/JSP Engine, RI for Servlet 3.0/JSP 2.2 API |
| `tomcat-el-2.2-api` | Expression Language v1.0 API |
| `tomcat-jsp-2.2-api` | Apache Tomcat JSP API implementation classes |
| `tomcat-lib` | Libraries needed to run the Tomcat Web container |
| `tomcat-servlet-3.0-api` | Apache Tomcat Servlet API implementation classes |
| `xalan-j2` | Java XSLT processor |
| `xerces-j2` | Java XML parser |
| `xml-commons-apis` | APIs for DOM, SAX, and JAXP |
| `xml-commons-resolver` | Resolver subproject of xml-commons |
## RHEL/CentOS Tomcat 6
### Installation Layout
Unlike a traditional unpacked archive, the SysVinit script will read two files for variables; the first is a static-named file `/etc/tomcat6/tomcat6.conf` for defaults, then a dynamic file `/etc/sysconfig/tomcat6`. The file in sysconfig that will be read is the same name as the script; this allows copying of the `/etc/init.d/` and `/etc/sysconfig/` files to a new name to run a 2nd, 3rd, etc. instance of Tomcat.
| **Item** | **Purpose** |
| ---------------------------- | ------------------------------------------------------------------------------------------ |
| `/etc/init.d/tomcat6` | SysVinit script used to start/stop Tomcat |
| `/etc/tomcat6/tomcat6.conf` | System-wide configuration file for tomcat6 services |
| `/etc/sysconfig/tomcat6` | Service-specific configuration file for tomcat6; exact name loaded matches the init script |
| `/usr/sbin/tomcat6` | Script that somewhat emulates the traditional `catalina.sh` for starting/stopping Tomcat |
| `/var/run/tomcat6.pid` | PID of the running tomcat6 java service |
| `/etc/logrotate.d/tomcat6` | Logrotate configuration for `catalina.out` |
| `/var/log/tomcat6/` | Logging directory |
| `/usr/share/tomcat6/` | Main directory that mimics a traditional unpacked archive |
| `/usr/share/tomcat6/bin` | Contains only the `bootstrap.jar` and `tomcat-juli.jar` files |
| `/usr/share/tomcat6/conf` | Symlink to `/etc/tomcat6` |
| `/usr/share/tomcat6/lib` | Symlink to `/usr/share/java/tomcat6` |
| `/usr/share/tomcat6/logs` | Symlink to `/var/log/tomcat6` |
| `/usr/share/tomcat6/temp` | Symlink to `/var/cache/tomcat6/temp` |
| `/usr/share/tomcat6/webapps` | Symlink to `/var/lib/tomcat6/webapps` |
| `/usr/share/tomcat6/work` | Symlink to `/var/cache/tomcat6/work` |
| `/usr/share/java/` | Main directory containing various .jar (java archive) files |
| `/usr/share/java/tomcat6/` | Contains classes and resources for Tomcat only |
### Common Packages
The core packages installed when `yum install tomcat6` takes place; others may be installed as dependencies depending on what's already installed on the system.
| **Package** | **Summary** |
| ----------------------------- | --------------------------------------------------------------------------- |
| `axis` | A SOAP implementation in Java |
| `classpathx-jaf` | GNU JavaBeans(tm) Activation Framework |
| `classpathx-mail` | GNU JavaMail(tm) |
| `ecj` | Eclipse Compiler for Java |
| `jakarta-commons-collections` | Provides new interfaces, implementations and utilities for Java Collections |
| `jakarta-commons-daemon` | Defines API to support an alternative invocation mechanism |
| `jakarta-commons-dbcp` | Jakarta Commons DataBase Pooling Package |
| `jakarta-commons-discovery` | Jakarta Commons Discovery |
| `jakarta-commons-httpclient` | Jakarta Commons HTTPClient implements the client side of HTTP standards |
| `jakarta-commons-logging` | Jakarta Commons Logging Package |
| `jakarta-commons-pool` | Jakarta Commons Pool Package |
| `jpackage-utils` | JPackage utilities |
| `libgcj` | Java runtime library for gcc |
| `log4j` | Java logging package |
| `mx4j` | Open source implementation of JMX Java API |
| `regexp` | Simple regular expressions API |
| `sinjdoc` | Documentation generator for Java source code |
| `tomcat6` | Apache Servlet/JSP Engine, RI for Servlet 2.5/JSP 2.1 API |
| `tomcat6-el-2.1-api` | Expression Language v1.0 API |
| `tomcat6-jsp-2.1-api` | Apache Tomcat JSP API implementation classes |
| `tomcat6-lib` | Libraries needed to run the Tomcat Web container |
| `tomcat6-servlet-2.5-api` | Apache Tomcat Servlet API implementation classes |
| `wsdl4j` | Web Services Description Language Toolkit for Java |
| `xml-commons-apis` | APIs for DOM, SAX, and JAXP |
| `xml-commons-resolver` | Resolver subproject of xml-commons |
## RHEL/CentOS Tomcat 5
### Installation Layout
Unlike a traditional unpacked archive, the SysVinit script will read two files for variables; the first is a static-named file `/etc/tomcat5/tomcat5.conf` for defaults, then a dynamic file `/etc/sysconfig/tomcat5`. The file in sysconfig that will be read is the same name as the script; this allows copying of the `/etc/init.d/` and `/etc/sysconfig/` files to a new name to run a 2nd, 3rd, etc. instance of Tomcat.
| **Item** | **Purpose** |
| ---------------------------- | ------------------------------------------------------------------------------------------ |
| `/etc/init.d/tomcat5` | SysVinit script used to start/stop Tomcat |
| `/etc/tomcat5/tomcat5.conf` | System-wide configuration file for tomcat5 services |
| `/etc/sysconfig/tomcat5` | Service-specific configuration file for tomcat5; exact name loaded matches the init script |
| `/usr/sbin/dtomcat5` | The traditional `catalina.sh` for starting/stopping Tomcat renamed |
| `/var/run/tomcat5.pid` | PID of the running tomcat5 java service |
| `/etc/logrotate.d/tomcat5` | Logrotate configuration for `catalina.out` |
| `/var/log/tomcat5/` | Logging directory |
| `/usr/share/tomcat5/` | Main directory that mimics a traditional unpacked archive |
| `/usr/share/tomcat5/bin` | Contains the `bootstrap.jar`, `tomcat-juli.jar` and some other basic files |
| `/usr/share/tomcat5/common` | Symlink to `/var/lib/tomcat5/common` |
| `/usr/share/tomcat5/conf` | Symlink to `/etc/tomcat5` |
| `/usr/share/tomcat5/logs` | Symlink to `/var/log/tomcat5` |
| `/usr/share/tomcat5/server` | Symlink to `/var/lib/tomcat5/server` |
| `/usr/share/tomcat5/shared` | Symlink to `/var/lib/tomcat5/shared` |
| `/usr/share/tomcat5/temp` | Symlink to `/var/cache/tomcat5/temp` |
| `/usr/share/tomcat5/webapps` | Symlink to `/var/lib/tomcat5/webapps` |
| `/usr/share/tomcat5/work` | Symlink to `/var/cache/tomcat5/work` |
| `/usr/share/java/` | Main directory containing various .jar (java archive) files |
| `/usr/share/java/tomcat5/` | Contains classes and resources for Tomcat only |
### Common Packages
The core packages installed when `yum install tomcat5` takes place; others may be installed as dependencies depending on what's already installed on the system.
| **Package** | **Summary** |
| ----------------------------- | ------------------------------------------------------------------------------------ |
| `ant` | Ant build tool for java |
| `axis` | A SOAP implementation in Java |
| `bcel` | Byte Code Engineering Library |
| `classpathx-jaf` | GNU JavaBeans(tm) Activation Framework |
| `classpathx-mail` | GNU JavaMail(tm) |
| `eclipse-ecj` | Eclipse Compiler for Java |
| `gcc-java` | Java support for GCC |
| `geronimo-specs` | Geronimo J2EE server J2EE specifications |
| `gjdoc` | GNU Javadoc |
| `jakarta-commons-beanutils` | Jakarta Commons BeanUtils Package |
| `jakarta-commons-collections` | Jakarta Commons Collections Package |
| `jakarta-commons-daemon` | Jakarta Commons Daemon Package |
| `jakarta-commons-dbcp` | Jakarta Commons DataBase Pooling Package |
| `jakarta-commons-digester` | Jakarta Commons Digester Package |
| `jakarta-commons-discovery` | Jakarta Commons Discovery |
| `jakarta-commons-el` | The Jakarta Commons Extension Language |
| `jakarta-commons-fileupload` | Jakarta Commons Fileupload Package |
| `jakarta-commons-httpclient` | Jakarta Commons HTTPClient Package |
| `jakarta-commons-launcher` | The Launcher Component is designed to be a cross platform Java application launcher. |
| `jakarta-commons-logging` | Jakarta Commons Logging Package |
| `jakarta-commons-modeler` | Jakarta Commons Modeler Package |
| `jakarta-commons-pool` | Jakarta Commons Pool Package |
| `jpackage-utils` | JPackage utilities |
| `ldapjdk` | The Mozilla LDAP Java SDK |
| `libgcj` | Java runtime library for gcc |
| `log4j` | Java logging package |
| `mx4j` | Open source implementation of JMX Java API |
| `regexp` | Simple regular expressions API |
| `tomcat5` | Apache Servlet/JSP Engine, RI for Servlet 2.4/JSP 2.0 API |
| `tomcat5-common-lib` | Libraries needed to run the Tomcat Web container (part) |
| `tomcat5-jasper` | Compiler JARs and associated scripts for tomcat5 |
| `tomcat5-jsp-2.0-api` | Jakarta Tomcat Servlet and JSP implementation classes |
| `tomcat5-server-lib` | Libraries needed to run the Tomcat Web container (part) |
| `tomcat5-servlet-2.4-api` | Jakarta Tomcat Servlet implementation classes |
| `wsdl4j` | Web Services Description Language Toolkit for Java |
| `xalan-j2` | Java XSLT processor |
| `xerces-j2` | Java XML parser |
| `xml-commons` | Common code for XML projects |
| `xml-commons-apis` | APIs subproject of xml-commons |
| `xml-commons-resolver` | Resolver subproject of xml-commons. |
## Ubuntu Tomcat 7
### Installation Layout
Ubuntu does not use a dynamic config based on the init script name, it's single instance only with a basic install. An additional package `tomcat7-user` contains the tools needed to generate more instances.
| **Item** | **Purpose** |
| -------------------------- | ----------------------------------------------------------------- |
| `/etc/init.d/tomcat7` | SysVinit script used to start/stop Tomcat |
| `/etc/default/tomcat7` | System-wide configuration file for init script overrides |
| `/var/run/tomcat7.pid` | PID of the running tomcat7 java service |
| `/etc/logrotate.d/tomcat7` | Logrotate configuration for `catalina.out` |
| `/usr/share/tomcat7/bin` | Contains the `catalina.sh`, `bootstrap.jar` and other basic files |
| `/usr/share/tomcat7/lib` | Contains symlinks for all the .jar files to load |
| `/var/lib/tomcat7/common` | Empty directory structure for classes and resources |
| `/var/lib/tomcat7/conf` | Symlink to `/etc/tomcat7` |
| `/var/lib/tomcat7/logs` | Symlink to `../../log/tomcat7 |
| `/var/lib/tomcat7/server` | Empty directory structure for classes and resources |
| `/var/lib/tomcat7/shared` | Empty directory structure for classes and resources |
| `/var/lib/tomcat7/webapps` | Empty directory structure for classes and resources |
| `/var/lib/tomcat7/work` | Symlink to `../../cache/tomcat7` |
| `/var/log/tomcat7/` | Logging directory |
| `/usr/share/java/` | Main directory containing various .jar (java archive) files |
### Common Packages
The core packages installed when `apt-get install tomcat7` takes place; others may be installed as dependencies depending on what's already installed on the system. This list is **without** the _recommends_ or _suggests_ packages.
| **Package** | **Summary** |
| ------------------------------ | ---------------------------------------------------------------- |
| `ca-certificates-java` | Common CA certificates (JKS keystore) |
| `default-jre-headless` | Standard Java or Java compatible Runtime (headless) |
| `java-common` | Base of all Java packages |
| `libcommons-collections3-java` | A set of abstract data type interfaces and implementations |
| `libcommons-dbcp-java` | Database Connection Pooling Services |
| `libcommons-pool-java` | pooling implementation for Java objects |
| `libecj-java` | Eclipse Java compiler (library) |
| `libservlet3.0-java` | Servlet 3.0 and JSP 2.2 Java API classes |
| `libtomcat7-java` | Servlet and JSP engine -- core libraries |
| `tomcat7` | Servlet and JSP engine |
| `tomcat7-common` | Servlet and JSP engine -- common files |
| `tzdata-java` | time zone and daylight-saving time data for use by java runtimes |
## Ubuntu Tomcat 6
### Installation Layout
Ubuntu does not use a dynamic config based on the init script name, it's single instance only with a basic install. An additional package `tomcat6-user` contains the tools needed to generate more instances.
| **Item** | **Purpose** |
| -------------------------- | ----------------------------------------------------------------- |
| `/etc/init.d/tomcat6` | SysVinit script used to start/stop Tomcat |
| `/etc/default/tomcat6` | System-wide configuration file for init script overrides |
| `/var/run/tomcat6.pid` | PID of the running tomcat6 java service |
| `/etc/logrotate.d/tomcat6` | Logrotate configuration for `catalina.out` |
| `/usr/share/tomcat6/bin` | Contains the `catalina.sh`, `bootstrap.jar` and other basic files |
| `/usr/share/tomcat6/lib` | Contains symlinks for all the .jar files to load |
| `/var/lib/tomcat6/common` | Empty directory structure for classes and resources |
| `/var/lib/tomcat6/conf` | Symlink to `/etc/tomcat6` |
| `/var/lib/tomcat6/logs` | Symlink to `../../log/tomcat6` |
| `/var/lib/tomcat6/server` | Empty directory structure for classes and resources |
| `/var/lib/tomcat6/shared` | Empty directory structure for classes and resources |
| `/var/lib/tomcat6/webapps` | Empty directory structure for classes and resources |
| `/var/lib/tomcat6/work` | Symlink to `../../cache/tomcat6` |
| `/var/log/tomcat6/` | Logging directory |
| `/usr/share/java/` | Main directory containing various .jar (java archive) files |
### Common Packages
The core packages installed when `apt-get install tomcat6` takes place; others may be installed as dependencies depending on what's already installed on the system. This list is **without** the _recommends_ or _suggests_ packages.
| **Package** | **Summary** |
| ------------------------------ | ---------------------------------------------------------------- |
| `ca-certificates-java` | Common CA certificates (JKS keystore) |
| `default-jre-headless` | Standard Java or Java compatible Runtime (headless) |
| `java-common` | Base of all Java packages |
| `libcommons-collections3-java` | A set of abstract data type interfaces and implementations |
| `libcommons-dbcp-java` | Database Connection Pooling Services |
| `libcommons-pool-java` | pooling implementation for Java objects |
| `libecj-java` | Eclipse Java compiler (library) |
| `libservlet2.5-java` | Servlet 2.5 and JSP 2.1 Java API classes |
| `libtomcat6-java` | Servlet and JSP engine -- core libraries |
| `tomcat6` | Servlet and JSP engine |
| `tomcat6-common` | Servlet and JSP engine -- common files |
| `tzdata-java` | time zone and daylight-saving time data for use by java runtimes |

36
md/tuning_nf_conntrack.md Normal file
View file

@ -0,0 +1,36 @@
# Tuning nf_conntrack
## Contents
- [Overview](#overview)
- [Updating the module loading](#updating-the-module-loading)
- [References](#references)
## Overview
The kernel parameter for the maximum number of network connections tracked when the _nf\_conntrack_ has been loaded is dynamic based on the amount of RAM in the system; in practice this has shown to be a value too low for a high traffic server and result in dropped packets.
## Updating the module loading
The iptables oriented `nf_conntrack` module may or may not be loaded at boot; if there are no rules requiring it to be loaded, it will be skipped. However as soon as a rule is added which requires it, such as this simple one:
```
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
```
...the module will be loaded by the kernel. There is no mechanism or design via sysctl to apply a tuning value in `/etc/sysctl.conf` or `/etc/sysctl.d/*.conf` in this scenario in most major Linux distributions; as such, the process must happen when the module loading itself occurs. This can be accomplished via the modprobe subsystem.
Create the file `/etc/modprobe.d/nf_conntrack.conf` like so:
```
cat << EOF >> /etc/modprobe.d/nf_conntrack.conf
install nf_conntrack /sbin/modprobe --ignore-install nf_conntrack ; echo 262144 > /proc/sys/net/nf_conntrack_max
EOF
```
The value **262144** is roughly 4 times the default normally seen on a 1-2GB RAM server and is setting an upper limit on the number of hash table entries; it is not a preallocation and only used as needed. If the kernel module is already loaded, after creating the above file run the `echo 262144 > /proc/sys/net/nf_conntrack_max` manually to adjust immediately.
## References
- <https://major.io/2014/01/07/nf-conntrack-table-full-dropping-packet/>

35
md/understanding_swap_use.md Normal file
View file

@ -0,0 +1,35 @@
# Understanding Swap Use
## Contents
- [Soliloquy](#soliloquy)
- [Citations](#citations)
## Soliloquy
A frequent set of questions asked - "Why doesn't my swapfile reduce in size? I've looked in `/proc/(PID)/smaps` with various scripts/tools and they all report 0 use? Why is there still swap allocated?"
Lots of arms and legs to these questions; first, understand that it is more appropriate to use `sar -B` (paging statistics) and `sar -r` (memory utilization) to look at the Major faults/sec (loading a memory page from disk/swap) to find out if it's actively being used or not (as well as `/proc/meminfo`, see below). Allocation is a red-herring, active use is what counts.
When we're entering a swap situation, the kernel first uses an algorithm [(1)][c1] to determine what to page out. Remember that you can't actually work with the memory page out of the swapfile, it has to be paged back into RAM for operations - but it is _not_ deleted from the swapfile at that page-in time as this is expensive for no gain, especially if you have to page back out the exact same data later. Thrashing occurs when you're paging in and out rapidly because the active processes need to operate on ALL those pages and can't throw any of them away. Thrash too much and our friend the OOM Killer kicks in and starts killing things.
When the swap page is paged back in, it's possible that there exists a copy in RAM and a copy in swap at this point; if the PTE structure changes in RAM (the process does something and alters it's data), _then_ the swap space blocks are deleted (called a delete on write operation) as the RAM copy is now unique and new, the swap data is truly old and invalid. But, in the case that the PTE hasn't changed and it needs to swap out again then it's already in swap thereby reducing I/O - score\! This is what the kernel is counting on - it will not have to physically page out unchanged data.
If you examine the `/proc/meminfo` for `SwapCached` that will show you how much of swap is actually in this state [(2)][c2], especially when you can't find it using common tools. So why can't you find it? Because this page has to be swapped back into RAM in order for the kernel to determine who is truly left using it, and that's too expensive (and not useful). The best writeup on this complicated procedure is via the kernel's `swapoff` system call doc [(3)][c3]. By running `swapoff` on a swap partition/file you're performing this "page in the data. used? nope, delete." procedure over and over on every page in the swap file. This is why it's safer to add another swapfile (of same or larger size) temporarily when running swapoff, just in case you truly don't have enough RAM to hold what's in swap and accidentally trigger the OOM Killer.
This should help explain why until the process that allocated that PTE to swap quits/exits (thereby freeing all memory pointers/reference counts) it's still used per se; the kernel at one point swapped it to disk, it has since been swapped back into RAM (which is now what the process sees) but an identical copy still exists in swap. Upon quit, the in-RAM page is freed which is a change to the PTE, which triggers an invalidation of the page in swap so the delete on write is triggered (here I guess you could say technically it's not a write) and bye-bye goes that allocated swap space. UNLESS it was a shared memory structure and someone else still has a reference/map to it (another can of worms).
This is what I call "opportunistic swapping", the kernel has paged out once and is banking on the possibility it will happen again -- if the page is already in swap, no need to incur I/O to write it again. Only the kernel sees this mapping (RAM PTEs to swap PTEs), not the actual processes themselves - reduce unnecessary disk I/O as much as possible while remaining abstract to the user level applications.
## Citations
1. <http://en.wikipedia.org/wiki/Page_replacement_algorithm>
2. <http://www.redhat.com/advice/tips/meminfo.html>
3. <https://www.kernel.org/doc/gorman/html/understand/understand014.html#toc81>
[c1]: http://en.wikipedia.org/wiki/Page_replacement_algorithm
[c2]: http://www.redhat.com/advice/tips/meminfo.html
[c3]: https://www.kernel.org/doc/gorman/html/understand/understand014.html#toc81

446
md/vsftpd_setup.md Normal file
View file

@ -0,0 +1,446 @@
# Vsftpd Setup
## Contents
- [Overview](#overview)
- [Installation](#installation)
- [Configuration Files](#configuration-files)
- [Configuration Basics](#configuration-basics)
- [TCP Wrappers](#tcp-wrappers)
- [Login Banner](#login-banner)
- [Directory Message](#directory-message)
- [Logging](#logging)
- [Local Users](#local-users)
- [Chroot Security](#chroot-security)
- [Access Control](#access-control)
- [Anonymous FTP](#anonymous-ftp)
- [Connection Restrictions](#connection-restrictions)
- [SSL Connections](#ssl-connections)
- [Active vs. Passive](#active-vs-passive)
- [Virtual chroot Users](#virtual-chroot-users)
- [vsftpd.conf](#vsftpdconf)
- [PAM ftp](#pam-ftp)
- [SELinux](#selinux)
- [Adding a User](#adding-a-user)
- [Reloading Users](#reloading-users)
- [Updating a User](#updating-a-user)
- [Removing a User](#removing-a-user)
- [References](#references)
## Overview
[vsftpd](https://security.appspot.com/vsftpd.html) is a GPL licensed FTP server for UNIX systems, including Linux. It is the default FTP server daemon as installed and taught by Red Hat, and as such enjoys the status of being the most popular FTP daemon in modern use with most Linux distributions. As a precursor, it's recommended to fully understand Active vs. Passive FTP mode:
- <http://slacksite.com/other/ftp.html>
This document will cover the basic installation and configuration of vsftpd; more options are available in the *vsftpd.conf* man page.
## Installation
The vsftpd daemon is included in the core distribution repositories:
```
## RHEL / CentOS
yum -y install vsftpd
service vsftpd start && chkconfig vsftpd on
## Debian / Ubuntu
apt-get update && apt-get -y install vsftpd
mkdir /etc/vsftpd /var/ftp
```
**NOTE**: on Ubuntu the _vsftpd_ service is controlled by Upstart; it's automatically started and enabled by default. On Debian the `insserv` command must be used to enable it.
## Configuration Files
There's one key difference between the distributions:
| **RHEL / CentOS** | **Debian / Ubuntu** |
| ------------------------- | ------------------- |
| `/etc/vsftpd/vsftpd.conf` | `/etc/vsftpd.conf` |
| `/etc/vsftpd/ftpusers` | `/etc/ftpusers` |
This also extends into the entire `/etc/vsftpd/` subdirectory and a default directory for anonymous users in `/var/ftp/` which are only present on a RHEL-style installations from the package install. The Ubuntu package comes with both a systemd and upstart service that are not present on Debian; creating an `/etc/vsftpd/` to work in on these distributions is preferred to allow for cross-distro compatibility in setup below.
## Configuration Basics
All options refer to the `vsftpd.conf` configuration file and are detailed in the man page; other options exist than the ones listed herein.
### TCP Wrappers
```
# Disable TCP Wrappers
tcp_wrappers=NO
```
### Login Banner
```
# Set a direct banner (hide name, version, etc.)
ftpd_banner=FTP server ready.
# Overrides 'ftpd_banner' by specifying a filename
banner_file=/etc/vsftpd/banner.txt
```
### Directory Message
```
# Shown when changing into the directory
message_file=.message
dirmessage_enable=YES
```
### Logging
```
# Log all uploads and downloads
xferlog_enable=YES
# Log all FTP commands
log_ftp_protocol=YES
xferlog_std_format=NO
# Log transfers to both /var/log/xferlog and /var/log/vsftpd.log
dual_log_enable=YES
```
### Local Users
```
# Disable 'write' (upload)
write_enable=NO
# Disable local users
local_enable=NO
```
### Chroot Security
By default users are **not** in a chroot upon login.
```
# Basic Local user login chroot to home directory
chroot_local_user=YES
chroot_list_enable=NO
# Enable list of users ONLY to chroot
chroot_local_user=NO
chroot_list_enable=YES
chroot_list_file=/etc/vsftpd/chroot_list
# Enable list of users NOT to chroot
chroot_local_user=YES
chroot_list_enable=YES
chroot_list_file=/etc/vsftpd/chroot_list
```
### Access Control
By default, User ACLs are enabled in `/etc/pam.d/vsftpd` using the `/etc/ftpusers` / `/etc/vsftpd/ftpusers` file:
```
auth required pam_listfile.so item=user sense=deny file=/etc/vsftpd/ftpusers onerr=succeed
```
To also allow manipulation by Group, a line can be added right below it using a (new) group file:
```
auth required pam_listfile.so item=group sense=deny file=/etc/vsftpd/ftpgroups onerr=succeed
```
Within `vsftpd.conf` are parameters determining how the ACLs are used:
```
# Should be configured already by upstream packagers
pam_service_name=vsftpd
# Enable list of users ONLY allowed to log in
userlist_enable=YES
userlist_deny=NO
userlist_file=/etc/vsftpd/ftpusers
# Enable list of users NEVER allowed to log in
userlist_enable=YES
userlist_deny=YES
userlist_file=/etc/vsftpd/ftpusers
```
### Anonymous FTP
```
# Enable anonymous FTP w/password (email address, usually)
ftp_username=ftp
anonymous_enable=YES
anon_root=/var/ftp
anon_max_rate=0 # bytes/second, 0 = unlimited
anon_umask=077
# Enable only anon users who specify a certain email as password
no_anon_password=NO
secure_email_list_enable=YES
email_password_file=/etc/vsftpd/email_passwords
# Reverse the above, deny these emails when used as anon password
deny_email_enable=YES
banned_email_file=/etc/vsftpd/banned_emails
# Allow anon users to upload/delete/etc. (requires write_enable=YES)
anon_mkdir_write_enable=YES
anon_other_write_enable=YES
anon_upload_enable=YES
anon_world_readable_only=YES
chown_uploads=YES
chown_username=ftp
```
### Connection Restrictions
```
# Change limits and umask
connect_timeout=60 # in seconds
data_connection_timeout=300
idle_session_timeout=300
max_clients=2000
max_per_ip=10
max_login_fails=3
delay_failed_login=1 # in seconds
local_max_rate=0 # bytes/second, 0 = unlimited
local_umask=077
```
It's preferred to use `iptables` to perform restrictions at the IP level; however, if required TCP Wrappers can be used:
```
# Enable TCP Wrappers
tcp_wrappers=YES
# Deny a subnet
echo 'vsftpd:192.168.1.0/255.255.255.0:twist /bin/cat /etc/vsftpd/denied_message' >> /etc/hosts.deny
```
Using `iptables` is preferred.
### SSL Connections
```
# General SSL Options
ssl_enable=YES
ssl_request_cert=YES
validate_cert=NO # breaks self-signed certs
ca_certs_file=/path/to/ca-bundle.crt
rsa_cert_file=/path/to/server.crt # can be a PEM
rsa_private_key_file=/path/to/server.key # can be a PEM
ssl_ciphers=HIGH:!ADH
ssl_sslv2=NO
ssl_sslv3=NO
ssl_tlsv1=YES
debug_ssl=NO
# Always expect a SSL handshake (not normal - use TLS instead)
implicit_ssl=NO
# Force various SSL options
require_cert=NO # client SSL cert, not server
force_local_data_ssl=NO
force_local_logins_ssl=NO
require_ssl_reuse=YES
strict_ssl_read_eof=NO
strict_ssl_write_shutdown=NO
# SSL options for Anonymous FTP
allow_anon_ssl=YES
force_anon_ssl=NO
force_anon_data_ssl=NO
force_anon_logins_ssl=NO
force_local_data_ssl=NO
```
## Active vs. Passive
Configuration options exist to facilitate setting of Active ("PORT") and Passive ("PASV") FTP to work around various firewall restrictions. One of the most common techniques is to use the kernel netfilter modules `nf_nat_ftp` and `nf_conntrack_ftp` (aliased internally to `ip_nat_ftp` and `ip_conntrack_ftp` for compatibility) which will track the FTP connections and attempt to maintain a data connection to the client dynamically.
```
# RHEL / CentOS == /etc/sysconfig/iptables-config
IPTABLES_MODULES="ip_nat_ftp ip_conntrack_ftp"
# Debian / Ubuntu == /etc/modules
ip_nat_ftp
ip_conntrack_ftp
```
These can be loaded immediately using `modprobe` (`modprobe ip_nat_ftp ip_conntrack_ftp`) instead of rebooting.
Additionally, vsftpd includes a number of variables which can also be used to set the behaviour of the static/dynamic ports used in either mode:
```
# Enable PORT (active) mode FTP and force 20 for data
port_enable=YES
connect_from_port_20=YES
ftp_data_port=20
# Set a specific range of data ports in PASV (passive) mode
pasv_enable=YES
pasv_addr_resolve=NO
pasv_min_port=23000 # 0 = use any port
pasv_max_port=23100 # 0 = use any port
```
## Virtual chroot Users
This section is a description of the general method used in this article (but not exactly):
- <http://wiki.centos.org/HowTos/Chroot_Vsftpd_with_non-system_users>
The scripts on that page can be used to perform a majority of this work, however those scripts are missing a few items outlined below based on using the ftp system user to own files rather than the virtual user - use those scripts with caution.
The account user/pass are stored in a Berkeley DB format file, so the proper package must be installed to provide the `db_load` command:
```
## RHEL / CentOS
yum -y install db4-utils
## Debian / Ubuntu
apt-get update && apt-get -y install db-util
```
### vsftpd.conf
The configuration can be built to use virtual users based on the `user_config_dir` variable combined with the others; the key settings for vsftpd.conf:
```
vsftpd.conf
# These are in addition to any other normal config (SSL, etc.)
anon_world_readable_only=NO
anonymous_enable=NO
chroot_local_user=YES
guest_enable=YES
guest_username=ftp
hide_ids=YES
local_enable=YES
nopriv_user=ftp
pam_service_name=ftp
user_config_dir=/etc/vsftpd/users
userlist_enable=YES
userlist_file=/etc/vsftpd/denied_users
anon_umask=022
local_umask=022
dirlist_enable=NO
download_enable=NO
virtual_use_local_privs=YES
```
Create the core virtual chroot parent directory as listed above:
```
mkdir -p /var/ftp/virtual_users
```
Create the plain text file to store user/pass information imported to a DB:
```
# This is the input to create a DB hash file
# - for ultimate security it would not be left on the server
touch /etc/vsftpd/accounts.tmp
chmod 0600 /etc/vsftpd/accounts.tmp
```
### PAM ftp
The above config specifies an alternate PAM configuration; create the file `/etc/pam.d/ftp` with the required config to use an alternate database:
```
/etc/pam.d/ftp
auth required pam_userdb.so db=/etc/vsftpd/accounts
account required pam_userdb.so db=/etc/vsftpd/accounts
```
### SELinux
If required, enable SELinux anonymous writes:
```
setsebool -P allow_ftpd_anon_write 1
```
### Adding a User
First, the core login config for a user has to be created under the directory outlined by `user_config_dir` which matches the login name. The name _testuser_ will be used as an example:
```
/etc/vsftpd/users/testuser
dirlist_enable=YES
download_enable=YES
local_root=/var/ftp/virtual_users/testuser
write_enable=YES
```
Next, add the user to the core denied users specified by `userlist_file` above:
```
echo "testuser" >> /etc/vsftpd/denied_users
```
Create their system level user and chown their virtual user to _ftp_ instead due to the `virtual_use_local_privs` setting above:
```
useradd -d /var/ftp/virtual_users/testuser -s /sbin/nologin testuser
chmod 0750 /var/ftp/virtual_users/testuser
chown ftp:ftp /var/ftp/virtual_users/testuser
# If SELinux is active
chcon -t public_content_rw_t /var/ftp/virtual_users/testuser
```
Add their account/password to the text DB import file and create the DB:
```
echo "testuser" >> /etc/vsftpd/accounts.tmp
echo "testpass" >> /etc/vsftpd/accounts.tmp
db_load -T -t hash -f /etc/vsftpd/accounts.tmp /etc/vsftpd/accounts.db.new
mv -f /etc/vsftpd/accounts.db.new /etc/vsftpd/accounts.db
chmod 0600 /etc/vsftpd/accounts.db
```
### Reloading Users
The daemon needs reloaded after user changes:
```
service vsftpd reload
```
### Updating a User
The Update process is changing the same info used during Add:
1. Use `usermod` to alter the system user info as needed
2. Update the user/pass in `/etc/vsftpd/accounts.tmp` and rebuild `/etc/vsftpd/accounts.db`
Typically in this scenario the password is being updated.
### Removing a User
The process is an exact reverse of the Add process:
1. Remove the 2 lines in `/etc/vsftpd/accounts.tmp` for their user/pass and rebuild `/etc/vsftpd/accounts.db`
2. Remove the user from the system using `userdel`
3. Remove the user from `/etc/vsftpd/denied_users`
4. Remove the user config from `/etc/vsftpd/users/` directory
Parts of the steps can be left – for example, if the content of the old `$HOME` directory is to be preserved, do not run userdel.
## References
- <https://security.appspot.com/vsftpd.html>
- <http://slacksite.com/other/ftp.html>

193
md/xfs_info.md Normal file
View file

@ -0,0 +1,193 @@
# XFS Info
## Contents
- [Installation](#installation)
- [Operational Commands](#operational-commands)
- [Creating](#creating)
- [Checking](#checking)
- [Mounting](#mounting)
- [Resizing](#resizing)
- [Fragmentation](#fragmentation)
- [Others](#others)
- [LVm Snapshots](#lvm-snapshots)
- [Creating](#creating-1)
- [Mounting](#mounting-1)
- [Troubleshooting](#troubleshooting)
- [Filesystem Repairs](#filesystem-repairs)
- [Stalled Repair](#stalled-repair)
- [References](#references)
## Installation
Where its not already installed and available via the configured repositories (CentOS/Fedora/Ubuntu/RHEL7+):
```
yum install xfsprogs
```
or:
```
apt-get update && apt-get install xfsprogs
```
## Operational Commands
### Creating
```
# mkfs.xfs /dev/sdx1
# mkfs.xfs -d su=64k,sw=2 /dev/sdx1
```
Stripe size and width: only useful for local hardware arrays, not VMware vDisks, Cloud xen disks, SAN, etc.
- **su** = RAID controllers stripe size in bytes (or kB when used with k)
- **sw** = Number of data disks (don't count parity disks)
### Checking
```
# xfs_repair -v /dev/sdx1
```
> It is safe to use Ctrl+C to cancel `xfs_repair`
### Mounting
- Disable barriers if a write cache is present with the `nobarrier` mount option
- XFS places inodes only in the first 1TB of a disk with default 32bit inodes. Use the `inode64` option to override
```
/dev/sdx1 /mountpoint xfs inode64,nobarrier 0 0
```
### Resizing
```
# xfs_growfs /mountpoint
```
### Fragmentation
First check the fragmentation level, then start the defragmentation:
```
# xfs_db -c frag -r /dev/sdx1
# xfs_fsr /dev/sdx1
```
### Others
- **xfs\_info** - show filesystem geometry
- **xfs\_admin** - change filesystem parameters
- **xfs\_bmap** - print block map of filesystem
## LVM Snapshots
### Creating
It seems that using the xfs\_freeze command before and after the snapshot creation is recommended; there are [articles indicating](http://www.paragon-cs.com/wordpress/?p=67) that some versions of LVM2 hang though when doing this.
```
xfs_freeze -f /some/path
lvcreate -s ...
xfs_freeze -u /some/path
```
### Mounting
XFS has UUIDs which are unique identifiers of the filesystem; two file systems with same UUID can not be mounted on the same server. Possible solutions are to mount the snapshot with no UUID, or generate a new UUID for it.
No UUID mount:
```
mount -o nouuid /dev/mapper/mysql-snap /mnt/mysql
```
Generate a new UUID:
```
xfs_admin -U generate /dev/mapper/mysql-snap
```
## Troubleshooting
### Filesystem Repairs
XFS has a journal just like ext3/ext4 does; with XFS it's important to try and mount the filesystem to let that journal replay it's transactions to ensure data consistency; however, this is not always possible, and a xfs\_repair may also fail - even when following the [XFS FAQ](http://xfs.org/index.php/XFS_FAQ#Q:_I_see_applications_returning_error_990_or_.22Structure_needs_cleaning.22.2C_what_is_wrong.3F) about this error:
```
# mount /data01
mount: Structure needs cleaning
# umount /data01
umount: /data01: not mounted
# xfs_repair /dev/mapper/data01
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem
to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem,
then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause
corruption -- please attempt a mount of the filesystem before doing this.
```
In this case, your only solution is to tell xfs\_repair to zero out the journal – this could possibly lose data, so it should really be your last option after trying the above.
```
# xfs_repair -L /dev/mapper/data01
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used.
- scan filesystem freespace and inode maps...
bad magic # 0xb10bb320 for agi 27
bad version # 322677957 for agi 27
bad sequence # -517814273 for agi 27
bad length # -1743834476 for agi 27, should be 159793115
reset bad agi for ag 27
[...]
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
error following ag 27 unlinked list
- process known inodes and perform inode discovery...
- agno = 0
2b71c2c29940: Badness in key lookup (length)
bp=(bno 7756256, len 16384 bytes) key=(bno 7756256, len 8192 bytes)
[...]
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
Phase 5 - rebuild AG headers and trees...
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
disconnected inode 15512571, moving to lost+found
disconnected inode 51038624, moving to lost+found
Phase 7 - verify and correct link counts...
done
```
### Stalled Repair
Older versions of xfs\_repair may have bugs such as [this one](http://oss.sgi.com/archives/xfs/2009-09/msg00124.html) - unfortunately no newer versions are packaged for older RHEL5 systems. The way to feel the bug is if it hits Phase 6 and you use something like top and see no activity on the system (xfs\_repair looks idle, no I/O wait, etc.). It is safe to hit Ctrl+C and restart the xfs\_repair with the additional flags.
## References
- <http://www.redhat.com/products/enterprise-linux-add-ons/file-systems/>
- <http://oss.sgi.com/projects/xfs/>
- <http://xfs.org/index.php/XFS_FAQ>
- <http://oss.sgi.com/archives/xfs/2009-09/msg00124.html>