Setting up Docker for Direct LVM
Devicemapper for the rest of us.
So guess what isn't in the upstream kernel? If you said AUFS then you'd be right.For all of us using RHEL/CentOS/OEL you might have noticed the default storage backend for Docker is devicemapper. You might have also noticed that it uses loop files for data storage which are slow and have a host of other problems. Seriously just do a google search on devicemapper + docker + problems.
According to the Docker graph driver README.
The device mapper graphdriver uses the device mapper thin provisioning module (dm- thinp) to implement CoW snapshots. The preferred model is to have a thin pool reserved outside of Docker and passed to the daemon via the --storage-opt dm.thinpooldev option.
As a fallback if no thin pool is provided, loopback files will be created. Loopback is very slow, but can be used without any pre-configuration of storage. It is strongly recommended that you do not use loopback in production. Ensure your Docker daemon has a --storage-opt dm.thinpooldev argument provided.
Well that's cool...
Thinpool?
So what is a thin pool anyway?
Blocks in a standard logical volume are allocated when the LV is created, but blocks in a thin provisioned logical volume are allocated as they are written. Because of this, a thin provisioned LV is given a virtual size, and can then be much larger than physically available storage. The amount of physical storage provided for thin provisioned LVs can be increased later as the need arises.
Basically I'm presenting a Volume that claims to have x amount of space when in reality it only takes up space as needed. In addition I can grow that volume as needed so Docker doesn't run out of space.
Getting Started
So the easiest thing to do is grab a spare disk and slap it in your machine. I'm not going to get in the weeds on how to do that but suffice it to say you would like to see something like this.
> sudo fdisk -l
Disk /dev/sdb: 299.4 GB, 299439751168 bytes, 584843264 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sda: 299.4 GB, 299439751168 bytes, 584843264 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0000ca89
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 526335 262144 83 Linux
/dev/sda2 526336 67635199 33554432 82 Linux swap / Solaris
/dev/sda3 67635200 88606719 10485760 83 Linux
/dev/sda4 88606720 584843263 248118272 5 Extended
/dev/sda5 88608768 584843263 248117248 8e Linux LVM
Disk /dev/sdc: 999.7 GB, 999653638144 bytes, 1952448512 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000
That last disk there is the one I'm going to use. Docker has a really good walk through on setting up direct LVM, but I'm going to cheat a little. The Project Atomic team has a nifty little script that will configure a thin volume for you. Docker Storage Setup
jray@dbox:~
> git clone https://github.com/projectatomic/docker-storage-setup
Cloning into 'docker-storage-setup'...
remote: Counting objects: 300, done.
remote: Total 300 (delta 0), reused 0 (delta 0), pack-reused 300
Receiving objects: 100% (300/300), 100.77 KiB | 0 bytes/s, done.
Resolving deltas: 100% (174/174), done.
jray@dbox:~
> cd docker-storage-setup
jray@dbox:~/docker-storage-setup [git:master]
> ls
docker-storage-setup.1 docker-storage-setup-override.conf docker-storage-setup.sh libdss.sh
docker-storage-setup.conf docker-storage-setup.service docker-storage-setup.spec NEWS.md
What I care about here is the docker-storage-setup.conf, docker-storage-setup.sh, and libdss.sh. The first time I did this ran in to issues where one of the functions was missing from the setup script. Half way through writing up a GitHub issue I realized that this was intended to be installed as an rpm. Sure enough if you look at the script it tries to source libdss.sh which doesn't exist in the place the script thinks it should. This means I need to make a small modification so that the script will run correctly.
598
599 # Source library
600 if [ -e /usr/lib/docker-storage-setup/libdss.sh ]; then
601 source /usr/lib/docker-storage-setup/libdss.sh
602 fi
603
604 source ./libdss.sh # I added this :)
The script also looks for a configuration file in /etc/sysconfig/docker-storage-setup. You can copy docker-storage-setup.conf to /etc/sysconfig/docker-storage-setup but there are really on four options I'm going to use.
jray@dbox:~/docker-storage-setup [git:master+]
> vim /etc/sysconfig/docker-storage-setup
# Config file for the PA docker-storage-setup script
STORAGE_DRIVER=devicemapper
DEVS=/dev/sdc
VG=dvol
DATA_SIZE=75%FREE
Here we let the script know that we want to use devicemapper as our storage driver, that we are going to use /dev/sdc as our disk, that we want to name our volume group dvol, and that I want the size of the data volume to be 75% of the free space on the disk.
NOTE:
The docker-storage-config.sh script is a one off thing. You should modify this file every time you run it for that instance of the script.
So let's run our script and see what we get.
jray@dbox:~/docker-storage-setup [git:master+]
> sudo ./docker-storage-setup.sh
Checking that no-one is using this disk right now ...
OK
Disk /dev/sdc: 121534 cylinders, 255 heads, 63 sectors/track
Old situation:
Units: cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sdc1 0 - 0 0 0 Empty
/dev/sdc2 0 - 0 0 0 Empty
/dev/sdc3 0 - 0 0 0 Empty
/dev/sdc4 0 - 0 0 0 Empty
New situation:
Units: sectors of 512 bytes, counting from 0
Device Boot Start End #sectors Id System
/dev/sdc1 2048 1952448511 1952446464 8e Linux LVM
/dev/sdc2 0 - 0 0 Empty
/dev/sdc3 0 - 0 0 Empty
/dev/sdc4 0 - 0 0 Empty
Warning: partition 1 does not end at a cylinder boundary
Warning: no primary partition is marked bootable (active)
This does not matter for LILO, but the DOS MBR will not boot this disk.
Successfully wrote the new partition table
Re-reading the partition table ...
If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
Physical volume "/dev/sdc1" successfully created
Volume group "dvol" successfully created
Rounding up size to full physical extent 956.00 MB
Logical volume "docker-poolmeta" created.
Wiping xfs signature on /dev/dvol/docker-pool.
Logical volume "docker-pool" created.
WARNING: Converting logical volume dvol/docker-pool and dvol/docker-poolmeta to pool's data and metadata volumes.
THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Converted dvol/docker-pool to thin pool.
Logical volume "docker-pool" changed.
Let's check out our new pool.
jray@dbox:~
> sudo lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
docker-pool dvol twi-a-t--- 697.55G 0.00 0.30
backup vglocal -wi-ao---- 5.00G
fisc vglocal -wi-ao---- 4.00G
home vglocal -wi-ao---- 4.00G
log vglocal -wi-ao---- 4.00G
opt vglocal -wi-ao---- 2.00G
perf vglocal -wi-ao---- 4.00G
root vglocal -wi-ao---- 12.00G
tmp vglocal -wi-ao---- 2.00G
tmq vglocal -wi-ao---- 4.00G
var vglocal -wi-ao---- 4.00G
jray@dbox:~
> sudo pvs
PV VG Fmt Attr PSize PFree
/dev/sda5 vglocal lvm2 a-- 236.62G 191.62G
/dev/sdc1 dvol lvm2 a-- 931.00G 231.58G
jray@dbox:~
> sudo vgs
VG #PV #LV #SN Attr VSize VFree
dvol 1 1 0 wz--n- 931.00G 231.58G
vglocal 1 10 0 wz--n- 236.62G 191.62G
You can also see the applicable device files in /dev/dvol and /dev/mapper.
Systemd
If you were to restart docker right now you might notice that running docker info would still reflect the old loop files instead of your new LVM Thin Volume. That's because you forgot to edit the systemd startup. While you can search through the daemon options the Atomic folks have done you a solid.
jray@dbox:~
> cat /etc/sysconfig/docker-storage
DOCKER_STORAGE_OPTIONS=--storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/dvol-docker--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true
Grab that storage options line and edit your systemd startup file.
jray@dbox:~
> sudo vim /etc/systemd/system/docker.service.d/daemon.conf
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// -g /docker --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/dvol-docker--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true
# Start Docker
jray@dbox:~
> sudo systemctl start docker
# Check yourself
jray@dbox:~
> docker info
Containers: 0
Images: 20
Server Version: 1.9.0
Storage Driver: devicemapper
Pool Name: dvol-docker--pool
Pool Blocksize: 65.54 kB
Base Device Size: 107.4 GB
Backing Filesystem: xfs
...
Keep Calm and Contain
