John R. Ray

It Worked in Development...

Thu, 09 Mar 2017 14:18:00 -0500

Everywhere you look are pizza boxes and half-empty Red Bulls. Admins, illuminated by the glow of 30″ monitors, sit hunched over keyboards typing furiously. It’s 3 AM, the day of your new software release and right now those admins are desperately trying to rollback a botched deployment.

Suddenly the status widgets, which had to be silenced to stop the flood of emails and text messages, blink green. Disaster averted, you send your team home while you prepare for post mortem. After spending hours trying to figure out what went wrong, you are no closer to the answer. After all, it worked fine in development.

So let’s unpack that last statement a second. What’s so special about Development? Why do applications always seem to work there, but not in production?

The answer is that it’s development. It’s not Production, it is different. We constantly allow things to happen in Dev that we would never allow in production. We relax the root ssh requirement, have less restrictive firewall rules, newer versions of Java, and the latest Ruby gems. Development often times looks nothing like your production environment.

Are you ready to put a stop to that? Here are three ways to get started.

1. Provision Machines From Ground Zero

The cost of the cloud is cheap. The cost of keeping sysadmins at the office until 3 AM to rollback a botched deployment is not. Whether it’s your own internal IaaS offering or Azure’s new hotness, it’s time to stop treating your servers as pets.

Always provision from nothing, or at least a sane baseline, and let your configuration management tool do the rest. Environmental problems should NEVER be fixed by hand. Hand fixes aren’t documented and are a major cause of “It works on my machine.”

When your infrastructure is defined as code you can reliably and repeatedly build the same infrastructure across your enterprise. Fix the problem in your configuration management (CM) code and rebuild.

2. Create Security Requirements for Dev Too

Security shouldn’t be an afterthought. It’s part of your environment and should be treated as a first class citizen.

Stick to your security policy no matter what environment you are in. Don’t allow anyone to bend the rules without a REALLY GOOD and REALLY DOCUMENTED reason. Relaxed security rules in Dev almost always lead to security holes in Prod.

Production isn’t the only place you should be scanning for compliance. Tools like InSpec allow you to describe your security requirements as code and easily assess your security posture no matter what environment you are in. Bonus points for bundling it in with your developers Rake file so that they can validate compliance even on their local machine.

3. Abstract Environments with Containers

Security tends to make developers cringe. The old saying goes that the more secure something is the harder it is to use. This way of thinking is especially prevalent when the developers have to worry about the underlying operating system.

Container technologies like Docker allow you to abstract the underlying infrastructure. How many of you would like to code in the exact same environment on your laptop as is running in production? Docker, in many ways, is the answer to the question of why your code works in Development but not in production.

Docker images function like version control for your applications entire environment. The image contains ALL of the dependencies required for your application to run.

When you deploy a container from that image the environment goes with it. Not only does this prevent the dreaded up until 3 AM rollbacks, but think of all the money you are going to save on Pizza?

Want to learn more? Here are a few articles.

Chef Provisioning
InSpec
Dockerizing Applications
Docker: A Quick Look

Happy Hacking!

First Published HERE

Chef, Inspec, and Dirty COW

Mon, 28 Nov 2016 12:58:57 -0500

Many of you know about or will hear about CVE-2016-5195 aka Dirty COW.

CVE-2016-5195
Bugzilla
FAQ

This particularly nasty kernel vulnerability has been around for years and likely affects a majority of the Linux nodes are currently running. The short form is that it allows an unprivileged user to gain root access to a system. I'm not going to go into actually exploiting this vulnerability but I am going to show you how to detect and remediate using Inspec and Chef.

Using Chef Compliance to Detect CVE-2016-5195

If you don't already have a master CVE profile go ahead and create one.

inspec init profile CVE-2016-5195

Go ahead and edit the inspec.yml file so that it accurately reflects what we are doing.

inspec.yml

name: CVE-2016-5195
title: CVE-2016-5195 aka Dirty COW
maintainer: John R. Ray
copyright: John R. Ray
copyright_email: ia@johnray.io
license: All Rights Reserved
summary: Red Hat Product Security has been made aware of a vulnerability in the Linux kernel that has been assigned CVE-2016-5195. This issue was publicly disclosed on October 19, 2016 and has been rated as Important.
version: 0.1.0

Add something to the README.md while you are at it.

# Inspec profile for detecting CVE-2016-5195 aka Dirty COW.

This profile contains one control which executes the provided Red Hat detection script to assess vulnerability.

The script can be found at https://access.redhat.com/sites/default/files/rh-cve-2016-5195_2.sh

Red Hat has conveniently provided a script that can be used to determine if your system has an affected kernel so there is no point in rewriting that particular piece of code. The resulting inspec control looks like this:

title 'CVE-2016-5195'

# Run the Red Hat check script to see if we are vulnerable
control 'CVE-2016-5195' do
  impact 1.0
  title 'CVE-2016-5195 aka Dirty COW'
  desc 'A race condition was found in the way the Linux kernel\'s memory subsystem handled the copy-on-write (COW) breakage of private read-only memory mappings. An unprivileged local user could use this flaw to gain write access to otherwise read-only memory mappings and thus increase their privileges on the system.

This could be abused by an attacker to modify existing setuid files with instructions to elevate privileges. An exploit using this technique has been found in the wild. This flaw affects most modern Linux distributions.

  if os[:family] == 'redhat'
    script = File.join(File.dirname(__FILE__), 'rh-cve-2016-5195_2.sh')
    describe bash(script) do
      its('stdout') { should match /NOT vulnerable|normally vulnerable|You have a partial mitigation applied/ }
    end
  end
end

I've added an OS family check into the control just in case I want to go back and expand this control to other operating systems as the method of checking and mitigation is slightly different.

At this point we can either upload the profile to the Compliance server, where it can be applied to any number of registered systems, or we can run it one off like so:

inspec exec CVE-2016-5195
F

Failures:

  1) Bash command CVE-2016-5195/controls/rh-cve-2016-5195_2.sh stdout should match /NOT vulnerable|normally vulnerable|You have a partial mitigation applied/
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected "\e[1;31mYour kernel is \e[0m3.10.0-327.el7.x86_64\e[1;31m which IS vulnerable.\e[0m\nRed Hat recommends that you update your kernel. Alternatively, you can apply partial\nmitigation described at https://access.redhat.com/security/vulnerabilities/2706661 .\n" to match /NOT vulnerable|normally vulnerable|You have a partial mitigation applied/
       Diff:
       @@ -1,2 +1,4 @@
       -/NOT vulnerable|normally vulnerable|You have a partial mitigation applied/
       +Your kernel is 3.10.0-327.el7.x86_64 which IS vulnerable.
       +Red Hat recommends that you update your kernel. Alternatively, you can apply partial
       +mitigation described at https://access.redhat.com/security/vulnerabilities/2706661 .

     # CVE-2016-5195/controls/cve-2016-5195.rb:17:in `block (3 levels) in load'

Finished in 0.76524 seconds (files took 3.04 seconds to load)
1 example, 1 failure

So bummer my node is vulnerable. We can do something similar by uploading to the Chef Compliance Server and running the profile against a node or group of nodes. Log in to your compliance server.

[jray@c0wp1n ~]$ inspec compliance login -k https://c0wp1n.eastus.cloudapp.azure.com --user=jray --refresh-token=''

API access token verified and stored

Now upload your profile.

[jray@c0wp1n ~]$ inspec compliance upload CVE-2016-5195/
I, [2016-10-26T13:42:48.067101 #38431]  INFO -- : Checking profile in CVE-2016-5195/
I, [2016-10-26T13:42:48.067197 #38431]  INFO -- : Metadata OK.
I, [2016-10-26T13:42:48.070005 #38431]  INFO -- : Found 1 controls.
I, [2016-10-26T13:42:48.070072 #38431]  INFO -- : Control definitions OK.
Profile is valid
Generate temporary profile archive at /tmp/CVE-2016-519520161026-38431-1frxmq0.tar.gz
I, [2016-10-26T13:42:48.100275 #38431]  INFO -- : Generate archive /tmp/CVE-2016-519520161026-38431-1frxmq0.tar.gz.
I, [2016-10-26T13:42:48.102666 #38431]  INFO -- : Finished archive generation.
Start upload to jray/CVE-2016-5195
Uploading to Chef Compliance
Successfully uploaded profile

On the Compliance server under the compliance section you will now see your newly uploaded profile.

Go ahead and scan a node to see the report.

Remediation

Traditional remediation means going out to every system and manually updating. The remediation for this particular vulnerability means updating the kernel. While we can use chef to do this requiring a reboot is needed and let’s just say we aren't able to take down production right now. A short term remediation described in the bug report has you build a system tap module and run it as root. The chef code to put the short term fix in place looks like this.

Warning! I've tested this code but YMMV. The main issue is correctly resolving kernel-devel and kernel-debuginfo to the correct kernel version.

# vim: ft=chef.ruby:
#
# Author:: John R. Ray 
# Cookbook Name:: cve-2016-5195
# Recipe:: default
#

# Place scripts on the machine
scripts = %w(rh-cve-2016-5195_2.sh cve-2016-5195.stp)
scripts.each do |s|
  cookbook_file "/tmp/#{s}" do
    owner 'root'
    group 'root'
    mode '0700'
  end.run_action(:create)
end

# Check Kernel Status
check = Mixlib::ShellOut.new('/tmp/rh-cve-2016-5195_2.sh').run_command
vuln = true if check.error?

# Temporary mitigation if needed.
# Install system packages needed for stap
packages = %W(systemtap kernel-devel kernel-debuginfo)
packages.each do |p|
  package p do
    action :install
    only_if { vuln == true }
  end
end

# Run temporary script so prevent Dirty COW
# Note: This has the side effect of making virus scanners useless until the kernel is updated.
execute 'Run Temporary CVE-2016-5195 Mitigation' do
  command 'stap -g /tmp/cve-2016-5195.stp'
  only_if { vuln == true }
end

# Clean up scripts if we aren't vulerable anymore
execute 'Clean up files' do
  command 'rm -rf /tmp/{cve-2016-5195.stp,rh-cve-2016-5195_2.sh}'
  not_if { vuln == true }
end

The systemtap module looks like this.

probe kernel.function("mem_write").call ? {
        $count = 0
}

probe syscall.ptrace {  // includes compat ptrace as well
        $request = 0xfff
}

probe begin {
        printk(0, "CVE-2016-5195 mitigation loaded")
}

probe end {
        printk(0, "CVE-2016-5195 mitigation unloaded")
}

We can now mitigate by applying the resulting recipe to a node.

Putting it all together

Using compliance we can run the vulnerability scan against many nodes and see a report of the number of nodes affected. Using Chef we can the remediate the issue. What we gain here is a common platform, Chef, and language, Ruby/Inspec/DSL, to not only describe an audit but also remediate it if needed. In an environment with a running Compliance server and a running chef server we would not be able to not only detect, but also remediate and then prove that we had remediated, by providing the end client with exactly what commands we used to validate our results.

Happy Hacking!

Setting up Docker for Direct LVM

Thu, 13 Oct 2016 16:52:39 -0400

So guess what isn't in the upstream kernel? If you said AUFS then you'd be right.For all of us using RHEL/CentOS/OEL you might have noticed the default storage backend for Docker is devicemapper. You might have also noticed that it uses loop files for data storage which are slow and have a host of other problems. Seriously just do a google search on devicemapper + docker + problems.

According to the Docker graph driver README.

The device mapper graphdriver uses the device mapper thin provisioning module (dm- thinp) to implement CoW snapshots. The preferred model is to have a thin pool reserved outside of Docker and passed to the daemon via the --storage-opt dm.thinpooldev option.

As a fallback if no thin pool is provided, loopback files will be created. Loopback is very slow, but can be used without any pre-configuration of storage. It is strongly recommended that you do not use loopback in production. Ensure your Docker daemon has a --storage-opt dm.thinpooldev argument provided.

Well that's cool...

Thinpool?

So what is a thin pool anyway?

Blocks in a standard logical volume are allocated when the LV is created, but blocks in a thin provisioned logical volume are allocated as they are written. Because of this, a thin provisioned LV is given a virtual size, and can then be much larger than physically available storage. The amount of physical storage provided for thin provisioned LVs can be increased later as the need arises.

Basically I'm presenting a Volume that claims to have x amount of space when in reality it only takes up space as needed. In addition I can grow that volume as needed so Docker doesn't run out of space.

Getting Started

So the easiest thing to do is grab a spare disk and slap it in your machine. I'm not going to get in the weeds on how to do that but suffice it to say you would like to see something like this.

> sudo fdisk -l

Disk /dev/sdb: 299.4 GB, 299439751168 bytes, 584843264 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sda: 299.4 GB, 299439751168 bytes, 584843264 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0000ca89

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      526335      262144   83  Linux
/dev/sda2          526336    67635199    33554432   82  Linux swap / Solaris
/dev/sda3        67635200    88606719    10485760   83  Linux
/dev/sda4        88606720   584843263   248118272    5  Extended
/dev/sda5        88608768   584843263   248117248   8e  Linux LVM

Disk /dev/sdc: 999.7 GB, 999653638144 bytes, 1952448512 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000

That last disk there is the one I'm going to use. Docker has a really good walk through on setting up direct LVM, but I'm going to cheat a little. The Project Atomic team has a nifty little script that will configure a thin volume for you. Docker Storage Setup

jray@dbox:~
> git clone https://github.com/projectatomic/docker-storage-setup
Cloning into 'docker-storage-setup'...
remote: Counting objects: 300, done.
remote: Total 300 (delta 0), reused 0 (delta 0), pack-reused 300
Receiving objects: 100% (300/300), 100.77 KiB | 0 bytes/s, done.
Resolving deltas: 100% (174/174), done.
jray@dbox:~
> cd docker-storage-setup

jray@dbox:~/docker-storage-setup [git:master]
> ls
docker-storage-setup.1     docker-storage-setup-override.conf  docker-storage-setup.sh    libdss.sh
docker-storage-setup.conf  docker-storage-setup.service        docker-storage-setup.spec  NEWS.md

What I care about here is the docker-storage-setup.conf, docker-storage-setup.sh, and libdss.sh. The first time I did this ran in to issues where one of the functions was missing from the setup script. Half way through writing up a GitHub issue I realized that this was intended to be installed as an rpm. Sure enough if you look at the script it tries to source libdss.sh which doesn't exist in the place the script thinks it should. This means I need to make a small modification so that the script will run correctly.

598
599 # Source library
600 if [ -e /usr/lib/docker-storage-setup/libdss.sh ]; then
601   source /usr/lib/docker-storage-setup/libdss.sh
602 fi
603
604 source ./libdss.sh # I added this :)

The script also looks for a configuration file in /etc/sysconfig/docker-storage-setup. You can copy docker-storage-setup.conf to /etc/sysconfig/docker-storage-setup but there are really on four options I'm going to use.

jray@dbox:~/docker-storage-setup [git:master+]
> vim /etc/sysconfig/docker-storage-setup

# Config file for the PA docker-storage-setup script
STORAGE_DRIVER=devicemapper
DEVS=/dev/sdc
VG=dvol
DATA_SIZE=75%FREE

Here we let the script know that we want to use devicemapper as our storage driver, that we are going to use /dev/sdc as our disk, that we want to name our volume group dvol, and that I want the size of the data volume to be 75% of the free space on the disk.

NOTE:
The docker-storage-config.sh script is a one off thing. You should modify this file every time you run it for that instance of the script.

So let's run our script and see what we get.

jray@dbox:~/docker-storage-setup [git:master+]
> sudo ./docker-storage-setup.sh
Checking that no-one is using this disk right now ...
OK

Disk /dev/sdc: 121534 cylinders, 255 heads, 63 sectors/track
Old situation:
Units: cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdc1          0       -       0          0    0  Empty
/dev/sdc2          0       -       0          0    0  Empty
/dev/sdc3          0       -       0          0    0  Empty
/dev/sdc4          0       -       0          0    0  Empty
New situation:
Units: sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sdc1          2048 1952448511 1952446464  8e  Linux LVM
/dev/sdc2             0         -          0   0  Empty
/dev/sdc3             0         -          0   0  Empty
/dev/sdc4             0         -          0   0  Empty
Warning: partition 1 does not end at a cylinder boundary
Warning: no primary partition is marked bootable (active)
This does not matter for LILO, but the DOS MBR will not boot this disk.
Successfully wrote the new partition table

Re-reading the partition table ...

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
  Physical volume "/dev/sdc1" successfully created
  Volume group "dvol" successfully created
  Rounding up size to full physical extent 956.00 MB
  Logical volume "docker-poolmeta" created.
  Wiping xfs signature on /dev/dvol/docker-pool.
  Logical volume "docker-pool" created.
  WARNING: Converting logical volume dvol/docker-pool and dvol/docker-poolmeta to pool's data and metadata volumes.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
  Converted dvol/docker-pool to thin pool.
  Logical volume "docker-pool" changed.

Let's check out our new pool.

jray@dbox:~
> sudo lvs
  LV          VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  docker-pool dvol    twi-a-t--- 697.55G             0.00   0.30
  backup      vglocal -wi-ao----   5.00G
  fisc        vglocal -wi-ao----   4.00G
  home        vglocal -wi-ao----   4.00G
  log         vglocal -wi-ao----   4.00G
  opt         vglocal -wi-ao----   2.00G
  perf        vglocal -wi-ao----   4.00G
  root        vglocal -wi-ao----  12.00G
  tmp         vglocal -wi-ao----   2.00G
  tmq         vglocal -wi-ao----   4.00G
  var         vglocal -wi-ao----   4.00G

jray@dbox:~
> sudo pvs
  PV         VG      Fmt  Attr PSize   PFree
  /dev/sda5  vglocal lvm2 a--  236.62G 191.62G
  /dev/sdc1  dvol    lvm2 a--  931.00G 231.58G

jray@dbox:~
> sudo vgs
  VG      #PV #LV #SN Attr   VSize   VFree
  dvol      1   1   0 wz--n- 931.00G 231.58G
  vglocal   1  10   0 wz--n- 236.62G 191.62G

You can also see the applicable device files in /dev/dvol and /dev/mapper.

Systemd

If you were to restart docker right now you might notice that running docker info would still reflect the old loop files instead of your new LVM Thin Volume. That's because you forgot to edit the systemd startup. While you can search through the daemon options the Atomic folks have done you a solid.

jray@dbox:~
> cat /etc/sysconfig/docker-storage
DOCKER_STORAGE_OPTIONS=--storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/dvol-docker--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true

Grab that storage options line and edit your systemd startup file.

jray@dbox:~
> sudo vim /etc/systemd/system/docker.service.d/daemon.conf
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// -g /docker --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/dvol-docker--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true

# Start Docker
jray@dbox:~
> sudo systemctl start docker

# Check yourself
jray@dbox:~
> docker info
Containers: 0
Images: 20
Server Version: 1.9.0
Storage Driver: devicemapper
 Pool Name: dvol-docker--pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
...

Keep Calm and Contain