Chef, Inspec, and Dirty COW: Using Compliance to remediate CVE-2016-5195

Many of you know about or will hear about CVE-2016-5195 aka Dirty COW.

CVE-2016-5195

Bugzilla

FAQ

This particularly nasty kernel vulnerability has been around for years and likely affects a majority of the Linux nodes are currently running. The short form is that it allows an unprivileged user to gain root access to a system. I’m not going to go into actually exploiting this vulnerability but I am going to show you how to detect and remediate using Inspec and Chef.

Using Chef Compliance to Detect CVE-2016-5195

If you don’t already have a master CVE profile go ahead and create one.

[[code]]czozMzpcImluc3BlYyBpbml0IHByb2ZpbGUgQ1ZFLTIwMTYtNTE5NVwiO3tbJiomXX0=[[/code]]

Go ahead and edit the inspec.yml file so that it accurately reflects what we are doing.

inspec.yml

[[code]]czo0MDE6XCJuYW1lOiBDVkUtMjAxNi01MTk1DQp0aXRsZTogQ1ZFLTIwMTYtNTE5NSBha2EgRGlydHkgQ09XDQptYWludGFpbmVyOiB7WyYqJl19Sm9obiBSLiBSYXkNCmNvcHlyaWdodDogSm9obiBSLiBSYXkNCmNvcHlyaWdodF9lbWFpbDogaWFAam9obnJheS5pbw0KbGljZW5zZXtbJiomXX06IEFsbCBSaWdodHMgUmVzZXJ2ZWQNCnN1bW1hcnk6IFJlZCBIYXQgUHJvZHVjdCBTZWN1cml0eSBoYXMgYmVlbiBtYWRlIGF3YXJle1smKiZdfSBvZiBhIHZ1bG5lcmFiaWxpdHkgaW4gdGhlIExpbnV4IGtlcm5lbCB0aGF0IGhhcyBiZWVuIGFzc2lnbmVkIENWRS0yMDE2LTUxOTV7WyYqJl19LiBUaGlzIGlzc3VlIHdhcyBwdWJsaWNseSBkaXNjbG9zZWQgb24gT2N0b2JlciAxOSwgMjAxNiBhbmQgaGFzIGJlZW4gcmF0ZWQgYXtbJiomXX1zIEltcG9ydGFudC4NCnZlcnNpb246IDAuMS4wXCI7e1smKiZdfQ==[[/code]]

Add something to the README.md while you are at it.
[[code]]czoyNzI6XCIjIEluc3BlYyBwcm9maWxlIGZvciBkZXRlY3RpbmcgQ1ZFLTIwMTYtNTE5NSBha2EgRGlydHkgQ09XLg0KDQpUaGlzIHB7WyYqJl19cm9maWxlIGNvbnRhaW5zIG9uZSBjb250cm9sIHdoaWNoIGV4ZWN1dGVzIHRoZSBwcm92aWRlZCBSZWQgSGF0IGRldGVjdGlvbiBzY3tbJiomXX1yaXB0IHRvIGFzc2VzcyB2dWxuZXJhYmlsaXR5Lg0KDQpUaGUgc2NyaXB0IGNhbiBiZSBmb3VuZCBhdCBodHRwczovL2FjY2Vzcy5ye1smKiZdfWVkaGF0LmNvbS9zaXRlcy9kZWZhdWx0L2ZpbGVzL3JoLWN2ZS0yMDE2LTUxOTVfMi5zaFwiO3tbJiomXX0=[[/code]]

Red Hat has conveniently provided a script that can be used to determine if your system has an affected kernel so there is no point in rewriting that particular piece of code. The resulting inspec control looks like this:
[[code]]czo5NzY6XCJ0aXRsZSBcJ0NWRS0yMDE2LTUxOTVcJw0KDQojIFJ1biB0aGUgUmVkIEhhdCBjaGVjayBzY3JpcHQgdG8gc2VlIGlmIHdlIHtbJiomXX1hcmUgdnVsbmVyYWJsZQ0KY29udHJvbCBcJ0NWRS0yMDE2LTUxOTVcJyBkbw0KICBpbXBhY3QgMS4wDQogIHRpdGxlIFwnQ1ZFLTIwMTYte1smKiZdfTUxOTUgYWthIERpcnR5IENPV1wnDQogIGRlc2MgXCdBIHJhY2UgY29uZGl0aW9uIHdhcyBmb3VuZCBpbiB0aGUgd2F5IHRoZSBMaW51eHtbJiomXX0ga2VybmVsXFxcJ3MgbWVtb3J5IHN1YnN5c3RlbSBoYW5kbGVkIHRoZSBjb3B5LW9uLXdyaXRlIChDT1cpIGJyZWFrYWdlIG9mIHByaXZ7WyYqJl19YXRlIHJlYWQtb25seSBtZW1vcnkgbWFwcGluZ3MuIEFuIHVucHJpdmlsZWdlZCBsb2NhbCB1c2VyIGNvdWxkIHVzZSB0aGlzIGZsYXtbJiomXX13IHRvIGdhaW4gd3JpdGUgYWNjZXNzIHRvIG90aGVyd2lzZSByZWFkLW9ubHkgbWVtb3J5IG1hcHBpbmdzIGFuZCB0aHVzIGluY3Jle1smKiZdfWFzZSB0aGVpciBwcml2aWxlZ2VzIG9uIHRoZSBzeXN0ZW0uDQoNClRoaXMgY291bGQgYmUgYWJ1c2VkIGJ5IGFuIGF0dGFja2VyIHR7WyYqJl19byBtb2RpZnkgZXhpc3Rpbmcgc2V0dWlkIGZpbGVzIHdpdGggaW5zdHJ1Y3Rpb25zIHRvIGVsZXZhdGUgcHJpdmlsZWdlcy4gQW4gZXtbJiomXX14cGxvaXQgdXNpbmcgdGhpcyB0ZWNobmlxdWUgaGFzIGJlZW4gZm91bmQgaW4gdGhlIHdpbGQuIFRoaXMgZmxhdyBhZmZlY3RzIG1ve1smKiZdfXN0IG1vZGVybiBMaW51eCBkaXN0cmlidXRpb25zLg0KDQogIGlmIG9zWzpmYW1pbHldID09IFwncmVkaGF0XCcNCiAgICBzY3JpcHQgPXtbJiomXX0gRmlsZS5qb2luKEZpbGUuZGlybmFtZShfX0ZJTEVfXyksIFwncmgtY3ZlLTIwMTYtNTE5NV8yLnNoXCcpDQogICAgZGVzY3JpYmUgYmF7WyYqJl19c2goc2NyaXB0KSBkbw0KICAgICAgaXRzKFwnc3Rkb3V0XCcpIHsgc2hvdWxkIG1hdGNoIC9OT1QgdnVsbmVyYWJsZXxub3JtYWxseSB2e1smKiZdfXVsbmVyYWJsZXxZb3UgaGF2ZSBhIHBhcnRpYWwgbWl0aWdhdGlvbiBhcHBsaWVkLyB9DQogICAgZW5kDQogIGVuZA0KZW5kXCI7e1smKiZdfQ==[[/code]]

I’ve added an OS family check into the control just in case I want to go back and expand this control to other operating systems as the method of checking and mitigation is slightly different.

At this point we can either upload the profile to the Compliance server, where it can be applied to any number of registered systems, or we can run it one off like so:

[[code]]czoxMjAyOlwiaW5zcGVjIGV4ZWMgQ1ZFLTIwMTYtNTE5NQ0KRg0KDQpGYWlsdXJlczoNCg0KICAxKSBCYXNoIGNvbW1hbmQgQ1ZFLTJ7WyYqJl19MDE2LTUxOTUvY29udHJvbHMvcmgtY3ZlLTIwMTYtNTE5NV8yLnNoIHN0ZG91dCBzaG91bGQgbWF0Y2ggL05PVCB2dWxuZXJhYmxlfHtbJiomXX1ub3JtYWxseSB2dWxuZXJhYmxlfFlvdSBoYXZlIGEgcGFydGlhbCBtaXRpZ2F0aW9uIGFwcGxpZWQvDQogICAgIEZhaWx1cmUvRXJye1smKiZdfW9yOiBERUZBVUxUX0ZBSUxVUkVfTk9USUZJRVIgPSBsYW1iZGEgeyB8ZmFpbHVyZSwgX29wdHN8IHJhaXNlIGZhaWx1cmUgfQ0KDQp7WyYqJl19ICAgICAgIGV4cGVjdGVkIFwiXFxlWzE7MzFtWW91ciBrZXJuZWwgaXMgXFxlWzBtMy4xMC4wLTMyNy5lbDcueDg2XzY0XFxlWzE7MzFtIHd7WyYqJl19aGljaCBJUyB2dWxuZXJhYmxlLlxcZVswbVxcblJlZCBIYXQgcmVjb21tZW5kcyB0aGF0IHlvdSB1cGRhdGUgeW91ciBrZXJuZWwuIEFse1smKiZdfXRlcm5hdGl2ZWx5LCB5b3UgY2FuIGFwcGx5IHBhcnRpYWxcXG5taXRpZ2F0aW9uIGRlc2NyaWJlZCBhdCBodHRwczovL2FjY2Vzcy5ye1smKiZdfWVkaGF0LmNvbS9zZWN1cml0eS92dWxuZXJhYmlsaXRpZXMvMjcwNjY2MSAuXFxuXCIgdG8gbWF0Y2ggL05PVCB2dWxuZXJhYmxlfG5vcntbJiomXX1tYWxseSB2dWxuZXJhYmxlfFlvdSBoYXZlIGEgcGFydGlhbCBtaXRpZ2F0aW9uIGFwcGxpZWQvDQogICAgICAgRGlmZjoNCiAgICAge1smKiZdfSAgQEAgLTEsMiArMSw0IEBADQogICAgICAgLS9OT1QgdnVsbmVyYWJsZXxub3JtYWxseSB2dWxuZXJhYmxlfFlvdSBoYXZlIGEgcGF7WyYqJl19cnRpYWwgbWl0aWdhdGlvbiBhcHBsaWVkLw0KICAgICAgICtZb3VyIGtlcm5lbCBpcyAzLjEwLjAtMzI3LmVsNy54ODZfNjQgd2hpY3tbJiomXX1oIElTIHZ1bG5lcmFibGUuDQogICAgICAgK1JlZCBIYXQgcmVjb21tZW5kcyB0aGF0IHlvdSB1cGRhdGUgeW91ciBrZXJuZWwuIEFse1smKiZdfXRlcm5hdGl2ZWx5LCB5b3UgY2FuIGFwcGx5IHBhcnRpYWwNCiAgICAgICArbWl0aWdhdGlvbiBkZXNjcmliZWQgYXQgaHR0cHM6Ly97WyYqJl19YWNjZXNzLnJlZGhhdC5jb20vc2VjdXJpdHkvdnVsbmVyYWJpbGl0aWVzLzI3MDY2NjEgLg0KDQogICAgICMgQ1ZFLTIwMTYtNTE5NXtbJiomXX0vY29udHJvbHMvY3ZlLTIwMTYtNTE5NS5yYjoxNzppbiBgYmxvY2sgKDMgbGV2ZWxzKSBpbiBsb2FkXCcNCg0KRmluaXNoZWQgaW4gMHtbJiomXX0uNzY1MjQgc2Vjb25kcyAoZmlsZXMgdG9vayAzLjA0IHNlY29uZHMgdG8gbG9hZCkNCjEgZXhhbXBsZSwgMSBmYWlsdXJlXCI7e1smKiZdfQ==[[/code]]

So bummer my node is vulnerable. We can do something similar by uploading to the Chef Compliance Server and running the profile against a node or group of nodes. Log in to your compliance server.
[[code]]czoxNzY6XCJbanJheUBjMHdwMW4gfl0kIGluc3BlYyBjb21wbGlhbmNlIGxvZ2luIC1rIGh0dHBzOi8vYzB3cDFuLmVhc3R1cy5jbG97WyYqJl19dWRhcHAuYXp1cmUuY29tIC0tdXNlcj1qcmF5IC0tcmVmcmVzaC10b2tlbj1cJyZsdDtyZWZyZXNoLXRva2VuJmd0O1wnDQoNCkFQSSBhe1smKiZdfWNjZXNzIHRva2VuIHZlcmlmaWVkIGFuZCBzdG9yZWRcIjt7WyYqJl19[[/code]]

Now upload your profile.
[[code]]czo3NDk6XCJbanJheUBjMHdwMW4gfl0kIGluc3BlYyBjb21wbGlhbmNlIHVwbG9hZCBDVkUtMjAxNi01MTk1Lw0KSSwgWzIwMTYtMTB7WyYqJl19LTI2VDEzOjQyOjQ4LjA2NzEwMSAjMzg0MzFdICBJTkZPIC0tIDogQ2hlY2tpbmcgcHJvZmlsZSBpbiBDVkUtMjAxNi01MTk1Lw0KSXtbJiomXX0sIFsyMDE2LTEwLTI2VDEzOjQyOjQ4LjA2NzE5NyAjMzg0MzFdICBJTkZPIC0tIDogTWV0YWRhdGEgT0suDQpJLCBbMjAxNi0xMC0ye1smKiZdfTZUMTM6NDI6NDguMDcwMDA1ICMzODQzMV0gIElORk8gLS0gOiBGb3VuZCAxIGNvbnRyb2xzLg0KSSwgWzIwMTYtMTAtMjZUMTM6NDJ7WyYqJl19OjQ4LjA3MDA3MiAjMzg0MzFdICBJTkZPIC0tIDogQ29udHJvbCBkZWZpbml0aW9ucyBPSy4NClByb2ZpbGUgaXMgdmFsaWQNCkdlbntbJiomXX1lcmF0ZSB0ZW1wb3JhcnkgcHJvZmlsZSBhcmNoaXZlIGF0IC90bXAvQ1ZFLTIwMTYtNTE5NTIwMTYxMDI2LTM4NDMxLTFmcnhtcTAue1smKiZdfXRhci5neg0KSSwgWzIwMTYtMTAtMjZUMTM6NDI6NDguMTAwMjc1ICMzODQzMV0gIElORk8gLS0gOiBHZW5lcmF0ZSBhcmNoaXZlIC97WyYqJl19dG1wL0NWRS0yMDE2LTUxOTUyMDE2MTAyNi0zODQzMS0xZnJ4bXEwLnRhci5nei4NCkksIFsyMDE2LTEwLTI2VDEzOjQyOjQ4LjEwMntbJiomXX02NjYgIzM4NDMxXSAgSU5GTyAtLSA6IEZpbmlzaGVkIGFyY2hpdmUgZ2VuZXJhdGlvbi4NClN0YXJ0IHVwbG9hZCB0byBqcmF5L0NWe1smKiZdfUUtMjAxNi01MTk1DQpVcGxvYWRpbmcgdG8gQ2hlZiBDb21wbGlhbmNlDQpTdWNjZXNzZnVsbHkgdXBsb2FkZWQgcHJvZmlsZVwiO3tbJiomXX0=[[/code]]

On the Compliance server under the compliance section you will now see your newly uploaded profile.

Silvrback blog image

Silvrback blog image

Go ahead and scan a node to see the report.

Silvrback blog image

Remediation

Traditional remediation means going out to every system and manually updating. The remediation for this particular vulnerability means updating the kernel. While we can use chef to do this requiring a reboot is needed and let’s just say we aren’t able to take down production right now. A short term remediation described in the bug report has you build a system tap module and run it as root. The chef code to put the short term fix in place looks like this.

Warning! I’ve tested this code but YMMV. The main issue is correctly resolving kernel-devel and kernel-debuginfo to the correct kernel version.

[[code]]czoxMTc1OlwiIyB2aW06IGZ0PWNoZWYucnVieToNCiMNCiMgQXV0aG9yOjogSm9obiBSLiBSYXkgJmx0O215LmVtYWlsQGpvaG5yYXl7WyYqJl19LmlvJmd0Ow0KIyBDb29rYm9vayBOYW1lOjogY3ZlLTIwMTYtNTE5NQ0KIyBSZWNpcGU6OiBkZWZhdWx0DQojDQoNCiMgUGxhY2Ugc3tbJiomXX1jcmlwdHMgb24gdGhlIG1hY2hpbmUNCnNjcmlwdHMgPSAldyhyaC1jdmUtMjAxNi01MTk1XzIuc2ggY3ZlLTIwMTYtNTE5NS5zdHApe1smKiZdfQ0Kc2NyaXB0cy5lYWNoIGRvIHxzfA0KICBjb29rYm9va19maWxlIFwiL3RtcC8je3N9XCIgZG8NCiAgICBvd25lciBcJ3Jvb3RcJw0KICAge1smKiZdfSBncm91cCBcJ3Jvb3RcJw0KICAgIG1vZGUgXCcwNzAwXCcNCiAgZW5kLnJ1bl9hY3Rpb24oOmNyZWF0ZSkNCmVuZA0KDQojIENoZWNrIEtle1smKiZdfXJuZWwgU3RhdHVzDQpjaGVjayA9IE1peGxpYjo6U2hlbGxPdXQubmV3KFwnL3RtcC9yaC1jdmUtMjAxNi01MTk1XzIuc2hcJykucnVuX3tbJiomXX1jb21tYW5kDQp2dWxuID0gdHJ1ZSBpZiBjaGVjay5lcnJvcj8NCg0KIyBUZW1wb3JhcnkgbWl0aWdhdGlvbiBpZiBuZWVkZWQuDQoje1smKiZdfSBJbnN0YWxsIHN5c3RlbSBwYWNrYWdlcyBuZWVkZWQgZm9yIHN0YXANCnBhY2thZ2VzID0gJVcoc3lzdGVtdGFwIGtlcm5lbC1kZXZ7WyYqJl19ZWwga2VybmVsLWRlYnVnaW5mbykNCnBhY2thZ2VzLmVhY2ggZG8gfHB8DQogIHBhY2thZ2UgcCBkbw0KICAgIGFjdGlvbiA6aW5zdHtbJiomXX1hbGwNCiAgICBvbmx5X2lmIHsgdnVsbiA9PSB0cnVlIH0NCiAgZW5kDQplbmQNCg0KIyBSdW4gdGVtcG9yYXJ5IHNjcmlwdCBzbyBwe1smKiZdfXJldmVudCBEaXJ0eSBDT1cNCiMgTm90ZTogVGhpcyBoYXMgdGhlIHNpZGUgZWZmZWN0IG9mIG1ha2luZyB2aXJ1cyBzY2FubmVycyB7WyYqJl19dXNlbGVzcyB1bnRpbCB0aGUga2VybmVsIGlzIHVwZGF0ZWQuDQpleGVjdXRlIFwnUnVuIFRlbXBvcmFyeSBDVkUtMjAxNi01MTk1IE17WyYqJl19aXRpZ2F0aW9uXCcgZG8NCiAgY29tbWFuZCBcJ3N0YXAgLWcgL3RtcC9jdmUtMjAxNi01MTk1LnN0cFwnDQogIG9ubHlfaWYgeyB2dWxuIHtbJiomXX09PSB0cnVlIH0NCmVuZA0KDQojIENsZWFuIHVwIHNjcmlwdHMgaWYgd2UgYXJlblwndCB2dWxlcmFibGUgYW55bW9yZQ0KZXhlY3V0ZXtbJiomXX0gXCdDbGVhbiB1cCBmaWxlc1wnIGRvDQogIGNvbW1hbmQgXCdybSAtcmYgL3RtcC97Y3ZlLTIwMTYtNTE5NS5zdHAscmgtY3ZlLTIwMTYte1smKiZdfTUxOTVfMi5zaH1cJw0KICBub3RfaWYgeyB2dWxuID09IHRydWUgfQ0KZW5kXCI7e1smKiZdfQ==[[/code]]

The systemtap module looks like this.
[[code]]czozMDQ6XCJwcm9iZSBrZXJuZWwuZnVuY3Rpb24oXCJtZW1fd3JpdGVcIikuY2FsbCA/IHsNCiAgICAgICAgJGNvdW50ID0gMA0KfQ0KDXtbJiomXX0KcHJvYmUgc3lzY2FsbC5wdHJhY2UgeyAgLy8gaW5jbHVkZXMgY29tcGF0IHB0cmFjZSBhcyB3ZWxsDQogICAgICAgICRyZXF1ZXN0e1smKiZdfSA9IDB4ZmZmDQp9DQoNCnByb2JlIGJlZ2luIHsNCiAgICAgICAgcHJpbnRrKDAsIFwiQ1ZFLTIwMTYtNTE5NSBtaXRpZ2F0aW9uIGxve1smKiZdfWFkZWRcIikNCn0NCg0KcHJvYmUgZW5kIHsNCiAgICAgICAgcHJpbnRrKDAsIFwiQ1ZFLTIwMTYtNTE5NSBtaXRpZ2F0aW9uIHVubG9hZHtbJiomXX1lZFwiKQ0KfVwiO3tbJiomXX0=[[/code]]

We can now mitigate by applying the resulting recipe to a node.

Putting it all together

Using compliance we can run the vulnerability scan against many nodes and see a report of the number of nodes affected. Using Chef we can the remediate the issue. What we gain here is a common platform, Chef, and language, Ruby/Inspec/DSL, to not only describe an audit but also remediate it if needed. In an environment with a running Compliance server and a running chef server we would not be able to not only detect, but also remediate and then prove that we had remediated, by providing the end client with exactly what commands we used to validate our results.

Happy Hacking!

Related Posts

Filled under: Uncategorized