It may not initially be obvious why your Puppet configuration management and your Icinga monitoring should be coordinated since they are products in such different disciplines. After a bit of thought and examination it should become more and more clear that your Puppet system is probably the best way to create the bulk of your Icinga monitoring configuration, and that your Icinga cluster is just about the perfect tool for monitoring your Puppet infrastructure. A successful implementation requires a bit of careful planning but can yield excellent results.
The first step is always going to be having Puppet automatically install and configure the Icinga client on your monitored hosts, but what about creating their configurations on the Icinga master? This can be a time consuming effort if there are a large number of hosts to monitor, but with a reasonable amount of effort your Puppet installation can help out significantly.
|Figure 1 – How Icinga and Puppet work with each other.|
Generating Icinga Configuration with Puppet
In order to generate Icinga configuration with Puppet a decision must first be made about the approach: should static configuration files on the Icinga master be created by Puppet runs, or should your Puppet runs communicate directly with the Icinga Core API, a REST API for creating dynamic monitoring configuration objects within Icinga. Your best initial approach may have a lot to do with the history and size of your existing Icinga footprint.
Generating Static Configuration
If you are currently using Icinga and have a lot of manual configuration invested in connecting it to clients on existing hosts, you should consider having Puppet generate static configuration files for new machines. One way to do this is to have a Puppet run on a machine create an Icinga Host object file that describes the services that system provides, push that file to the Icinga master and cause it to reload its configuration.
This last item, reloading the configuration, is typically low-cost on a relatively small system (up to 200 or 300 hosts monitored), but can get expensive on a much larger system monitoring thousands of hosts. If the configuration reload needs to be measured in minutes rather than seconds consideration should be given to generating dynamic configurations instead. As with all Puppet jobs on any size platform it is important to make sure that the job is idempotent, not trying to make changes or reloading configurations if no changes have actually been made.
Generating Dynamic Configuration
Alternatively, a dynamic configuration methodology can be implemented using the Icinga Core Rest API. This API allows the creation of hosts and services within the Icinga configuration without the need to write configuration files to the master and without causing the monitoring daemon to reload the entire set of configuration files.
The trickier aspect of this method is making it idempotent, to make sure not to tell your Icinga master that there are changes without any actual changes to what is being monitored. API calls are not expensive but they are also not zero cost, both to the Puppet client running on your host or to the Icinga master processing the request. Also, redefining existing hosts and services continuously may interrupt the Icinga monitoring history, preventing historical data and reporting from working correctly.
Monitoring your Puppet Infrastructure
In addition to thinking about how your Puppet system can populate data for your Icinga monitoring, you should also use Icinga to monitor your Puppet servers, ensuring that they are both running and healthy and that they are providing the services necessary for your configuration management to continue.
For a Puppetmaster server, monitoring the basics is essential to assuring health, but checking that it is listening on the correct ports and required processes are running is crucial as well.
Your Puppet clients should be monitored by Icinga as well, just as you would any of your hosts, but you should specifically check that the “puppet” agent” process is running continuously.
Although it requires a significant upfront investment to integrate Puppet into your Icinga implementation, new hosts will be automatically and accurately added to your monitoring system avoiding human error and mundane, repetitive work. As your environment grows, this benefit becomes greater and greater.
Want to learn more about Puppet and Icinga?
Join us for Icinga Camp. Come join us October 18 as we discuss monitoring best practices, complementary add-ons (with use cases), and the future roadmap of Icinga. Seats are limited, so secure your spot today! We’ll start with a welcome reception at 11:00 am and we’ll have a full day of monitoring, metrics, trends, logs, and of course, Icinga!
What you get:
- A full day of monitoring madness (Madness, I tell you!)
- Great food and soft drinks during the day (One word: yum)
- Free Icinga Camp T-Shirt (What else are you going to wear this weekend?)
- Free WiFi (Who doesn’t love free Wifi? No one, that’s who.)
Whether you are already in San Diego for PuppetConf or you’re just coming for us, we look forward to seeing you in October!
Click HERE for 50% off registration fees!