"An expert is a man who has made all the mistakes which can be made, in a narrow field."
Niels Bohr

When I setup Logstash for the very first time I got bitten by an empty search, aparently no logs were indexed. Reading the log files indeed told me about

1. WARN: org.elasticsearch.discovery.zen.ping.unicast: [Blaire, Allison] failed to send ping to [[#zen_unicast_1#][inet[/127.0.0.1:9300]]]
2. INFO | jvm 1 | 2012/02/06 22:45:55 | org.elasticsearch.transport.RemoteTransportException: [Page, Karen][inet[/127 .0.0.1:9300]][discovery/zen/unicast]
3. INFO | jvm 1 | 2012/02/06 22:45:55 | Caused by: java.io.EOFException

The above is the typical error when the ElasticSearch version you are using externally is not in sync with the one Logstash is using, yes those versions need to match.

Fast forward a couple of weeks.. and I`m upgrading Logstash and therefore also ElasticSearch .. I have a Vagrant setup to play with so all of the components are running on 1 node.

I kept running into a similar problem, this time however I saw log entries being indexed, I could get data from my ElasticSearch setup using
wget -q -S -O - http://localhost:9200/_status?pretty=true

But the web interface kept showing no results ;(

While nagging about it on irc .. Jordan gave me the insight :

2012-01-31.194347+0100CET.txt:(07:55:36 PM) whack: slight caveat that elasticsearch clients also join the cluster, so if you point everyone at 127.0.0.1:9300, that :9300 could be one of your clients, not the server

Indeed when you by accident start any of the logstash instances (server/shipper/web) before you start your ElasticSearch instance you can be in trouble.
Ordering really matters , you really need to start ElasticSearch before you start the clients.

Obviously is you don't use the unicast setup you don't run into this problem ..

So what other mistakes should I make ?

Kent Skaar pinged me last week , asking for feedback on Lisa'11 and input for Lisa 2012.

Thought I should share my advise to him with the rest of the world

So If I were to host an event similar to Lisa I'd had either
Jordan Sissel or Mitchell Hashimoto give the keynote because over the past 24 months those people have written more relevant tools for me than anyone else :)

I'd have someone talk about Kanban for Operations, There's 2 names that pop up Dominica DeGrandis and Mattias Skarin

I'd have the Ubuntu folks talk about JuJu and I'd have RI Pienaar talk about MCollective .. while you have RI have him talk about Hiera too. Have Dean Wilson carry RI's bags and put him unknowingly on a panel. (Masquerade it as a Pub with hidden cameras)

Obviously as #monitoringsucks you want to hear about new monitoring tools initiatives and how people are dealing with them , so you want people talking about Graphite, Collectd, Statsd, Sensu , Icinga-MQ And how people are reviving Ganglia and using that in large scale environments.

You want someone to demistify Queues, I mean .. who still knows about the differences between Active, Rabbit , Zero, Hornet and many other Q's ?

You want people talking about how they deal with logs, so talks about Logstash and Graylog2.

You want to cover Test Driven Infrastructure How do you test your infrastructure , someone to demystify Cucumber and Webrat , and talk about testing Charms, Modules, and Cookbooks.

Oh and Filesystems , distributed ones the Ceph, FraunhoverFS, Moose, KosmosFS, Glusters, Swifts of this world ... you want people to talk about their experiences , good and bad with any of the above, someone who can actually compare those rather than heresay stuff. :) With recent updates on what's going on in these projects.

Now someone please organise this for me :) In a warm and sunny place ... preferably with 27 holes next door , and daycare for my kids :)

PS. Yes the absence of any openstack related topic is on purpose .. that's for 2013 :)

MonitoringSucks and we didn't fix it.

Earlier this week Inuits hosted a 2 day hackfest titled #MonitoringSucks. A good number of people with a variety of backgrounds showed up on monday morning. I don't know why but people had high expectations for this event , did they really expect us to fix the #monitoringsucks problem in a mere 2 days ?

Next to myselve we had Patrick Debois , Grégory Karékinian, Stefan Jourdan, Colin Humphreys, Andrew Crump, Ohad Levy , Frank Marien, Toshaan Bharvani, Devdas Bhagat, Maciej Pasternacki Axel Beckert Jelle Smet, Noa Resare @blippie , John John Tedro @udoprog, Christian Trabold @ctrabold and obviously some people I missed


A good mixture of Fosdem visitors that stayed a litte longer in our cold country and locals with ideas. We had people from TomTom, RedHat , Spotify, Booking.com, Inuits, Atlassian, coming from Belgium, The Netherlands France, Israel, the UK, Sweden, Germany, Poland and Switzerland if I`m not mistaken.

The format was pretty open, much of the first day was spend around the drawingboard.

(Ohad Levy, Jelle Smet, PatrickDebois and Frank Marien) discussing a variety of topics

This monitoring topic is complex, there are different areas that need to be covered. The drawing below documents how we splitted the problem into different areas , and listed the different tools people use for these areas.

• Collection: Collectd, Nagios, Ganglia
• Transport: XMPP, Smiple, Smtp, 0mq , APMQ, rsyslog, irc, stomp
• Storage : rrd, graphite, opentsdb, hbase,
• Filtering: logstash, esper,
• Visualisation : Graphite,
• Notifcation: PagerDuty
• Reporting: Jasper

Obviously above list is far from complete.

The afternoon discussion continued where we left of before lunch, just after the powercut. Only now we started refocussing on filtering and aggregating values using Logstash
@patrickdebois had been talking about the idea to use Logstash as a way to collect data , transform it and throw it either to another tool, or onto a Queue before.
Looking at Logstash it makes kind of sense. Logstash already has a zillion of input types, filters and outputs. Including popular queues such as amqp and zeromq. Yes, the default behaviour for a lot of people is to get data from different inputs, filter it and then send it to ElasticSearch, but much more is possible with the available outputs.

It was only on tuesday that people really started writing code
So what did really come out of the #monitoringsucks hackfest. ?

A couple of people were working on packaging existing tools for their favourite distro. Others were working on integrating a number of other already existing tools (e.g Patrick working on more inputs for Logstash., me working on replacing logster with Logstash, setting up Kibana etc. New tools were learned, items were added to todolists (Kibana, (doesn't work on older Firefox instances) Tattle, statsd) and items were scratched from todolists (Graylog2 (Kibana replaces that as a good Frontend for Logstash) )

A lot of experiences with different tools were exchanged

Frank Marien showed us a demo of his freshly release ExtremeMon framework. A really promising project.

The sad part about a workshop like this one is that you enter with a bunch of ideas , and leave with even more ideas, hence more work. We haven't solved the problem yet, but a lot of more people are now thiking about the problem and how to solve it a more modulare (unix style) approach. With different litte tools, all being good at something and all being interconnectable.

As announced earlier next monday and tuesday we're opening up the Inuits offices for everybody working on monitoring problems.

There's already a good number of people that have confirmed their presence and some people have asked

As for practical details .. the plan is simple.
I`m going to be at the place somewhere between 8:30 and 9:00 on monday. ( Hey .. it's the day after Fosdem you know :))

The only thing I've planned is to do a get to know eachother round around 10:30 after that I`m expecting the hackathon to be self organising,

There will be water, coffee , etc , IP connectivity, and electricity.

The location is still Duboisstraat 50, Antwerp

Free parking is on the Hardenvoort or Kempenstraat ( 3minutes walk) , paid parking right in front of the door.

Recently (about 5 minutes ago), I was writing a custom puppet-function to offload some puppet magic. In short: I’m writing a wrapper around create_resources so I can keep syntax for the end-users of my module crispy clean. This means I need the create_resources function to be available in my custom function. This can be done by using Puppet::Parser::Functions.autoloader.loadall as suggested on the puppetlabs custom modules guide. Unfortunately, when using #loadall, all functions will be loaded.

Why unfortunately? In my case: A function defined in puppet-foreman depends on the rest-client gem and I do not have this installed. Some people might say: Just install the gem and be done with it! This is hardly a proper solution. The way to go would to be only include the function I really  need, being create_resources.

And here is how:

Puppet::Parser::Functions.autoloader.load(:create_resources) unless Puppet::Parser::Functions.autoloader.loaded?(:create_resources)

This will basically load the create_resources function after checking that it has not been loaded before. This (the function already being loaded) could be the case if you properly depend on puppetlabs-create_resources in your manifests. Side note: I added a small dummy class so my modules can depend on this function being available.

This has resolved my issues with #loadall, but if I ever needed to include another function that DOES use #loadall, I’ll be screwed all over again. So (pretty) pls, don’t use #loadall.

INTRODUCTION

I’ve used puppet quite intensively since a couple of months (about 4 I would guess). Before that, I’ve played with it, change something here and there. But quite not as much as now. I’ve used several puppet modules from wherever google leads me, roamed github, inherited a few from colleagues and created several from scratch. While doing so, I saw a lot of stuff I disliked and learned a lot on how we I can (ab)use puppet to do what I want it to do. Over those last months, I have grown my set of ideas on how a puppet module should look. So, before every statement I make, you should probably add ‘IMHO’.

WHO THE F.

Why the hell would this guy (me) have anything to say about puppet modules. Let’s situate first.

I’m now an Open Source Consultant. I’ve been (in order) a Java programmer, sysadmin, Drupal developer and now back sysadmin (doing devopsy things). Last 3 positions I worked for (and still work for): Inuits – Open Source company in Belgium. Currently, I’m positioned at UnifiedPost (About 100 people but thinking big!). I help out with daily maintenance (and there is plenty) and starting to adopt puppet as much as possible. Puppet was already in use at UP (UnifiedPost), but knowledge was rather thin as I came in. They did however manage some hosts with it (about 300-400). I dove in the puppet code rather fast and stumbled upon several patterns that increased pressure on my mouse heavily. Even modules I grabbed from the net (whatever the source is) made my grip firmer.

PROBLEMS

Before trying to fix the problem, we should find exactly what bothers me with all these modules I lay my eyes on. I’ll try to keep it organized.

1. Modules are not classes!
2. Too hard to use by non-developers
3. Poor interaction with third-party modules
4. Not versioned
5. Not pretty at all do down right f_ugly

1. Modules are not classes!

Although a module exists out of several classes, it should not behave like one.

An example to clarify what I mean: The accounts module. (I’m sure this is the case for may organizations that have an accounts module).

I can think of a several valid reasons why you would have one. (Keep reading nevertheless!). What does our accounts module contain: A definition to ease the use and do some customization (set defaults, create some files, …). We also find a list of users, passwords and authorized_ssh keys. This (specific) user information does not belong in a module. It should either be in a class (below the manifests folder) or stored externally. In my point of view: Nodes use classes. They register the kind of machine and define what should be installed. Classes include modules and change settings. Possibly parameterized so we can keep it node specific. All module parameters values reside in the node or the class(es) it includes.

2. Too hard to use by non-developers

This brings us to our next point. Can I just grab your module and start using it? Or do I need to weed out hardcoded strings, change host names or edit templates. Do I need to understand the complete way it works ‘under the hood’. I have written a short post recently, expressing my feelings about this. Bottom line: If I need to edit any file within your module to get it working the way I want it to, there is something wrong with it. Sure, features and/or support might be missing. but if my $::operatingsystem is supported, I should get it working without touching anything of the module code.

3. Poor interaction with third-party modules

I have reused (or attempted to) several modules found on github and always had the same problem, it does not play well with our current puppet-tree. The best example for this is probably the apache (or httpd) module. Almost any puppet modules that has a dependency on apache being installed, comes with its own implementation and/or dependency. Most companies already have a apache module and change the new module to work properly together with theirs. There goes upstream support. I have run into this issue with puppet-foreman recently and this will probably be my first big test case for my coding pattern.

4. Not versioned

Most modules you will find only live on one branch, master. Some may have a develop branch, but most of the time, there is no saying in what version you are using. Unless you elevate hash-tags to version numbers. (Using git submodules does this at some extend). But updating a submodule is always a dangerous thing to do, there is no way to tell what will break.

Besides the ‘version’ of a module, we also have to take the puppet version into account. I tend to be a cutting-edge user for all my software, but I can easily understand if you don’t for whatever reason. So keeping puppet-modules backwards compatible is a must (is it?).

5. Not pretty at all do down right f_ugly

Why is properly formatted code important: for anybody else that ever has to ever change or even use it. This could be a colleague or someone (anyone) else that found your module and wants to improve (here is when YOU win time, if somebody else does the job for you) or change it. Even if you did not have time for writing up documentation, most people will have to stroll through your code. Having properly formatted code is always a nice-to-have feature then.

REQUIREMENTS

So now that we know what I dislike about puppet modules, let’s try to define something more positive. What is a good baseline for a puppet-module.

0. VCS (This one is pretty obvious, I will not elaborate on this any further.)
1. Follows style guidelines
2. Use of centralized parameters / settings
3. Fully(!) parameterized
4. Easy and centralized handling of compatibility ($::operatingsystem-ish stuff)
5. Documented
6. Releases
7. Puppet compatibility
8. Integration: is uniquely identifiable
9. Easy to extend

1. Style Guidelines

Why? Some valid reasons are so you make fewer mistakes and are more aware of what you are doing Do you really need double quotes? Using single quotes for static string values will prevent you from forgetting to quote ‘$’ (commands anyone?). Always using a default case will make you more aware that more than one distro exists in the world. At least fail when your module is not fit for a certain operating system to prevent unexpected behavior. If you read through the style guideline, you will see that many of these items are easily to do if you just remember to do so when you are writing the actual code.

2. Centralized parameters / settings

Does your module support distro? Just check the params.pp file. Everything is there. It does not get much easier than this to add support for a new distro.

3. Fully Parameterized

This might seem like I’m making the same point twice, but we should differentiate between our general ‘settings’ that configure the working of our module, and specific definitions that have parameters. You define where all your vhosts are but each vhosts definition you create also takes parameters. Same rule applies for a definition as for a module, we should be able to use it without having to change anything in the code. Often, this means making more stuff than you need – at time of writing your module – dynamical (parameterized). You can hard-code a ‘Listen 80′ or template it using a $ports parameter.

4. Compatibility handling

As an exampe: my colleague asked me: Does your module work for Debian? I was happy to answer: You just need to add support to the params.pp file. That’s all whats needed (and maybe add some templates).

5. Documentation

When thinking as a developer when writing a module, we know we need to offer easy documentation for the end-user. This is no different when writing a puppet module. It’s always a good thing to keep people out of your code as much as possible. Proper documentation is the first step. I try to write as much as possible, main reason being when a colleague asks how to use it, I can point him to the documentation instead of going over the code myself to remember what exactly is going on. On a side note: ATM, I’m having some troubles with bug #11384. Votes – and a patch even more – welcome ;)

Beside top-level documentation, inline (code) documentation should also be written. Not for the obvious stuff, but when you do something more advanced, explain to a fellow coder (or yourself some weeks later) why and what you are doing.

6. Releases

Puppet modules should also have releases. This would an easier way of drawing attention when we change the API (or definition parameters) or when we fix a bug. This is also a great sign when our module is no longer backward compatible with old code (breaking API). I try to support old code as much as possible. But at some point, we will have to weed the old so it does not clutter the new. Keeping stuff simple/stupid (although I have passed that bridge a looooong time) is still a good principle.

7. Puppet Compatibility

We need to know with what version of puppet your module is compatible. Some features of the puppet language you use might or might not be available in older releases. You can check the Puppet Language Guide for what is introduced in what version, but there are a lot of other differences that are not so much documented. I’ve been using the create_resources function quite a lot but it’s only in core puppet for versions 2.7 and up. Luckily, there is a backport for 2.6 on the puppetlabs’ github.

8. Integration:  is uniquely identifiable

To improve compatibility, we first need must be able to tell what module we are integrating with. I personally started to use a $modulename and $moduleversion param in the main class of my module. Modulefiles like puppetlabs requires them for the puppet forge are cool, but we can not use them in our code.  We could write a fact for this so we don’t need to duplicate code. I won’t add this to my to-do list as I already have a way-to-long backlog, but feel free to add it to yours.

With this information, we could do different things based on the module and version we are working with.

9. Easy to extend

In this part, the developer in me is taking the overhand and it will depend on personal preference a lot more than any of the previous points. A quick example: I wanted to use a conf.d/* configuration style. Even more, for certain configuration files, order is important so we need to prefix files with 00_, 01_, … I could have easily done this for each type of configuration file I want to store here. In stead, I wrote a confd wrapper definition/class that does this for me. It’s a 2 step process: You initialize/setup a conf.d folder and then define yourresources within them. I’m realizing now that this should have been a separate module. I have added it to my to-do list.The main advantage is I can easily re-implement conf.d style folders now without worrying about the logic behind it.

SOLUTIONS

Quick wins! These go without saying, start using them now.

Check your code for formatting and style.

For this, we have puppet-lint. This tool will deal with most common problems and errors/warnings against the style guide. This tool takes one puppet manifest as argument and displays the errors/warnings it finds. You can easily integrate it with jenkins since the log-format argument has been added.

Documentation.

I suppose most people will have issues with this. Good documentation is essential and not much hard work if you do it right. I prefer to START with documenting what a class will do and implement afterwards. This is a lot like writing tests first and then use them to see if you are writing proper / working code. The danger is of course that you change the internal working but forget to update your documentation. After each feature I add, I tend to go over the documentation and see that everything is still up to date. Once documentation has been written, you can generate it using puppet doc. To work around certain puppet doc’s ugliness, I wrote a small wrapper script for my Jenkins jobs that does some post processing on them. See previous post for that.

Releases

I’ll be quick about this one: Use git-flow.

General / Initial module structure.

For creating your initial puppet module structure, there is always the puppet-module tool. Install it by installing the gem. I have tried using it, but I’m relying on my own bash magic for creating classes.

This is my basic structure I re-use over and over.

• ./manifests/init.pp
• ./manifests/params.pp
• ./manifests/packages.pp
• ./manifests/setup.pp

One note on these filenames: always try to avoid confusion! I have seen a lot of config.pp classes and params.pp classes where the config.pp actually does configuration of the package on the system while params.pp is for configuring the behavior of the puppet-module. I like setup.pp better than config.pp, since it’s easier to figure out what the class does: It sets up the system! Another good option would be install.pp.

OUTRO

I realize these solutions are no where near finished but since FOSDEM 2012 is coming up and I’m running low on time, I wanted to publish this post so anybody can starting giving their opinion on the matter before coming to a final out-of-the-box solution most people can relate to. So, actually, this is a big fat TO BE CONTINUED.

Matters we need to discuss:

• Compatibility handling (both to other modules and puppet)
• Making modules easy to integrate and/or extend.

 

Here are some tricks I use to make my vagrant boxes as small as possible:

Tips: Booting in single user mode:

I boot in single user mode since it will prevent running services that could output logs. I do this because I zero out all my logs before packaging the box.

Updating:

After updating any package, run yum clean (or the apt equivalent).

When booted in single user mode, don’t forget to start-up your network before updating.

When updating kernels, install the kernel packages, reboot and remove the old kernel packages that are no longer in use. Remember to re-install the VirtualBox add-ons too after a kernel update.

Cleanup:

After doing whatever you need to do with the box, I do some rather nasty stuff to make sure the box uses as little as possible place. If you are using a RAW hard-disks, these might be a bad idea (stuff gets BIG).

• Zero out all remaining unused disk space
• Zero out the swap
• Clear out all log files (I just make them empty, I do NOT delete them)

Script:

(You can find this script – or an older version in /root/tools/cleanup_diskspace.sh on my newer vagrant boxes.)

cat - << EOWARNING WARNING: This script will fill up your left over disk space.   DO NOT RUN THIS WHEN YOUR VIRTUAL HD IS RAW!!!!!   You should NOT do this on a running system. This is purely for making vagrant boxes damn small.   Press Ctrl+C within the next 10 seconds if you want to abort!!   EOWARNING sleep 10;   echo 'Cleanup log files'; find /var/log -type f | while read f; do echo -ne '' &gt; $f; done;   echo 'Whiteout root'; count=`df --sync -kP / | tail -n1 | awk -F ' ' '{print $4}'`; dd if=/dev/zero of=/tmp/whitespace bs=1024 count=$count; rm /tmp/whitespace;   echo 'Whiteout /boot' count=`df --sync -kP /boot | tail -n1 | awk -F ' ' '{print $4}'`; dd if=/dev/zero of=/boot/whitespace bs=1024 count=$count; rm /boot/whitespace;   ### Repeat the above for other partitions you have.   swappart=`cat /proc/swaps | tail -n1 | awk -F ' ' '{print $1}'` swapoff $swappart; dd if=/dev/zero of=$swappart mkswap $swappart; swapon $swappart;

Furthermore – about this script – USE IT AT YOUR OWN RISK

Code style checking Prerequisites:

• You will need a recent enough version of puppet-lint that supports the --log-format flag. Install the gem so that the Jenkins can use it.
• On Jenkins, you will need the Warnings Plugin and the HTML Publisher Plugin.
• Make sure that when checking the module from your VCS, it ends up in WORKSPACE/modules/module_name.

Configuration: Jenkins

Go to the Configure System page and find the Compiler Warnings settings. Add a new console log parser and call it puppet-lint. I use following configuration for parsing puppet-lint warnings and errors.

The warnings plugin has been updated and now has puppet-lint support out of the box! So configuring puppet-lint manually is kind of useless now.

Name:

puppet-lint

Regular Expression:

^\s*([^:]+):([0-9]+):([^:]+):([^:]+):\s*(.*)$

Mapping Script:

import hudson.plugins.warnings.parser.Warning // map regular expression to strings String fileName = matcher.group(1); String lineNumber = matcher.group(2); String kind = matcher.group(3); String check = matcher.group(4); String message = matcher.group(5); // return a Warning. return new Warning(fileName, Integer.parseInt(lineNumber), check, kind, message);

Example Log Message:

./manifests/params.pp:25:autoloader_layout:error:apache::params not in autoload module layout Jenkins job configuration

We will add several build steps that will run certain actions on our puppet modules.

1. Check syntax
2. Check style
3. Generate documentation

1. For the syntax check, I use following shell script (add a build step):

for file in $(find . -iname '*.pp'); do   puppet parser validate --color false --render-as s --modulepath=modules $file || exit 1; done;

2. For the style check, we use puppet-lint (add another build step):

find . -iname *.pp -exec puppet-lint --log-format "%{path}:%{linenumber}:%{check}:%{KIND}:%{message}" {} \;

3. And for generating documentation:

## Cleanup old docs. [ -d doc/ ] &amp;&amp; rm -rf doc/ ## Dummy manifests folder. ! [ -d manifests/ ] &amp;&amp; mkdir manifests/ ## Generate docs puppet doc --mode rdoc --manifestdir manifests/ --modulepath ./modules/ --outputdir doc   ## Fix docs to how I want them, I don't like that the complete workspace is included in all file paths. if [ -d ${WORKSPACE}/doc/files/${WORKSPACE}/modules ]; then mv -v "${WORKSPACE}/doc/files/${WORKSPACE}/modules" "${WORKSPACE}/doc/files/modules" fi; grep -l -R ${WORKSPACE} * | while read fname; do sed -i "s@${WORKSPACE}/@/@g" $fname; done;

In your post build section:

• Enable Scan for compiler warnings and select puppet-lint.
• Enable publish HTML reports (use ‘doc‘, ‘index.html‘ and ‘Puppet Docs‘ as values). This will add a link to the Job page linking your generated puppet docs.

That’s about it! Any suggestions / improvements on this are always welcome!

Notes:

• I have some examples/tests setup on my Jenkins instance for testing at http://jenkins.vstone.eu. Since I use this for testing, it might be offline / broken / buggy at times.
• The scripts I use may also require some changes if you are using an older version of puppet. I’m currently using 2.7.x for testing my modules.

Puppet modules…  How I feel about them in a dot file:

digraph PuppetModules {   node [     fontname = "Bitstream Vera Sans"     fontsize = 10     shape    = "record"   ]   edge [     fontname = "Bitstream Vera Sans"     fontsize = 10   ]   question [label="Do I need to edit a file in your module for changing settings?", shape="oval"] ok [label="Great.", shape="oval"] bah [label="You are doing it WRONG!", shape="oval"]   question -> ok [label="No"]   question -> bah [label="Yes"] }

Read on if you want a rendered version.

In the root directory of puppet trees, you could have a Vagrantfile. This is great, but a problem happens: when two people are working on the same tree, they will probably have their own vagrant files. That means that one of you do not commit his changes to the Vagrantfile, or you are managing multiple branches.

I have another idea: having multiple Vagrantfiles in the tree, named Vagrantfile.vm_name. Then, just add Vagrantfile to your .gitignore.

When you need to power up your specific Vagrant file, you just have to create a symbolic link like that:

$ ln -s Vagrantfile.roidelapluie Vagrantfile Easy and simple. If you want to go further you can create a "vagrantfiles" directory with all these Vagrantfiles.Tags: puppetvagrantgit

Given that @patrickdebois is working on improving data collection I thought it would be a good idea to describe the setup I currently have hacked together.

(Something which can be used as a starting point to improve stuff, and I have to write documentation anyhow)

I currently have 3 sources , and one target, which will eventually expand to at least another target and most probably more sources too.

The 3 sources are basically typical system data which I collect using collectd, However I`m using collectd-carbon from https://github.com/indygreg/collectd-carbon.git to send data to Graphite.

I`m parsing the Apache and Tomcat logfiles with logster , currently sending them only to Graphite, but logster has an option to send them to Ganglia too.

And I`m using JMXTrans to collect JMX data from Java apps that have this data exposed and send it to Graphite. (JMXTrans also comes with a Ganglia target option)

Rather than going in depth over the config it's probably easier to point to a Vagrant box I build https://github.com/KrisBuytaert/vagrant-graphite which brings up a machine that does pretty much all of this on localhost.

Obviously it's still a work in progress and lots of classes will need to be parametrized and cleaned up. But it's a working setup, and not just on my machine ..

If you are hacking on monitoring solutions, and want to talk to your peers solving the problem
Block the monday and tuesday after fosdem in your calendar !

That's right on february 6 and 7 a bunch of people interrested to fix the problem will be meeting , discussing and hacking stuff together in Antwerp

In short a #monitoringsucks hackathon

Inuits is opening up their offices for everybody who wants to join the effort Please let us (@KrisBuytaert and @patrickdebois) know if you want to join us in Antwerp

Obviously if you can't make it to Antwerp you can join the effort on ##monitoringsucks on Freenode or on Twitter.

The location will be Duboistraat 50 , Antwerp
It is about 10 minutes walk from the Antwerp Central Trainstation
Depending on Traffic Antwerp is about half an hour north of Brussels and there are hotels at walking distance from the venue.

Plenty of parking space is available on the other side of the Park

I`m parsing the responses of the Deploying Drupal survey I started a couple of months ago (more on that later)

One of the questions in the survey is "What is devops" , apparently when you ask a zillion people (ok ok, just a large bunch of Tweeps..), you get a large amount of different answers ranging from totally wrong to spot on.

So let's go over them and see what we can learn from them ..

The most Wrong definition one can give is probably :

• A buzzword

I think we've long passed the buzzword phase, definitely since it's not new, it's a new term we put to an existing practice. A new term that gives a lot of people that were already doing devops , a common word to dicuss about it. Also lots of people still seem to think that devops is a specific role, a job description , that it points to a specific group of people doing a certain job, it's not . Yes you'll see a lot of organisations looing for devops people, and giving them a devops job title. But it's kinda hard to be the only one doing devops in an organisation.

I described one of my current roles as Devops Kickstarter, it pretty much describes what I`m doing and it does contain devops :)

But devops also isn't

• The connection between operations and development.
• people that keep it running
• crazy little fellows who find beauty in black/white letters( aka code) rather than a view like that of Taj in a full moon light.
• the combination of developer and operations into one overall functionality
• The perfect mixture between a developer and a system engineer. Someone who can optimize and simplify certain flows that are required by developers and system engineers, but sometimes are just outside of the scope for both of them.
• Proxy between developer and management
• The people in charge of the build/release cycle and planning.
• A creature, made from 8-bit cells, with the knowledge of a seasoned developer, the skillset of a trained systems engineer and the perseverence of a true hacker.
• The people filling the gap between the developer world and the sysadmin world. They understand dev. issues and system issues as well. They use tools from both world to solve them.

Or

• Developers looking at the operations of the company and how we can save the company time and money

And it's definitely not

• Someone who mixes both a sysop and dev duties
• developers who know how to deploy and manage sites, including content and configuration.
• I believe there's a thin line line between Ops and Devs where we need to do parts of each others jobs (or at least try) to reach our common goal..
• A developer that creates and maintains environments tools to help other developers be more successful in building and releasing new products
• Developers who also do IT operations, or visa versa.
• Software developers that support development teams and assist with infrastructure systems

So no, developers that take on systems roles next to their own role and want to go for NoOps isn't feasable at all ..you really want collaboration, you want people with different skillsets that (try to) understand eachoter and (try to) work together towards a common goal.

Devops is also not just infrastructure as code

• Writing software to manage operations
• system administrators with a development culture.
• Bring code management to operations, automating system admin tasks.
• The melding of the art of Systems Administration and the skill of development with a focus on automation. A side effect of devops is the tearing down of the virtual wall that has existed between SA's and developers.
• Infrastructure as code.
• Applying some of the development worlds techniques (eg source control, builds, testing etc) to the operations world.
• Code for infrastructure

Sure infastructure as code is a big part of the Automation part listed in CAMS, but just because you are doing puppet/chef doesn't mean you are doing devops.
Devops is also not just continous delivery

• A way to let operations deploy sites in regular intervals to enable developers to interact on the systems earlier and make deployments easier.
• Devops is the process of how you go from development to release.

Obviously lots of people doing devops also often try to achieve Continuous delivery, but just like Infrastructure as Code it devops is not limited to that :)

But I guess the truth is somewhere in the definitions below ...

• That sweet spot between "operating system" or platform stack and the application layer. It is wanting sys admins who are willing to go beyond the normal package installers, and developers who know how to make their platform hum with their application.
• Breaking the wall between dev and ops in the same way agile breaks the wall between business and dev e.g. coming to terms with changing requirements, iterative cycles
• Not being an arsehole!
• Sysadmin best-practise, using configuration as code, and facilitating communication between sysadmins and developers, with each understanding and participating in the activities of the other.
• Devops is both the process of developers and system operators working closer together, as well as people who know (or who have worked in) both development and system operations.
• Culture collaboration, tool-chains
• Removing barriers to communication and efficiency through shared vocabulary, ideals, and business objectives to to deliver value.
• A set of principles and good practices to improve the interactions between Operations and Development.
• Collaboration between developers and sysadmins to work towards more reliable platforms
• Building a bridge between development and operations
• The systematic process of building, deploying, managing, and using an application or group of applications such as a drupal site.
• Devops is collaboration and Integration between Software Development and System Administration.
• Devops is an emerging set of principles, methods and practices for communication, collaboration and integration between software development (application/software engineering) and IT operations (systems administration/infrastructure) professionals.[1] It has developed in response to the emerging understanding of the interdependence and importance of both the development and operations disciplines in meeting an organization's goal of rapidly producing software products and services.
• bringing together technology (development) & content (management) closer together
• Making developers and admins understand each other.
• Communication between developers and systems folk.
• a cultural movement to improve agility between dev and ops
• The cultural extension of agile to bring operations into development teams.
• Tight collaboration of developers, operations team (sys admins) and QA-team.

But I can only conclude that there is a huge amount of evangelisation that still needs to be done, Lots of people still don't understand what devops is , or have a totally different view on it.

A number of technology conferences are and have taken up devops as a part of their conference program, inviting experienced people from outside of their focus field to talk about how they improve the quality of life !

There is still a large number of devops related problems to solve, so that's what I`ll be doing in 2012

(Warning some Ubuntu ranting ahead)

1. apt-get install virtualbox-ose
2. apt-get install rubygems
3. gem install vagrant

That's what I assumed it would take me to install vagrant on a spare Ubuntu (Natty) laptop.

Well it's not. after that I was greeted with some weirdness.

1. $vagrant
2. vagrant: command not found...

Yet gem list --local showed the vagrant gem installed.

1. $ruby
2. ruby: command not found

I looked twice, checked again and indeed it seems you can install rubygems on natty with no ruby installed #dazedandconfused

So unlike other distro's on Ubuntu doesn't add the rubygems binary path to it's default path
After adding that to my .bashrc things started working better.

The active reader has noticed that by now half of the Twittersphere was pointing me to the already implemented
above solution and the other half was telling me to not install rubygems using apt-get, or to use rvm for all my rubygem troubles

Apart from the point that if you need tools to like rvm to fix things that are fundamentally broken, the fact is that joe average java developer doens't want to be bothered with RubyGem hell , he just wants to do apt-get install Vagrant and get on with his real work, and that's exactly what I'd expect from Linux for human beings

I'd expect any junior guy to be able to go to vagrantup.com read the 4 commands on the main page and be up and running
Coz that's how it works on my Bleeding Edge Enterprise Development Distro, the one I usually would not advise those people (and my mother) to use.

This is a repost of my article earlier posted at Jordan Sissel's awesome SysAdvent

After years of working in Java-based environments, there are a number of things that I like to implement together with the teams I`m working with - the application doesn't matter much, whether it's plain java, Tomcat, JBoss, etc, these deployment strategies will help your ops and dev teams build more managable services.

Packaging

The first step is to have the native operating system packages as build artifacts rolling out of your continuous integration server - No .ear, .war or .jar files: I want to have rpms or debs. With things like fpm or the maven rpm plugin this should not be an extra hassle, and the advantages you get from doing this are priceless.

What advantages? Most native package systems support dependency resolution, file verification, and upgrades (or downgrades). These are things you would have to implement yourself or cobble together from multiple tools. As a bonus, your fellow sysadmins are likely already comfortable with the native package tool used on your systems, so why not do it?

Proxied, not running as root

Shaken, not stirred

Just like any other daemon, for security reasons, I prefer to run run Tomcat or JBoss as its own user, rather than as root. In most cases, however, only root can bind to ports below 1024, so you need to put a proxy in front. This is a convenient requirement because proxying (with something like Apache) can be used to terminate SSL connections, give improved logging (access logs, etc), and provides the ability to run multiple java application server instances on the same infrastructure.

Service Management

Lots of Java application servers have a semi functional shell script that allows you to start the service. Often, these services don't daemonize in a clean way, so that's why I prefer to use the Java Service wrapper from Tanuki to manage most Java based services. With a small config file, you get a clean way to stop and start java as a service and even the possibility to add some more monitoring to it.

However, there are some problems the Java Service wrapper leaves unsolved. For example, after launching the service, the wrapper can return back with a successful exit code while your service is not ready yet. The application server might be ready, but your applications themselves are still starting up. If you are monitoring these applications (e.g for High Availability), you really only want to treat them as 'active' when the application is ready, so you don't want your wrapper script to return, "OK," before the application has been deployed and ready. Otherwise, you end up with false positives or nodes that failover before the application has ever started. It's pretty easy to create a ping-pong service flapping scenario on a cluster this way.

One application per host

I prefer to deploy one application per host even though you can easily deploy multiple applications within a single Java VM. With one-per-host, management becomes much easier. Given the availability and popularity of good virtualization, the overhead of launching multiple Linux VM's for different applications is so low that there are more benefits than disadvantages.

Configuration

What about configuration of the application? Where should remote API urls, database settings, and other tunables go? A good approach is to create a standard location for all your applications, like /etc/$vendor/app/, where you place the appropriate configuration files. Volatile application configuration must be outside the artifact that comes out the build (.ear , .jar, .war, .rpm). The content of these files should be managed by a configuration management tool such as puppet, chef, or cfengine. The developers should be given a basic training so they can provide the systems team with the appropriate configuration templates.

Logs

Logs are pretty important too, and very easy to neglect. There are plenty of alternative tools around to log from a Java application: Log4j, Logback, etc .. Use them and make sure that they are configured to log to syslog, then they can be collected centrally and parsed by tools much easier than if they were spread all over the filesystem.

Monitoring

You also want your application to have some ways to monitor it besides just checking if it is running - it is usually insufficient to simply check if a tcp server is listening. A nice solution is to have a simple plain text page with a list of critical services and whether they are OK or not (true/false), for example:

1. someService: true
2. otherService: false

This benefits humans as well as machines. Tools like mon, heartbeat or loadbalancers can just grep for "false" in the file. If the file contains false, it reports a failure and fails over. This page should live on a standard location for all your applications, maybe a pattern like this http://host / servicename/health.html and an example "http://10.0.129.10:8080/mrs-controller/health.html". The page should be accessible as soon as the app is deployed.

This true/false health report should not be a static HTML file; it should be a dynamically generated page. Text means that you can also use curl, wget, or any command-line tool or browser to check the status of your service.

The 'health.html' page should report honestly about health, executing any code necessary to compute 'health' before yielding a result. For example, if your app is a simple calculator, it should verify health by doing tests internally like adding up some numbers before sharing 'myCalculator:true' in the health report.

The 'health.html' page should report honestly about health, executing any code necessary to compute 'health' before yielding a result. For example, if your app is a simple calculator, then before reporting health it should put two and two together and get four.

This kind of approach could also be used to provide you with metrics you can't learn from the JVM, such as number of concurrent users or other valid application metadata for measurement and trending purposes.

Conclusion

If you can't convince your developers, then maybe more data can help: Check out Martin Jackson's (presentation on java deployments) Automated Java Deployments with RPM

With good strategies in packaging, deployment, logging, and monitoring, you are in a good position to have an easily manageable, reproducible, and scalable environment. You'll give your developers the opportunity to focus on writing the application, they can use the same setup on their local development boxes (e.g. by using vagrant) as you are using on production.

Last week I was in Boston for my 1st and their 25th Edition of the Large Infrastructure System Administration Conferences
Lisa was pretty much all I expected from it. Old Unix wizards with long hair and white beards, the usual suspects, and a mix of devops practitioners on a devops themed conference with on one side awesome and well positioned content and on the other side absolutely basic stuff.

On tuesday I had a devops bof scheduled for 2 hours.

My goal of the session was to not talk myselve, and let the audience figure out the 4 key components of devops as documented by @botchagalupe and @damonedwards being , Culture, Automation, Measurement and Sharing. I have to admit it took me a while to get them to that point .. but they figured out themselves .. the bof was standing room only , and there was a good discussion going on

On wednesday I gave my talk titled , Devops the Future is here, it's just not evenly distributed yet.

Devops, the future is here, it's just not evenly distributed yet.
View more presentations from Kris Buytaert

During my talk I realized that there was some more explanation needed for the crowd explaining Vagrant ... so I proposed a Bof on that topic too ... I used @patrickdebois 's awesome slides and hosted a small bof on Vagrant on thursday evening.

Vagrant devopsdays Mountain View 2011
View more presentations from Patrick Debois

Friday morning I was scheduled to be in a panel discussing featuring a #devops guy, a storage guy and a network guy ..
as my voice was starting to break down I wasn't really confident . however by the time the panel started I could talk normal again :)
The setup was weird.. it were basically 3 people with totally different backgrounds discussing a variety of topics. There were no rea
lly opposing views , mostly we agreed with eachother , so I`m not really sure if the audience was really entertained :)

Anyhow 2 bofs, a talk and a panel later .. I was exhausted and ready to fly back to Belgium.

Tomorrow I have another presentation together with Patrick at the BeJug .. problem is .. I`m still looking for my voice ;(

So worst case .. I`m just gonna turn on the recording that the Usenix folks made of my talk ...

Must admit .. I've given better talks ..

For further reference .. and preventing me from googling it a 3rd time

1. qrencode "http://www.krisbuytaert.be/" -o qrcode.png -s 10

or

1. cat Kris_Buytaert.vcf | qrencode -o vcf.png -s 5

If you set a SUID flag on a file, this flag will disappear when the ownership (user or group) is changed of that file!

Let me show you:

[root@raskas ~]# touch test
[root@raskas ~]# ll test
-rw-r--r-- 1 root root 0 2011-06-22 17:10 test
[root@raskas ~]# chmod u+s test
[root@raskas ~]# ll test
-rwSr--r-- 1 root root 0 2011-06-22 17:10 test
[root@raskas ~]# chown johan:johan test
[root@raskas ~]# ll test
-rw-r--r-- 1 johan johan 0 2011-06-22 17:10 test
[root@raskas ~]# chmod u+s test
[root@raskas ~]# ll test
-rwSr--r-- 1 johan johan 0 2011-06-22 17:10 test
[root@raskas ~]# chown root:root test
[root@raskas ~]# ll test
-rw-r--r-- 1 root root 0 2011-06-22 17:10 test
[root@raskas ~]#

We are running a heartbeat cluster with several resource groups. One of these groups is depending on a drbd disk which is configured as a master/slave resource.
Everything was working fine with heartbeat v3.0.1 and pacemaker v1.0.7…

Until we upgraded to heartbeat v3.0.3 and pacemaker v1.1.5.
When we tried to swap the resource group depending on the drbd disk, the resource group was stopped but the drbd master / slave resource refused to demote the current master.
Resulting in the unavailability of the resource group.

Mentioning this problem on #linux-ha they pointed me to resource-stickiness in the configuration.

In our configuration the resource-stickiness is set to INFINITY, we have configured it like that because we only want resources to swap during a failure or when manually requested, in all other cases the resource should remain on the node where it currently is. This is not the case when the resource-stickiness is not configured.

Apparently setting it to INFINITY was a bit radical, setting the resource-stickiness to 1000 is sufficient. The resources only swaps during a failure or manually triggered and the drbd master / slave resources follows the resource group using it.

Found some time to recreate a test environment for various things once and again. The ‘previous version’ used libvirt to control KVM instances with bridged networking. All of this quite well working setup was running on my home ’server’. On my day-to-day work laptop I have been using Virtualbox to quickly test some stuff from time to time. And that’s why Vagrant got my attention.
After a very quick re-install of my old and beloved linux distro Arch Linux (got it booted from the LVM root finally, thank you dracut) it was time to get going.

Installation of Vagrant is a breeze (no distro packages yet, just a gem)
Following the quick start on their site got me going in no time.
I stopped right after getting port forwards in. The next step handled packaging your configured VMs, which Vagrant builds using some base image downloaded somewhere. Getting a base image might already be a bit evil (who knows what ‘base’ means to someone else anyway) , and packaging your configured instance afterwards is just more evil
So I enabled my favourite config management tool (Vagrant had default Chef and Puppet support), added the manifests, and that’s all that’s needed. Next time you ‘vagrant up’ it’ll re-apply it all.

Right now, I’m having a look at Veewee, kind of an ‘addon’ tool to create your own ‘boxes’!