EzDevInfo.com

How can I pre-sign puppet certificates?

Puppet requires certificates between the client (puppet) being managed and the server (puppetmaster). You can run manually on the client and then go onto the server to sign the certificate, but how do you automate this process for clusters / cloud machines?

How can the little guys effectively learn and use Puppet?

Six months ago, in our not-for-profit project we decided to start migrating our system management to a Puppet-controlled environment because we are expecting our number of servers to grow substantially between now and a year from now.

Since the decision has been made our IT guys have become a bit too annoyed a bit too often. Their biggest objections are:

• "We're not programmers, we're sysadmins";
• Modules are available online but many differ from one another; wheels are being reinvented too often, how do you decide which one fits the bill;
• Code in our repo is not transparent enough, to find how something works they have to recurse through manifests and modules they might have even written themselves a while ago;
• One new daemon requires writing a new module, conventions have to be similar to other modules, a difficult process;
• "Let's just run it and see how it works"
• Tons of hardly known 'extensions' in community modules: 'trocla', 'augeas', 'hiera'... how can our sysadmins keep track?

I can see why a large organisation would dispatch their sysadmins to Puppet courses to become Puppet masters. But how would smaller players get to learn Puppet to a professional level if they do not go to courses and basically learn it via their browser and editor?

Puppet Security and Network Topologies

Background:

I am finally setting aside some time to join the 21st Century and look at Puppet.

As it stands today we version control all server configurations in a repository that is held internally at the office. When an update needs making, the changes are checked back into the repos and manually pushed out to the machine in question. This usually means SFTP'ing to the remote machine and then moving files into place, with the relevant permissions, from a shell.

So I am hopeful that Puppet is going to be an simple yet amazing extension to what we already have.

Now I consider the process that we currently have to be reasonably secure. On the assumption that our internal network will always be relatively more secure than the public networks in our datacentres.

• The process is always one way. Changes traverse from a secure environment to insecure and never the other way round.

• The master store is in the safest possible place. The risk of compromise, either by stealing configurations or sending out malicious modifications, is greatly reduced.

Question:

From what I understand of the Puppet server/client model is that the clients poll and pull updates down directly from the server. The traffic is SSL wrapped so cannot be intercepted or spoofed. But it differs from what we currently do because the Puppet server[s] would need to be hosted in a public location. Either centrally, or one for each datacentre site that we maintain.

So I am wondering:

• Am I being unnecessarily paranoid about the change from push to pull?

• Am I being unnecessarily paranoid about centrally storing all of that information on a public network?

• How are others maintaining multiple networks - separate server for each site?

Update 30/07/09:

I guess that one of my other big concerns is placing so must trust in a single machine. The puppetmaster(s) would be firewalled, secured and such. But even so any public machine with listening services has an attack surface of a certain size.

Presumably if the master has permission to update any file on any one of the puppet clients, then it's compromise would ultimately result in the compromise of all it's clients. The "kings to the kingdom" so to speak.

• Is that hypothesis correct?

• Is there any way that it can be mitigated?

Puppet: Node name seems dependent on reverse dns?

I seem to be running into a little bit of a problem understanding how to get this to work. I have a new server I'm building sitting behind the office NAT at work, its reverse dns maps to office.mydomain.com, but I want the machine to be ns2.mydomain.com for the sake of puppet.

nodes.pp snippet:

node 'ns2.mydomain.com' inherits basenode {
  info('ns2.mydomain.com')
}

node 'office.mydomain.com' inherits basenode {
  info('office.mydomain.com')
}

And my 'puppet.conf' on the client:

[main]
#was node_name=ns2.mydomain.com
#was fqdn=ns2.mydomain.com
certname=ns2.mydomain.com
node_name=cert

My syslog on the server reports:

Sep 16 22:59:12 support puppetmasterd[2800]: Host is missing hostname and/or domain: office.mydomain.com
Sep 16 22:59:12 support puppetmasterd[2800]: (Scope(Node[office.mydomain.com])) office.mydomain.com
Sep 16 22:59:12 support puppetmasterd[2800]: Compiled catalog for office.mydomain.com in 0.03 seconds
Sep 16 22:59:12 support puppetmasterd[2800]: Caching catalog for ns2.mydomain.com

How can I make it grab the config for ns2.mydomain.com without doing something like this:

node 'ns2.mydomain.com' inherits basenode {
  info('ns2.mydomain.com')
}

node 'office.mydomain.com' inherits 'ns2.mydomain.com' {
  info('office.mydomain.com')
}

UPDATE: This problem seems to be causing other issues as well. For instance if I info("$fqdn") while the machine is sitting behind office.mydomain.com the fqdn fact is empty, as well as the $operatingsystem. Its almost like the facts aren't being discovered properly. Is there perhaps a NAT issue? Are there any suggestions for tracking down this cause of this problem?

How to update a package using puppet and a .deb file

I am trying to figure out the proper way to update/upgrade a deb package using puppet from a local source deb file. My current config looks like this...

class adobe-air-2-0-4 {

  file { "/opt/air-debs":
    ensure => directory
  }

  file { "/opt/air-debs/adobeair-2.0.4.deb":
    owner   => root,
    group   => root,
    mode    => 644,
    ensure  => present,
    source  => "puppet://puppet/adobe-air-2-0-4/adobeair-2.0.4.deb"
  }

  package { "adobeair":
    provider => dpkg,
    ensure => installed,
    source => "/opt/air-debs/adobeair-2.0.4.deb"
  }

}

I first copy the deb file down to the client machine and then use 'package' with the provider set to 'dpkg'. This works and I get the correct version installed.

My question is what is the proper way to update this package in the future. Can I simply change out the source file and puppet will know that it's a different version and update this package? How does puppet determine what version of a package it has installed versus the version of the source deb file?

I am pretty new to puppet, so if you have an suggestions for improvements to my existing config they are very much appreciated.

Are configuration management tools (Puppet, Chef) capable of keeping installed packages up to date?

This is probably a simple question for those of you already running configuration management tools. Are configuration management tools such as Puppet or Chef the right approach for keeping installed packages up to date?

Suppose I run a number of servers, mostly based on Debian and Ubuntu. Do configuration management tools make it easier to update packages installed from the repositories when security updates or bug fixes come along?

I currently run "unattended upgrades" to let the systems automatically install security updates, but I still have to connect to the servers and run aptitude update && aptitude safe-upgrade every so often. Naturally this gets boring, tedious and error-prone the more servers there are.

Are tools such as Puppet or Chef the right approach to keeping installed packages up to date? Do any of you use these tools to avoid manually running aptitude or an equivalent on 15 servers? I am quite certain the answer to these questions is "Yes, of course!"

But where can I find more information about this particular use case? I have not yet had the time to study Puppet or Chef in-depth, and the example cookbooks or classes only show more or less trivial examples of installing one particular package, such as ssh. Do you have any resources to recommend, other than the official documentation (I am, of course, going to study the docs once I know which, if any, of the tools are right for me).

How can a Linux Administrator improve their shell scripting and automation skills?

In my organization, I work with a group of NOC staff, budding junior engineers and a handful of senior engineers; all with a focus on Linux. One interesting step in the way the company grows talent is that there's a path from the NOC to the senior engineering ranks. Viewing the talent pool as a relative newcomer, I see that there's a split in the skill sets that tends to grow over time...

• There are engineers who know one or several particular technologies well and are constantly immersed... e.g. MySQL, firewalls, SAN storage, load balancers...
• There are others who are generalists and can navigate multiple technologies.
• All learn enough Linux (commands, processes) to do what they need and use on a daily basis.

A differentiating factor between some of the staff is how well they embrace scripting, automation and configuration management methodologies. For instance, we have two engineers who do the bulk of Amazon AWS CloudFormation work, and another who handles most of the Puppet infrastructure. Perhaps a quarter of the engineers are adept at BASH shell scripting.

Looking at this in the context of the incredibly high demand for DevOps skills in the job market, I'm curious how other organizations foster the development of these skills and grow their internal talent. Scripting doesn't seem like a particularly-teachable concept.

• How does a sysadmin improve their shell scripting?
• Is there still a place for engineers who do not/cannot keep up in the DevOps paradigm?
• Are we simply to assume that some people will be left behind as these technologies evolve? Is that okay?

Adding a yum repo to puppet before doing anything else

Is there a way to force puppet to do certain things first? For instance, I need it to install an RPM on all servers to add a yum repository (IUS Community) before I install any of the packages.

Do chef and puppet cost money?

I intend to use chef or puppet to do administration (I'm thinking more of chef as it's younger and I get a better feeling about it).

In both home pages I saw there is an "enterprise edition" that costs money and I don't intend to buy anything. What would I miss in chef / puppet if I don't buy them?

What does chef offer that costs money exactly?
What does puppet offer that costs money exactly?

It was not so clear to me from their web site, as it's kind of obscure.

What advantages/features does Puppet or Chef offer over Salt (or vice versa)? [closed]

I am looking at rolling out a new configuration management tool to replace our home-grown solution. The defacto standards are Chef and Puppet, both of which are Ruby-centric (though can be used to deploy non-Ruby environment, obviously). The vast majority of our development is done in Python and our in-house deployment tools make heavy use of Fabric. Therefore I am learning towards Salt since it too is Python, even though it is not as mature as Chef or Puppet. But since I'm not familiar enough with the options, I'm finding it difficult to compare apples-to-apples.

Other than the smaller community, would I be giving up anything signifcant by using Salt rather than Puppet/Chef?

Update

It's been six months since I posted this question. And despite it being closed, it's been viewed over 1,000 times so I thought I'd comment on my experiences.

I eventually decided on Puppet since it had a bigger community. However, it was an immensely frustrating experience, mainly due to the convoluted Puppet configuration syntax. Since I now had a frame of reference to compare the two, I recently took another look at Salt--I'm not going back. It is very, very cool. The things I like best:

• Seamless integration of both push and pull configuration models. Puppet uses a pull model (node periodically polls server for updates) and has a sister component called Marionette for pushing changes. Both are important to me and I prefer how Salt works. Salt also executes much faster when you have a lot of nodes.

• Configuration syntax uses YAML, which is just a simple text format that uses indentation and bullet points. You can also choose to use other configuration formats via template. This makes Salt about 10x easier to learn and maintain, in my experience.

• Python-based. This was the biggest reason I started looking at Salt in the first place. It ended up being one of the more minor reasons I stayed. But if you're a Python shop like us, it makes it easier to develop Salt plugins.

automate dpkg-reconfigure tzdata

I'm using puppet to admin a cluster of debian servers. I need to change the timezone of each machine on the cluster. The proper debian way to do this is to use dpkg-reconfigure tzdata. But I can only seem to change it if I use the dialog. Is there some way to automate this from the shell so I can just write an Exec to make this easy?

If not, I think the next best way would probably be to have puppet distribute /etc/timezone and /etc/localtime with the correct data across the cluster.

Any input appreciated!

Puppet vs Chef, pro and contra from users and use cases [closed]

I already googled and read the "to-puppet-or-to-chef-that-is-the-question" article.

I'm interested in use cases, real world implementations in which people had choosen one or the other on real problems bases.

I'm particularly interested in integration with cobbler issues ( I know puppet is much a standard approach in this direction ); as anybody any experience in cobbler-chef integration ?

Thanks in advance

What should NOT be managed by puppet?

I'm learning my way through configuration management in general and using puppet to implement it in particular, and I'm wondering what aspects of a system, if any, should not be managed with puppet?

As an example we usually take for granted that hostnames are already set up before lending the system to puppet's management. Basic IP connectivity, at least on the network used to reach the puppetmaster, has to be working. Using puppet to automatically create dns zone files is tempting, but DNS reverse pointers ought to be already in place before starting up the thing or certificates are going to be funny.

So should I leave out IP configuration from puppet? Or should I set it up prior to starting puppet for the first time but manage ip addresses with puppet nonetheless? What about systems with multiple IPs (eg. for WAN, LAN and SAN)?

What about IPMI? You can configure most, if not all, of it with ipmitool, saving you from getting console access (physical, serial-over-lan, remote KVM, whatever) so it could be automated with puppet. But re-checking its state at every puppet agent run doesn't sound cool to me, and basic lights out access to the system is something I'd like to have before doing anything else.

Another whole story is about installing updates. I'm not going in this specific point, there are already many questions on SF and many different philosophies between different sysadmins. Myself, I decided to not let puppet update things (eg. only ensure => installed) and do updates manually as we are already used to, leaving the automation of this task to a later day when we are more confident with puppet (eg. by adding MCollective to the mix).

Those were just a couple of examples I got right now on my mind. Is there any aspect of the system that should be left out of reach from puppet? Or, said another way, where is the line between what should be set up at provisioning time and "statically" configured in the system, and what is handled through centralized configuration management?

Why is it so difficult to upgrade between major versions of Red Hat and CentOS?

"Can we upgrade our existing production EL5 servers to EL6?"

A simple-sounding request from two customers with completely different environments prompted my usual best-practices answer of "yes, but it will require a coordinated rebuild of all of your systems"...

Both clients feel that a complete rebuild of their systems is an unacceptable option for downtime and resource reasons... When asked why it was necessary to fully reinstall the systems, I didn't have a good answer beyond, "that's the way it is..."

I'm not trying to elicit responses about configuration management ("Puppetize everything" doesn't always apply) or how the clients should have planned better. This is a real-world example of environments that have grown and thrived in a production capacity, but don't see a clean path to move to the next version of their OS.

Environment A:
Non-profit organization with 40 x Red Hat Enterprise Linux 5.4 and 5.5 web, database servers and mail servers, running a Java web application stack, software load balancers and Postgres databases. All systems are virtualized on two VMWare vSphere clusters in different locations, each with HA, DRS, etc.

Environment B:
High-frequency financial trading firm with 200 x CentOS 5.x systems in multiple co-location facilities running production trading operations, supporting in-house development and back-office functions. The trading servers are running on bare-metal commodity server hardware. They have numerous sysctl.conf, rtctl, interrupt binding and driver tweaks in place to lower messaging latency. Some have custom and/or realtime kernels. The developer workstations are also running a similar version(s) of CentOS.

In both cases, the environments are running well as-is. The desire to upgrade comes from a need for a newer application or feature available in EL6.

• For the non-profit firm, it's tied to Apache, the kernel and some things that will make the developers happy.
• In the trading firm, it's about some enhancements in the kernel, networking stack and GLIBC, which will make the developers happy.

Both are things that can't be easily packaged or updated without drastically altering the operating system.

As a systems engineer, I appreciate that Red Hat recommends full rebuilds when moving between major version releases. A clean start forces you to refactor and pay attention to configs along the way.

Being sensitive to business needs of clients, I wonder why this needs to be such an onerous task. The RPM packaging system is more than capable of handling in-place upgrades, but it's the little details that get you: /boot requiring more space, new default filesystems, RPM possibly breaking mid-upgrade, deprecated and defunct packages...

What's the answer here? Other distributions (.deb-based, Arch and Gentoo) seem to have this ability or a better path. Let's say we find the downtime to accomplish this task the right way:

• What should these clients do to avoid the same problem when EL7 is released and stabilizes?
• Or is this a case where people need to resign themselves to full rebuilds every few years?
• This seems to have gotten worse as Enterprise Linux has evolved... Or am I just imagining that?
• Has this dissuaded anyone from using Red Hat and derivative operating systems?

I suppose there's the configuration management angle, but most Puppet installations I see do not translate well into environments with highly-customized application servers (Environment B could have a single server whose ifconfig output looks like this). I'd be interesting in hearing suggestions on how configuration management can be used to help organizations get across the RHEL major version bump, though.

Why use Chef/Puppet over shell scripts?

New to Puppet and Chef tools. Seems like the job that they are doing can be done with shell scripting. Maybe it was done in shell scripts until these came along.

I would agree they are more readable. But, are there any other advantages over shell scripts besides just being readable?