Getting Started with CFEngine

The first step toward using CFEngine is getting it installed on at least one machine so that you can start playing with it. CFEngine Community (the open source version) is available from many package management repositories, and you can also easily build it from source yourself. You can also install CFEngine Enterprise, which is free for use on up to 25 hosts. In this chapter we will go through the process of installing CFEngine on your machine, setting it up, and writing and running your first policy. Don’t worry if you do not understand at first glance what all the different pieces mean—-the idea of this chapter is to get you going. We will step back in CFEngine Basics to examine all the different CFEngine components.

I will mention one concept that you need to understand before we start. Your first CFEngine host will act as the policy hub, which is a server from where other CFEngine clients fetch their policy files. If you are just going to start playing with CFEngine, most likely you will be using it on a single host at the beginning, so the hub and the client can be on the same machine. As you grow your CFEngine installation, other machines will connect to the hub as well. Most CFEngine installations use a “star” configuration, with a single hub serving multiple machines. However, this is not a requirement—-CFEngine allows you to connect its components in any architecture you desire¹.

Remember that CFEngine exists in two versions: community edition and commercial edition. Therefore, I will describe the following options for installing CFEngine:

• Commercial edition (free for up to 25 nodes), installed from a binary package or on a pre-built virtual machine image using Vagrant.

• Community edition (free), installed from a binary package or from a package repository.

• Community edition (free), installed from source code.

You should always install the latest released version of CFEngine. Some operating systems include in their repositories older versions of CFEngine, but each new release includes new features, bug fixes and other improvements that make it worth staying up to date. All examples in this book have been tested with CFEngine 3.12 LTS, the latest Long-Term Support release at the time of writing.

The easiest way to give CFEngine a quick try is to use the CFEngine Enterprise binary packages, using the pre-made Vagrant configuration provided by CFEngine AS.

Vagrant allows us to quickly set up one or more VMs with a repeatable, consistent configuration. Using Vagrant, the CFEngine Enterprise Vagrant Environment quickly sets up a CFEngine Enterprise Hub and one client running on your own machine, both running the latest version of CFEngine Enterprise. You can find the full instructions at the CFEngine documentation page, but in short these are the steps you need to follow, applicable to Windows, Linux or macOS:

1. Download and install VirtualBox for your platform from https://www.virtualbox.org/

2. Download and install Vagrant for your platform from https://www.vagrantup.com/

3. Download and unpack the CFEngine Vagrant Environment tar file. When you unpack this file, it will create a directory named CFEngine_Enterprise_vagrant_quickstart_3.12.2 (or similar, depending on the current release).

4. Change into the directory created and request the status of the VMs using the vagrant status command:

   1 $ vagrant status
   2 Current machine states:
   3 
   4 hub                       not created (virtualbox)
   5 host001                   not created (virtualbox)
   6 
   7 This environment represents multiple VMs. The VMs are all listed
   8 above with their current state. For more information about a specific
   9 VM, run `vagrant status NAME`.
   

5. Create and start the VMs using vagrant up. This may take a few minutes and you will need to be connected to the Internet, since Vagrant will need to download some VM images the first time. Vagrant will set up the VMs, install and configure CFEngine Enterprise on them

    1 $ vagrant up
    2 Bringing machine 'hub' up with 'virtualbox' provider...
    3 Bringing machine 'host001' up with 'virtualbox' provider...
    4 ==> hub: Importing base box
    5     'centos-6.5-x86_64-cfengine_enterprise-vagrant-201501201245'...
    6 ...
    7 ==> host001: Thank you for downloading the
    8 Getting Started with CFEngine Enterprise
    9 ==> host001: Vagrant Virtualbox environment
   10 ==> host001: Please log into the mission portal:
   11 ==> host001: https://localhost:9002
   12 ==> host001: username: admin
   13 ==> host001: password: admin
   

6. After a few minutes, you will have a shiny new CFEngine Enterprise setup consisting of one hub and one client:

   1 $ vagrant status
   2 Current machine states:
   3 
   4 hub                       running (virtualbox)
   5 host001                   running (virtualbox)
   6 
   7 This environment represents multiple VMs. The VMs are all listed
   8 above with their current state. For more information about a specific
   9 VM, run `vagrant status NAME`.
   

You can explore the CFEngine Enterprise console using the URL and credentials printed. You can also login to any of the VMs using the vagrant ssh command. You can run most of our examples in the client VM, to which you can login like this:

By default the Vagrant configuration file will create only one client VM, but you can request more by setting the HOSTS environment variable:

In this way, you can experiment with a multi-node setup, all within your local machine. Be mindful when starting many VMs, to avoid overloading your machine.

CFEngine Community can also be installed from binary packages that are built for most popular Linux distributions. The easiest way to do this is by using the quick-start script provided by CFEngine AS. The script automatically downloads and installs the appropriate package according to your distribution. You can use it like this (make sure to type the whole command in a single line):

Alternatively, you can download the individual package files for different Linux distributions from https://cfengine.com/product/community/. Once you download the appropriate package, install it using the corresponding tool for your operating system (for example, rpm or dpkg).

Many Linux distributions and macOS package managers include CFEngine in their default package repositories. In these cases, it’s usually fine to install CFEngine from there. Be aware that in some cases, the packages available are for very old versions of CFEngine, so before you install it, make sure it is a recent version (preferably the latest) to have access to all the features we will discuss.

You can download the CFEngine source code in a compressed tar file from https://cfengine.com/source-code. You can also fetch the very latest code from the CFEngine git repository by issuing the following command:

CFEngine requires the following packages (with their development-related content, such as header files):

• OpenSSL (installed by default in most Linux and Unix systems)
• One of: LMDB, QDBM or Tokyo Cabinet
• PCRE, the Perl-Compatible Regular Expressions Library (http://www.pcre.org/)

In addition, the following libraries are supported for enabling optional features. If they are installed, the CFEngine configure script will automatically enable the corresponding features.

• libxml2 for the ability to edit XML files.
• libacl for the ability to manipulate POSIX ACLs.
• MySQL client library for the ability to manage MySQL databases.
• PostgreSQL client library for the ability to manage PostgreSQL databases.

Once the required packages are installed, compiling and installing CFEngine is as easy as running the following commands in the CFEngine source directory:

This will compile CFEngine and install all its binaries and support files under /var/cfengine/. The binaries will all be located in /var/cfengine/bin/, so you should add this directory to your PATH environment variable to be able to invoke the binaries conveniently.

You can run ./configure --help to get a list of all the valid options, and find the detailed, latest compilation instructions at https://github.com/cfengine/core/blob/master/INSTALL.

If you have purchased the commercial edition of CFEngine, you will get access to the binary packages of CFEngine Enterprise for all the supported operating systems, including a native Windows installer. You will also need to register your CFEngine policy server to get a license key for it.

The policy language in the commercial edition of CFEngine is a strict superset of the Community Edition, so you can start by practicing with the Community Edition, and move to the commercial edition as you gain more experience and need more advanced features, knowing that your existing policies will work just as before.

CFEngine Enterprise comes in two package files, called cfengine-nova and cfengine-nova-hub. The first one should be installed on CFEngine client machines, and the second one on the policy hub, the central host from where clients will fetch their policies, and where the CFEngine graphical console available with Enterprise is installed. The hub software must be installed on a 64-bit machine, so the hub packages are only available in 64-bit versions.

Similarly to the Community edition, the easiest way to install Enterprise is by using the existing quick-install script, which you can download as follows (type the command in a single line):

You then need to run the script with the argument hub or agent, depending on the type of host you want to install:

You can also download the packages individually from https://cfengine.com/product/cfengine-enterprise-free-25/, and install them on your hosts using their corresponding installation mechanism.

If you have a commercial license (this is, you have purchased a license to use Enterprise on more than 25 hosts), you need to install the license key license.dat that you get from CFEngine by running the following command:

After this, you can continue with the bootstrap process as described next.

After installation, there may be a few finishing steps left to perform, depending on your installation method. If you are using the CFEngine Enterprise Vagrant setup as described in Testing CFEngine Enterprise using a Vagrant VM, then all VMs are already correctly bootstrapped, and you can skip this section. Otherwise, follow these steps:

1. Run the command /var/cfengine/bin/cf-key. This will generate a private- and public-key pair for the current host.

   1 # /var/cfengine/bin/cf-key
   2 Making a key pair for cfengine, please wait, this could take a minute...
   

These keys are necessary when operating in a distributed CFEngine environment. This command also sets up under /var/cfengine/ the basic directory structure used by CFEngine. The generated keys will be stored in /var/cfengine/ppkeys/.

If the keys already exist (CFEngine-provided binary packages run this command automatically during the installation) you will see the following message:

1. CFEngine installs its binaries by default in /var/cfengine/bin/. Some binary packages may also copy them to /usr/local/sbin/ to have them in the same directory as other system utilities. You may want to add the correct directory to the PATH environment variable in your shell setup.

1. On the policy hub, CFEngine expects to find its “master files” under /var/cfengine/masterfiles/. This is meant to be the master copy of its policy files, from where they will be copied to the work directory (/var/cfengine/inputs/ by default). If the /var/cfengine/masterfiles/ directory is empty or nonexistent (this will be the case if you installed from source), you need to populate it with the sample masterfiles directory from the CFEngine distribution, which normally gets installed in /var/cfengine/share/CoreBase/:

   1 # ls /var/cfengine/masterfiles
   2 # cp -Rp /var/cfengine/share/CoreBase/* /var/cfengine/masterfiles/
   3 # ls -F /var/cfengine/masterfiles/
   4 cf-sketch-runfile.cf  controls/  def.cf     libraries/
   5 promises.cf           services/  update.cf
   

   We will examine these files in detail later on.

1. Finally, CFEngine needs to be “bootstrapped” to a CFEngine policy server. This registers the client with the server, copies the masterfiles to their final working location in /var/cfengine/inputs/ and starts the base cf-execd daemon. The policy server also needs to be bootstrapped to itself.

The cf-execd daemon controls the periodic execution of cf-agent, which is the one that actually executes the promises in the provided policies (we will look in more detail at the different components in CFEngine Components).

To bootstrap a client, first find the IP address of your policy server, using the ifconfig command (ipconfig under Windows). Let’s assume it is 10.0.2.15. Run the cf-agent command with the --bootstrap option. For example, if you run the command on the policy server itself, you will see the following:

The cf-agent command recognizes you are using the machine’s own IP address to bootstrap, and configures it as a policy server. You can verify the success of this command by looking at the process list. You should see at least the cf-execd process, and maybe some others that are started automatically by it:

The policy server should be the first host you bootstrap. Once you have it running, you can boostrap CFEngine on other hosts by providing the server’s IP address to the cf-agent --bootstrap command.

The CFEngine distribution includes not only the binaries, but also a large library of documentation and examples. The examples normally get installed in /var/cfengine/share/doc/ (in previous versions they were installed under /usr/local/share/cfengine/ or /usr/local/share/doc/cfengine/, and can be of big help for getting started. These directories include examples of CFEngine configurations for different tasks and demonstrate the use of different CFEngine constructs. The examples directory contains a large number of mostly-self-contained files that demonstrate and exercise different CFEngine abilities.

Now that you have CFEngine installed and running, let us start by writing a first simple policy. If you have finished the bootstrapping process described in Finishing the Installation and Bootstrapping, you can be sure that CFEngine is properly installed. You can also check this by running the following command:

For our first policy, let us tackle a task that is simple to explain, yet can be useful in real systems. We will add a line to the /etc/motd file to indicate that CFEngine is running on this machine. And to keep with tradition, we will also print out a “Hello world!” message to the console when the policy is run.

I recommend you type the code as you read through the example, and save it in a file named edit_motd.cf so you can run it from CFEngine when we are done.

CFEngine instructions (called “promises”) are contained in units called “bundles”. In our case, we will define a bundle called edit_motd. Here is the bundle code:

This is the part of the policy that tells CFEngine what to do. Here is how it works:

• In CFEngine, a bundle of type agent (identified by its declaration bundle agent, followed by an arbitrary identifier, in this case edit_motd) could be considered the equivalent of a subroutine, and contains promises that CFEngine evaluates and acts on, if needed. It is split into sections that correspond to different types of promises, which are the lines that start with a word and end with a single colon. In this bundle, we have three sections: vars:, files:, and reports:.

• The vars: section is used to declare variables. CFEngine has several variable types, including strings, lists, arrays, and numbers (both integers and floating-point numbers are supported). Here we are declaring a single string value named motd, which contains the path of the file we want to edit. If you are testing this on a system where you don’t have root privilege, you should change this path to some file you can edit, for example /tmp/motd.

  In a CFEngine policy, everything is expressed as promises, even variable declarations. In this case, motd promises to be a string variable containing the value "/etc/motd"​. We will reference this variable later in the policy. Scalar variable references are indicated by a dollar sign followed by the variable name enclosed in either parentheses or braces. Both ${motd} and $(motd) refer to the same variable.

• In the files: section we indicate the file-related operations we want to perform. In this case, the promiser is "$(motd)" which expands the motd variable into its value, so the promiser becomes "/etc/motd", telling CFEngine which file to edit.

The rest of the promise, up until the semicolon, is called the body of the promise, and is formed by attribute =​> value pairs, separated by commas. In this case we have two attribute specifications: create =​> "true" and edit_line =​> addmessage. The former simply indicates that the file needs to be created if it doesn’t exist yet. The latter means that lines in /etc/motd will be edited according to the specification given by a bundle named addmessage (which we haven’t seen yet).

Casting this into CFEngine terminology: All promisers in the files: section are interpreted by CFEngine as files (or directories) on the system, so the promise in our sample policy means that the /etc/motd file promises to be edited according to the instructions given by the body of the promise. The value of the edit_line parameter is the name of an edit_line bundle. This means that it’s not a single value, but rather the name of a named collection of attributes that specifies the behavior of edit_line. Here is its definition:

A bundle is also a container of promises, and is also divided in sections. The type of each bundle is given by the second word in its declaration (in this case, edit_line). You can see that the edit_motd bundle had agent as its type, which means it is an “execution” bundle that can be called directly (in this case, from the bundlesequence declaration, although there are other means for executing agent bundles that we will cover later). Thus, the first line assigns the type edit_line to the addmessage bundle, meaning that it can be used only as the value of an edit_line attribute. Additionally, the type of a bundle defines what sections are valid in it, and how the promises in it are interpreted. An edit_line bundle must contain promises that perform edits on a file. In this case, it contains an insert_lines: section, so promises are interpreted as lines that must be present in the file. The only promise in this bundle is a string that contains the message we want to insert in the file. The string itself is the promiser and no additional attributes are given, which means the line will always be inserted into the file, unless it is there already (this is CFEngine’s way of ensuring convergent behavior: if it always inserted a line, the file would never converge to a stable state).

In summary, what this means is that the given line will be inserted into /etc/motd if it’s not there already.

• Finally, the edit_motd bundle has a reports: section, which is meant to produce output during the execution of the policy. Promises in a reports: section indicate messages and how they will be handled. By default, the promised message will be printed to the console. In our case, we will print the message Hello world! to the console every single time the promise is executed.

This completes our first policy, so let us look at it in one piece:

Type this in and save it to a file, for example edit_motd.cf. You can then execute it with the following command:

These are the options we used:

• --bundlesequence (short -b) tells CFEngine which bundle to execute within the file (the bundle sequence can also be specified within the policy file, but specifying it in the command line will make it easier to integrate with the rest of the system policy).

• --no-lock (can be abbreviated as -K) means “Ignore locking constraints during execution,” which in practice means “always execute all promises.” Normally, CFEngine obeys certain time periods between successive evaluations of the same promise, to avoid overloading the systems. The --no-lock option disables those constraints, and so is useful for testing policies that you may run several times in quick succession.

• --inform (short -I) means “Print basic information about changes made to the system,” essentially telling CFEngine to show you the actions that it is taking. If not specified, CFEngine’s output is quite terse, so again, this is useful when you are testing policies to get more information about what CFEngine is doing.

• --file (short -f) tells CFEngine to use the specified file as its input. Otherwise it will try to read /var/cfengine/inputs/promises.cf You can also omit the -f and simply give the filename as the last argument to the command.

Now examine the /etc/motd file, and you will see that the following line has been added to it:

If you run the command again, the output changes slightly:

Note that the “Edit file…” message is missing. The /etc/motd file already contains the message, so it is not edited again. Now try editing it by hand and removing or modifying the existing line. If you run cf-agent again, the message will reappear.

Congratulations! You have written and executed your first CFEngine policy. This is a very basic operation, but its structure is very similar to that of any other CFEngine policy, and allows enough flexibility and expressibility to tackle the most complex configuration tasks.

Most of the examples in this book are shown running as the root user, since that is the normal conditions under which CFEngine should be executed to have the privileges necessary to exercise changes to the system. However, during development and testing it is perfectly possible to run CFEngine as a regular user. When you run it like this, CFEngine does not look under /var/cfengine/ for its input files, rather it looks under $HOME/.cfagent, so if you run cf-agent without specifying an input file, it will try to read $HOME/.cfagent/inputs/promises.cf.

For this to work, all the CFEngine executables and files (including its key files) need also to exist under $HOME/.cfagent/. So you need to run the following commands before you start:

In the example we just saw, you were running the policy file by hand using cf-agent. But CFEngine is meant to save you from running things by hand! For development and testing it is fine to run your policies this way, but introducing bundle agent __main__ can improve this workflow and make your policies capable of being executed directly as scripts without having to manually specify the bundlesequence and still allow them to be integrated into a larger policy set without alteration.

CFEngine supports two special bundle names as default entry point:

• If a bundle agent __main__ exists, it will be called by default for that file. Each file can have its own __main__ bundle without conflict, and it will only be called for the file which was explicitly executed.
• If a bundle agent main exists, it will be called by default for the whole set of policies loaded. There can be only one main bundle overall. If more than one loaded file defines the main bundle, an error will be produced.

In the example we just saw, you were running the policy file by hand using cf-agent. But CFEngine is meant to save you from running things by hand! For development and testing it is fine to run your policies this way, but once they are done, you need to integrate them into the main CFEngine execution loop so that they are evaluated continuously and automatically. This is done by integrating them into the policies executed by promises.cf, which is the file that CFEngine loads and executes by default.

The easiest way to do this integration is to make use of the built-in capabilities in the default CFEngine masterfiles policies to automatically load and execute certain bundles. In short, this is what we need to do:

• Enable the “autorun” feature in the policy (it comes disabled by default).
• Tag our edit_motd bundle with the autorun tag so that it is automatically executed.
• Copy our edit_motd.cf policy file to the services/autorun directory inside the policy directory, so that it is automatically loaded.

More specifically, these are the steps we need to follow to automatically load and execute the edit_motd bundle:

1. Edit the edit_motd bundle to add the following lines at the beginning, which specify some metadata for the bundle, including the autorun tag:

   1 meta:
   2   "tags" slist => { "autorun" };
   

2. Copy the modified edit_motd.cf to /var/cfengine/masterfiles/services/autorun/ in the policy server. This directory is special because all policy files in it will be automatically loaded.

1. Create file /var/cfengine/masterfiles/def.json with the following contents, to enable the autorun feature:

   1 {
   2   "classes":
   3   {
   4     "services_autorun": [ "any" ]
   5   },
   6 }
   

If you want to trigger the changes right away, run the following commands to force an immediate update of the policies, and then to execute them:

(you may want to remove the reports: promise that prints the message once you have this integrated)

That’s it! Now your edit_motd bundle will be run automatically as part of the regular CFEngine execution every five minutes, ensuring that the /etc/motd file is constantly kept correctly configured.