DevOps – Missing Version Control

For many DevOps implementation from the perspective of system deployment and configuration management tools such as Ansible, Chef, SaltStack and Puppet, just to name a few , are the staple for meeting DevOps requirements. Version control is used for the playbooks and recipes and of course any source code for components and applications that are compiled and deployed to systems. When I started my first job out of college I worked with a contractor named LeRoy Budnick who had done work on the development of the original IT Service Management and ITIL standards.

LeRoy introduced me to the idea of an expanded role for version control using RCS to manage versions of configuration files. If changes caused issues on systems they could quickly be backed out by checking out the previous version of configuration file. On production systems version control could be monitored to see if unapproved changes were made to a system and could easily be backed out. It was a great idea and I have used it ever since instead of just making backup copies of configuration files. In fact there are times where making backup copies of configuration files can cause problems. In RedHat’s /etc/sysconfig boot configuration directory if you make a backup copy just adding a suffix to a copy you will find that on boot the system will use not only the configuration files but these backups as well. The configuration that ends up running after boot will be dependent upon which config file was last and how the variables overlayed one another. In short you have no idea what your configuration is after boot. Using RCS the versions are placed in the RCS subdirectory and do not end up being executed since they are no longer in the sysconfig or sysconfig/networks-scripts directories.

Enter the world of DevOps and tools for deploying Operating Systems, compiling applications on the fly, installing them on systems and updating configurations across your data centers. One might think that the need for such version control has been deprecated by tools such as Ansible and Chef or at the very least we only need to use version control for the playbooks and recipies. But not so fast. My security team started using Ansible to roll out CIS Benchmark system hardening configurations and occassionally this would hang systems and make them inaccessible. If the system is inaccessible then the DevOps tools cannot access them to back out the changes and an administrator has to get involved. Then figuring out how far along the change was implemented and what changed and then backing those changes out was quite a chore. They then started having the Ansible playbooks make backup copies of the configuration files before they implemented changes. This made it a little easier but then enter /etc/sysconfig. If you recall we mentioned earlier that this doesn’t work; making date stamped backup copies in /etc/sysconfig. The security team ended up creating Frankenstien configs with many versions of config files all of which were being executed. Further not all configuration files are managed by configuration management tools; for example network TCP/IP configurations are specific to each system and sometimes require system specific tuning. At this point it should be clear that there is a good case for using version control on configuration files. In fact I wish devops tools like Chef or Ansible would build it into their implementations.

This prompted me to write the previous article about how to use RCS with Ansible to make backups of configurations before deploying changes to configurations. In the past I tried to use more modern version control like CVS or SubVersion but found that checking in and out the configuration files resulted in the configuration files being recreated with different permissions. RCS doesn’t recreate files but just over-writes them retaining their permissions and ACLs. However now RCS is being deprecated and is no longer available for install on some Linux BaseOS deployment images. It may be available on some supplimentary images but it is clear that RCS is going away. I have been working on transitioning to Git and have come up with some procedures for using git with hooks to backup and restore ACLs and permissions. I will go over how to use git for configuration file version control in the next Article.