Ansible – Backup Files Before Change

Assuming you’ve used ansible for creating your configuration file then you would not really need to back it up. That is because you should be using version control on your ansible playbooks and included variable files already so you should have a copy of all the configuration changes at various stages. But if you are talking about system configuration files many of these are part of the OS installation and were likely never touched by Ansible until a need arose.

I ran across this as the security team was using CIS Benchmarking to harden our systems. The benchmark would recommend steps to take to close potential vulnerabilities and we would create ansible playbooks to put these configurations in place. Some members of the DevOps team created playbooks that would create time stamped backup copies of the configuration files. This was helpful for the day to day admins that would receive calls when the benchmarks caused something to stop working sometimes days later. The system admin could see what had changed.

However, as more benchmarks were deployed the number of files became too much to handle. Also, if a /etc/sysconfig file were changed any copies with merely a timestamp in the same directory would still be processed by the system services and the configuration that ended up being used was the luck of the draw. That is when I went back to something I had done for years before DevOps and Ansible were a thing. On the systems I administered I would always install RCS, revision control system. I tried to use others like subversion and cvs but they would modify the file ownership and permissions which was a big problem for system configuration files. Before making manual changes on system configuration files, I would check them in and back out of RCS. The Ansible playbook below shows an example of using RCS to create a backup of a file prior to applying a configuration change to the multipath.conf file. This same process may be used with any file to store a version locally.

  pre_tasks:
    - name: Install RCS, device-mapper and device-mapper-multipath packages
      ansible.builtin.package:
        name:
          - device-mapper
          - device-mapper-multipath
          - rcs
        state: present

    - name: Check if /etc/multipath.conf exists
      ansible.builtin.stat:
        path: /etc/multipath.conf
      register: multipath_conf_stat

    - name: Create revision control backup directory
      ansible.builtin.file:
        path: /etc/RCS
        state: directory 
        mode: 0775

    - name: Check existing multipath.conf file into revision control
      ansible.builtin.shell: ci -t-"initial version" -i -l /etc/multipath.conf || ci -m"updated defaults" -l /etc/multipath.conf
      when: multipath_conf_stat.stat.exists


Finally, another reason to store a version locally is if the change happens to knock the system off the network then you will have to work through the console. The system may not have access to a remote version control or offline backup of the modified configuration files. Having the local RCS will allow for faster troubleshooting and recovery.

Linux Command History Logging

If you want to track what users are doing there is of course the command last which shows who has logged in and for how long. Then there is the auditd service which logs transactions in the /var/log/audit/audit.log file. This tracks some of the commands executed on systems and some arguments. Further the audit logs use sets of parameter=value syntax and some values are stored as hex making it not as user friendly.

It is not just about tracking what users are doing incorrectly but also being able to reproduce something which effectively worked on a system on another system. You may need to put the commands or arguments into a script or work them into a DevOps tool like Ansible.

To configure a linux systems so that users shells record command history add the following lines to the /etc/profile will create a .history directory under each users home directory and the HISTFILE will be created under this using the name of the real user not the su or sudo user.

## setup history
export REALUSER=$(/usr/bin/who -m | awk '{print $1}')
export EFCTUSER=$(/bin/whoami)
shopt -s histappend
[[ ! -d ${HOME}/.history ]] && mkdir -p ${HOME}/.history
export HISTTIMEFORMAT="%Y/%m/%d %T "
eval export HISTFILE=${HOME}/.history/.${REALUSER:-$USER}
export HISTSIZE=6000
if [ "${HISTCONTROL:-}" = "ignorespace" ] ; then
    export HISTCONTROL=ignoreboth
else
    export HISTCONTROL=ignoredups
fi
export PROMPT_COMMAND="history -a ; ${PROMPT_COMMAND}"
readonly HISTFILE HISTSIZE

The HISTTIMEFORMAT will date and time stamp each action taken. The ignoredups will only record the last instance of a command run multiple time to save space. The prompt command prepending history -a will force history to be stored after each command is run instead of after logout to ensure actions taken are recorded. If a user looses their connection the session may not record the command history.

Alternatively you may want to store histories under /var/spool/history/{effective user}/{real user}

Design a site like this with WordPress.com
Get started