SSH Agent Automation

On Linux systems many of us administrators and engineers have our favorite profiles and configuration file settings. One of the most used tools and a must for securing an environment is secure shell or ssh. Secure shell uses asymmetric encryption which is a public key and private key pair of keys; one used for encryption and the other for decryption. Open SSH allows for several different algorithms such as DES or RSA. The public encryption key may then be shared to other systems in the ~/.ssh/authorized_keys file indicating that a system having the correct key information may be allowed to ssh directly into a system using only the public key challenge. Further the public and private key pair may be associated with a passphrase requiring such to be entered before the asymmetric key pair may be used for authentication.

Many DevOps Infrastructure as Code tools and other management tools and even home grown scripts may use ssh to manage through inquiry and remote execution multiple systems in an environment. The ssh passphrase requirement may get in the way of such automation and cause such batch processes to fail. The ssh-agent was created to resolve this limitation by registering passphrases and keys so that subsequent ssh sessions would not be prompted for passphrases. The script below may be added to a .bashrc or .kshrc user profile to instantiate a ssh-agent which may be used by subsequent session. It createa a link to the ssh-agent special file as ~/.ssh/ssh_auth_sock and updates the SSH_AUTH_SOCK environment variable to point to this link. This then allows sessions going forward to piggyback off the initial ssh-agent instantiation. This may also be used with scheduled jobs.

## Check if the agent is accessible and if not remove socket file and kill agents
export SSH_AUTH_SOCK=~/.ssh/ssh_auth_sock
ssh-add -l >/dev/null 2>&1 ; RT=$?
if [ -h ~/.ssh/ssh_auth_sock -a ${RT} -gt 0 ]; then 
	echo "SSH Agent is dead ${RT}; removing socket link file and killing hung ssh agent!"
	rm -f ~/.ssh/ssh_auth_sock 
	pkill -u $(whoami) -i ssh-agent 
## if the auth socket does not exist start the agent and recreate the auth socket link
if [ ! -h ~/.ssh/ssh_auth_sock ]; then
	echo "Ssh agent socket link does not exist; starting new agent!"
	eval `ssh-agent`
	ln -sf "$SSH_AUTH_SOCK" ~/.ssh/ssh_auth_sock
export SSH_AUTH_SOCK=~/.ssh/ssh_auth_sock
ssh-add -l > /dev/null 2>&1 || ssh-add

Virtual Instance Discovery and Analysis

As an Infrastructure Engineer or Architect you need to have a good grasp on what systems comprise your environment. In the past this was somewhat straight forward. You kept a configuration item database in your CMDB and teams had their workbooks and playbooks. However in this new world of DevOps and CI-CD and their automation tool sets such as Terraform, Chef, Ansible, Bladelogic, and many others; developers can stand up their own virtual instances and tare them down. This can make it hard to have a complete picture of your environment. This can be especially difficult for storage infrastructure since many virtual instances can be deployed on large data stores and when there is an IO performance problem tracking down the related hardware can be like following the rabbit to wonderland.

I ran into this while leading several projects for storage infrastructure servicing an ESX environment and developed a powercli script to pull the necessary virtual instance and data-store data from the VSphere systems. First you will need to install powercli for VMware which can be found here:

Read more of this post

Optimizing Disk IO Through Abstraction

To Engineer or Not To…

When disk capacity is released to a new application or service many times the projects do not consider how best to use the storage that has been provided. Essentially the approaches fall into one of two schools of thought. The first is to reduce upfront engineering into a couple design options and resolve issues when they arise. The second is to engineer several solution sets with variable parameters that will provide a broader pallet of solutions and policies from which an appropriate solution may be selected.

Reduced Simplified Engineering

  • Apply one of a couple infrastructure designs to a project.
  • This approach involves less work upfront, has a simpler execution and involves less work gathering requirements.
  • Potentially more time and effort will be spent resolving issues when resources and design are insufficient.
Read more of this post

Pushing Your Profile and SSH Keys

When ever you start supporting a new environment especially in a large corporation usually you are confronted with many systems.  Security will take care of setting up your access across whatever platforms there may be.  But generally you are left holding the bag with setting up your ssh keys and any profile customizations not to mention distribution of any scripts or tools you have come to rely upon.  Of course before you put any tools on a system there are several things to consider.  You definitely want to consider the environments you are first performing the distributions on and it is always good to start with development or lab environments and move out from there.  Also you will need to consider the corporate policies related to the environment which might limit your ability to even have your own set of tools and scripts.  You may be limited down to simple .profile changes and ssh keys.  Implementing a script to push these keys and profiles out may need to go through various degrees of red tape.  Whatever policies and requirements exist in your organization are your responsibility to know and to determine how or if the tools discussed here may be used.

Read more of this post