Enterprise SAN Switch Upgrade

Introduction

In an Enterprise setting upgrading storage infrastructure is quite different from running updates on your home PC; or at least it should be. While updates expand functionality, simplify interfaces, fix bugs and close vulnerabilities they can also introduce new bugs and vulnerabilities. Sometimes the new bugs are contingent upon factors which exist in your environment and can result in encountering the issue the bug creates. In an Enterprise environment where many users and sometimes customers rely upon the storage infrastructure the impact of an issue caused by an upgrade can be broad and affect business credibility with potentially even legal ramifications. Therefore, having a process to mitigate as many risks as possible is a necessity. The process presented here rests in a general framework with specific steps related to Cisco and Brocade SAN switch upgrades.

Overview

The process described at a high-level here is a good general framework for any shared infrastructure upgrade in an Enterprise environment.

1. Planning
   • Document current environment cross section from CMDB and/or direct system inquiry.
   • (Server Hardware Model, OS and Adapter Model/Firmware/Driver as well as SAN Switch Model/Firmware and current Storage Model/Code Level)
   • Ensure the SAN infrastructure is under vendor support so that code may be downloaded and support may be engaged if any problems are encountered.
   • Download and Review Release notes for the top 3 recent code releases.
   • Use vendor interoperability documents or web applications to validate supportability in your environment using this previously gathered information.
   • Choose the target code level. (Often N-1 is preferred over N, bleeding edge latest releases, unless significant vulnerabilities or incompatibility with your environment exists.)
2. Preparation
   • Download the target release installation code and any upgrade test utilities provided by the vendor.
   • Upload the target code and test utility and run test utility. (clean up old diagnostics and install images no longer needed to provide necessary space for new code and upgrade process)
   • Run initial health checks on the storage systems.
   • Gather connectivity information from SAN and Storage devices and verify connection and path redundancy.
   • Initiate a resolution plan before scheduling the upgrade for any identified issues.
   • Submit change control and obtain approval for upgrade.
3. Upgrade
   • Rerun the upgrade test utility to verify issues are still resolved.
   • Perform health checks and gather interface status showing pre-upgrade connectivity
   • Clear stats and logs so that all events will be related to the upgrade
   • Run configuration backup, diagnostic snapshot and list logs to a file downloading each to a central configuration repository.
   • Initiate any prerequisite components microcode upgrades (transceiver firmware, etc) and validate completion.
   • Initiate system update and monitor upgrade process
   • Upon completion validate upgrade, perform health checks and gather post-upgrade interface status and validate the dependent systems connectivity.