Jay's Blog

Cisco Nexus 1000v manual VEM upgrade to Release 4.2(1) SV1(4)

Note 1: Even though Cisco fully supports VUM based upgrades we generally recommend the CLI for VEM upgrades as it provides more granular control on what gets upgraded when.

Note 2: If you plan on manually upgrading the VEM modules make sure you have disabled VMware Update Manager (VUM) service so that it does not automatically roll back changes we make. Once the vem upgrade is complete we can start it back up.

Step 0 Run a “show mod” to check the status of the modules

switch#  show mod
Mod  Ports  Module-Type                       Model               Status
---  -----  --------------------------------  ------------------  ------------
1    0      Virtual Supervisor Module         Nexus1000V          active *
3    248    Virtual Ethernet Module           NA                  ok
4    248    Virtual Ethernet Module           NA                  ok
5    248    Virtual Ethernet Module           NA                  ok
6    248    Virtual Ethernet Module           NA                  ok

Mod  Sw               Hw
---  ---------------  ------
1    4.0(4)SV1(3b)     0.0
3    4.0(4)SV1(3b)     1.11
4    4.0(4)SV1(3b)     1.11
5    4.0(4)SV1(3b)     1.11
6    4.0(4)SV1(3b)     1.11 

Mod  MAC-Address(es)                         Serial-Num
---  --------------------------------------  ----------
1    00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8  NA
3    02-00-0c-00-03-00 to 02-00-0c-00-03-80  NA
4    02-00-0c-00-04-00 to 02-00-0c-00-04-80  NA
5    02-00-0c-00-05-00 to 02-00-0c-00-05-80  NA
6    02-00-0c-00-06-00 to 02-00-0c-00-06-80  NA

Mod  Server-IP        Server-UUID                           Server-Name
---  ---------------  ------------------------------------  --------------------
1    172.16.165.55    NA                                    NA
3    172.16.165.91    dcedbaac-1dc0-11df-0000-00000000000e  172.16.165.91
4    172.16.165.92    dcedbaac-1dc0-11df-0000-00000000000d  172.16.165.92
5    172.16.165.93    dcedbaac-1dc0-11df-0000-00000000000c  172.16.165.93
6    172.16.165.94    dcedbaac-1dc0-11df-0000-00000000000b  172.16.165.94
* this terminal session

This is looking good, with one active VSM and multiple VEM modules and all running the same version.

Step 1 Download the Nexus1000v.4.2.1.SV1.4.zip file from Cisco.com and unzip it and you will find the new 4.2(1) SV1(4) vem images bundle vem-release_final.tar.gz under the VEM folder. Copy this file to the VSM bootflash.

switch# copy scp://[email protected]/srv/Cisco/Nexus1000v.4.2.1.SV1.4/VEM/vem-release_final.tar.gz bootflash:

Step 2 Update the VSM with the latest VEM modules

switch# vmware vem upgrade update-vibs bootflash:vem-release_final.tar.gz

Step 3 Notify vCenter Server/VUM that a latter version of VEM is available for upgrade.

switch# vmware vem upgrade notify
switch# show vmware vem upgrade status

Step 4 Confirm that VMware Update Manager is disabled (note 2) and then Go the vCenter Server, Home -> Inventory -> Networking and Select the Nexus 1000v swtich and Click the DVS Summary tab to check for the availability of a software upgrade and accept it. Now come back to VSM and check the status. And since we are planning to upgrade the VEM modules manually you can ahead and do “vmware vem upgrade proceed” and then complete the process.

switch# show vmware vem upgrade proceed
switch# show vmware vem upgrade status
switch# vmware vem upgrade complete
switch# show vmware vem upgrade staus

So far we let VSM, vCenter Server and by extension VMware update manager know that there is an VEM update available. Now we have to go ahead and actually install the VEM modules on the ESX hosts.

Step 5 Copy the VEM bundle to the ESX hosts /tmp folder or to a shared vmfs volume . Which VEM bundle to copy and install will depend on the build number of your ESX host, see the VEM compatibility table here for more details. Then depending on whether you are using the .vib or .zip files, you run either of the following command to install the VEM modules on the ESX hosts.

esx-host-2# esxupdate -b ./cross_cisco-vem-v130-4.2.1.1.3.9.0-1.9.1.vib update
esx-host-2# esxupdate --bundle VEM410-201101108-BG.zip update

Step 6 Now repeat the same steps on all the ESX hosts until you have, this on show mod.

switch#show mod
Mod  Ports  Module-Type                       Model               Status
---  -----  --------------------------------  ------------------  ------------
1    0      Virtual Supervisor Module         Nexus1000V          active *
3    248    Virtual Ethernet Module           NA                  ok
4    248    Virtual Ethernet Module           NA                  ok
5    248    Virtual Ethernet Module           NA                  ok
6    248    Virtual Ethernet Module           NA                  ok

Mod  Sw                Hw
---  ----------------  ------------------------------------------------
1    4.0(4)SV1(3b)      0.0
3    4.2(1)SV1(4)      VMware ESXi 4.1.0 Releasebuild-320137 (2.0)
4    4.2(1)SV1(4)      VMware ESXi 4.1.0 Releasebuild-320137 (2.0)
5    4.2(1)SV1(4)      VMware ESXi 4.1.0 Releasebuild-320137 (2.0)
6    4.2(1)SV1(4)      VMware ESXi 4.1.0 Releasebuild-320137 (2.0)

Mod  MAC-Address(es)                         Serial-Num
---  --------------------------------------  ----------
1    00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8  NA
3    02-00-0c-00-03-00 to 02-00-0c-00-03-80  NA
4    02-00-0c-00-04-00 to 02-00-0c-00-04-80  NA
5    02-00-0c-00-05-00 to 02-00-0c-00-05-80  NA
6    02-00-0c-00-06-00 to 02-00-0c-00-06-80  NA

Mod  Server-IP        Server-UUID                           Server-Name
---  ---------------  ------------------------------------  --------------------
1    172.16.165.55    NA                                    NA
3    172.16.165.91    dcedbaac-1dc0-11df-0000-00000000000e  172.16.165.91
4    172.16.165.92    dcedbaac-1dc0-11df-0000-00000000000d  172.16.165.92
5    172.16.165.93    dcedbaac-1dc0-11df-0000-00000000000c  172.16.165.93
6    172.16.165.94    dcedbaac-1dc0-11df-0000-00000000000b  172.16.165.94

Step 7 Once you confirm that all your ESX hosts have the latest VEM modules, Go ahead and start your VSM upgrade 🙂

Jay

9 responses to “Cisco Nexus 1000v manual VEM upgrade to Release 4.2(1) SV1(4)”

  1. joel Avatar
    joel

    I know the cisco documentation explain all these steps, but i’m curious as to why steps 1-4 are needed. Could I just extract the tar file, upload the necessary VEM bundle, and install the VEM module? So could you explain why steps 1-4 are needed?

    Reply
  2. Jay Avatar
    Jay

    In Steps 1-4 we let VSM, vCenter Server and by extension VMware update manager know that there is an VEM update available. This is important because if you dont do this, the vCenter Server and VMware Update Manager will not know about the new VEM and once the upgrade is complete and you enable VUM it will scan the host and see what “it-believes” to be incorrect VEM module and will promptly re-install the old VEM module. We have seen this happen more than once and hence the emphasis to make sure all the components are in sync at all stages of the upgrade. Also post upgrade if you add another host you will end up with a older version of VEM installed as well. So copying and updating the new VEM module in VSM and VC is pretty important.

    Hope this helps explain it a bit, if not feel free to ask.

    [edited to add more context]

    Reply
  3. Michael McNamara Avatar
    Michael McNamara

    Hi Jay,

    Thanks for taking the time to provide your perspective on the upgrade process.

    When you applied the VEM updates to the ESX host I’m assuming you placed the ESX host into maintenance mode? I’d guess that the VEM update would be service impacting if the ESX host isn’t in maintenance mode.

    Thanks,
    Mike

    Reply
    1. Jay Avatar
      Jay

      Technically speaking the VEM update is not that service impacting, if you were running a continuous ping to the VMs on the host, you will at the most lose 2-3 pings during the point where the current VEM module unloads and the new VEM completes loading.

      Having said that the correct answer is it depends. It depends on the VMs running on those ESX hosts, If they are mission critical (paycheck impacting) VMs then, YES, you should place the host in maintenance mode , perform the VEM upgrade and we also recommend that you reboot the ESX host after upgrading the VEM module to confirm that the the host reboot does not introduce other issues and then once the host is back, move some non critical VMs and test connectivity and if everything checks okay then bring the critical VMs.

      Reply
  4. marcel Avatar
    marcel

    Hi Jay,

    one question,

    as far as i read the cisco documentation, shouldn’t you manually upgrade the VEMs between the “upgrade proceed” and “upgrade complete” steps?

    Marcel

    Reply
    1. Jay Avatar
      Jay

      Yes, that would be ideal. However you could run into issues during host upgrade and may end up leaving the upgrade process in a limbo. So I generally prefer splitting the process to as many independent pieces as possible. This way the vem upgrade in the VSM/vCenter is independent of the vem upgrade on the hosts themselves. An if you run out of time in your maintenance window when upgrading VEMs, you can always get back them as the 1000v upgrade itself will be complete, pending a few hosts running older code, which is a better option than a partial upgrade.

      Reply
  5. viral Avatar
    viral

    I wish that I have read your blog before upgrading to latest code.
    TAC told me to ignore the step so all the vem are upgraded but vsphere is not aware of this new VEM. so when I add new host it adds the old vem .
    I believe that I need to know follow step 1 to 4.

    Jay, if I disable the VUM, step 1-4 should correct my problem & hopefully, I should not loose anything

    Sincerely,
    Viral

    Reply
    1. Jay Avatar
      Jay

      If you have already upgraded to 1.4 then steps 1 & 2 are not needed. they are only needed when you are running 1.3.x code and you want to upgrade the VEM before the VSM upgrade. So if you are already running 1.4, then you only need to run steps 3-4. After that you should be able add a new host with the latest VEM either manually or using VUM.

      Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.