Simple Local Configuration Management on Cisco NX-OS

Hello all! In this post I will cover a simple and effective way to manage configuration locally on Cisco NX-OS. Most companies will schedule backups of their devices configuration via a remotely managed tool server or manually download backups via a file transfer protocol to a central database. Using configuration management servers is by far the first ( and maybe best ) choice for most enterprises who want a central repository for their devices configuration, but let's explore adding a second layer of resiliency in the event that tool server becomes unavailable. This post is to showcase just one, of many ways, to achieve just that. For this example I will be using a Cisco Nexus 7009 running NX-OS 6.2(10). NX-OS has a very nice system management feature called Rollback. We will be leveraging this feature to store what is called checkpoints locally on the system. There are a few ways to generate these checkpoints:

1) Manually generated. ( Great practice to perform before major system changes ).

2) "Scheduling" the checkpoints. ( via the NX-OS scheduler feature )

3) System auto-generated. Cisco NX-OS automatically generates system checkpoints during the following events:

  • Disabling an enabled feature.
  • Removing an instance of a layer 3 protocol.
  • License expiration of a feature.

Lets go over the three! First up is the manually generated checkpoint:

Nexus_7K# checkpoint  
.
user-checkpoint-1 created Successfully

Done  

Well, that was super easy... So what just happened? We created a system checkpoint! Now we have a "checkpoint" of our system's configuration to rollback to in the event new configuration has an undesired impact to the network. We can view the checkpoint as follows:

Nexus_7K# show checkpoint summary  
User Checkpoint Summary  
--------------------------------------------------------------------------------
1) user-checkpoint-1:  
Created by admin  
Created at Mon, 19:25:16 09 Nov 2015  
Size is 2,831 bytes  
Description: None

Nexus_7K# show checkpoint user-checkpoint-1  
--------------------------------------------------------------------------------
Name: user-checkpoint-1


!Command: Checkpoint cmd vdc 5
!Time: Mon Nov  9 19:25:15 2015

version 6.2(10)  
hostname Nexus_7K

cfs eth distribute  
---config abbreviated for brevity---

Now let's make a change to the 7k and see if we can rollback:

Nexus_7K# conf t  
Enter configuration commands, one per line.  End with CNTL/Z.  
Nexus_7K(config)# int lo1000  
Nexus_7K(config-if)# ip add 10.10.10.10/32  
2015 Nov  9 19:38:29 Nexus_7K %ETHPORT-5-IF_UP: Interface loopback1000 is up  
Nexus_7K(config-if)# end  
Nexus_7K#show diff rollback-patch running-config checkpoint user-checkpoint-1  
Collecting Running-Config  
#Generating Rollback Patch

!!                                                                   
no interface loopback1000  

Above we created interface loopback 1000, added an IP address and performed a "diff" of the current running-configuration and the checkpoint we generated before the change. The system parsed the running-configuration and generated a "Rollback Patch" configuration which would be used to return the system back to its checkpoint state. At this point, we still have not initiated a system rollback, we've just viewed what the system will use, the rollback-patch, to return the system back to the checkpoint we generated earlier. Let's roll back:

Nexus_7K# rollback running-config checkpoint user-checkpoint-1  
Note: Applying config parallelly may fail Rollback verification  
Collecting Running-Config  
#Generating Rollback Patch
Executing Rollback Patch  
2015 Nov  9 19:48:28 Nexus_7K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface loopback1000 is down (Administratively down)  
Generating Running-config for verification on rollback  
Generating Patch for verification  
Verification is Sucessful.

Rollback completed successfully.

Nexus_7K#show int lo1000  
                        ^
Invalid range at '^' marker.  

Great! The rollback was successful and we no longer have interface loopback 1000 in our current running-configuration. As mentioned earlier in this post, manually creating checkpoints is a good habit to get into before making configurations that may have an adverse affect on your network. But what if another user or yourself made a change that needed to be removed and a checkpoint was not manually generated? In this case you would typically reference a configuration that was backed-up by a configuration management tool server, but what if the change made that tool server unreachable? If that event occurred, a locally stored checkpoint could save you from hours of headache and work. So let's create a scheduler to automatically create system checkpoints so we don't have to rely on ourselves to remember. First we will enable the scheduler feature by issuing "feature scheduler":

Nexus_7K# conf t  
Enter configuration commands, one per line.  End with CNTL/Z.  
Nexus_7K(config)# feature scheduler  

Then we will create a scheduled process for creating system checkpoints via a scheduler job.

Nexus_7K(config)# scheduler job name CREATE_CHECKPOINT  
Nexus_7K(config-job)# checkpoint

Next we will schedule the job we created to occur everyday at 11pm:

Nexus_7K(config-job)# exit  
Nexus_7K(config)# scheduler schedule name CHECKPOINT_SCHEDULE  
Nexus_7K(config-schedule)# job name CREATE_CHECKPOINT  
Nexus_7K(config-schedule)# time daily 23:00  

Now we wait until 11pm and the scheduler will issue the command "checkpoint" for us. We can verify the configuration of the scheduler and the scheduled operations using the following commands:

Nexus_7K# show scheduler config  
config terminal  
  feature scheduler
  scheduler logfile size 16
end


config terminal  
 scheduler job name CREATE_CHECKPOINT
checkpoint

end

config terminal  
  scheduler schedule name CHECKPOINT_SCHEDULE
    time daily 23:00
    job name CREATE_CHECKPOINT
end  
Nexus_7K# show scheduler logfile  
Job Name       : CREATE_CHECKPOINT                 Job Status: Success (0)  
Schedule Name  : CHECKPOINT_SCHEDULE               User Name : admin  
Completion time: Mon Nov  9 23:00:01 2015  
--------------------------------- Job Output ---------------------------------
`checkpoint`
.
user-checkpoint-2 created Successfully

Done  
==============================================================================

The output above shows our job was successfully completed and we have a new user generated checkpoint. The checkpoints can be viewed in the exact way we interacted with the manually generated checkpoints:

Nexus_7K# show checkpoint summary  
User Checkpoint Summary  
--------------------------------------------------------------------------------
1) user-checkpoint-1:  
Created by admin  
Created at Mon, 19:25:16 09 Nov 2015  
Size is 2,831 bytes  
Description: None

2) user-checkpoint-2:  
Created by admin  
Created at Mon, 23:00:01 09 Nov 2015  
Size is 2,983 bytes  
Description: None  

By default NX-OS will store up to 10 checkpoints. The scheduler will automatically overwrite the oldest checkpoint once it tries to generate an "11th" checkpoint, so basically it's circular. It is also worth stating these checkpoints are volatile, meaning after the system executes the "write erase" or "reload" command, checkpoints are deleted. You can also use the clear "checkpoint database command" to clear out all checkpoint files. Throughout this post we've been allowing NX-OS to name our checkpoints for us IE: "user-checkpoint-n". However, this may not be the most desirable name based on preference/opinion. Let's explore a way of naming the checkpoint based on a variable, that variable being the date. For this example I've created a very simple python file called create.checkpoint.py and uploaded it to the 7K's bootflash. The create.checkpoint.py script can be found below:

#!/usr/bin/env python
#Import the datetime module
from datetime import date

#Define a variable for the date
today = date.today()

#Define a CLI variable
cp_cmd = 'checkpoint SCHEDULER_CREATED_CHECKPOINT_%s' % (str(today))

#Call the NX-OS CLI function with the cp_cmd variable
cli(cp_cmd)  

Next we'll create a new scheduler job to run create.checkpoint.py at 11pm everyday:

Nexus_7K(config)# scheduler job name CREATE_CHECKPOINT  
Nexus_7K(config-job)# source create.checkpoint.py  
Nexus_7K(config-job)# exit  
Nexus_7K(config)# scheduler schedule name CHECKPOINT_SCHEDULE  
Nexus_7K(config-schedule)# job name CREATE_CHECKPOINT  
Nexus_7K(config-schedule)# time daily 23:00  

Let's see what checkpoint looks like now:

Nexus_7K# show checkpoint summary  
User Checkpoint Summary  
--------------------------------------------------------------------------------
1) SCHEDULER_CREATED_CHECKPOINT_2015-11-09:  
Created by admin  
Created at Mon, 23:00:00 09 Nov 2015  
Size is 3,025 bytes  
Description: None  

Ah, much better! Now onto our last topic: System Generated Checkpoints. To showcase system generated checkpoints I will simply remove a feature and see if NX-OS generates a checkpoint automatically:

Nexus_7K(config)# no feature udld  
2015 Nov  9 20:54:36 Nexus_7K %UDLD-5-UDLD_DISABLED: UDLD Disabled  
Nexus_7K(config)# show checkpoint summary  
User Checkpoint Summary  
--------------------------------------------------------------------------------
1) SCHEDULER_CREATED_CHECKPOINT_2015-11-09:  
Created by admin  
Created at Mon, 23:00:02 09 Nov 2015  
Size is 3,025 bytes  
Description: None

System Checkpoint Summary  
--------------------------------------------------------------------------------
2) system-fm-udld:  
Created by admin  
Created at Mon, 23:14:35 09 Nov 2015  
Size is 3,025 bytes  
Description: Created by Feature Manager.  

It appears the feature manager process was nice enough to generate a system checkpoint when we removed the udld feature! And with that, I'm going to wrap this post up. I hope you enjoyed going over system checkpoints with me!