InterCloud has been pursuing an effort to automate its infrastructure for some time now. We have experience in automated configuration management on Brocade, Cisco and Juniper hardware.
Since 2011, we have been using RANCID [Really Awesome New Cisco confIg Differ] for the backup and versioning of the configuration of our equipment as well as for systematizing the changes.
RANCID quickly showed us its limitations, and, in general, the community considers the project obsolete — the development stack has gone out of style.
But that’s not a big surprise since RANCID is based on Expect, the TCL extension designed by Don Libes in the early 1990s. And, although effective, RANCID has found a home among the “lost arts,” much like AWK and Perl, and is mastered by only a few (including ourselves), all of whom are becoming increasingly rare — and decreasingly vocal.
The fact is that RANCID is based on a number of assumptions that are no longer valid in the context of modern production network environments. These include its emulating the slow and random typing rate of a human so as not to “overload” the system’s CPU, storing configurations in a centralized and outdated file version management system (CVS), concentrating run commands in order to query devices on their configuration in vendor-dependent scripts and storing passwords in plain text in flat configuration files.
Our engineering team has patched RANCID in many places, even rewriting some of its scripts, in an attempt to correct many of these limitations. This task has become tedious and less feasible over time.
One open-source alternative is known as Oxidized. Written in Ruby, it aims to correct several of RANCID’s restrictions. Notably, Oxidized stores configurations in databases to facilitate scaling, integrates its own monitor for scheduling, and is open to other file versioning systems, such as Git. Alas, the Oxidized developer doesn’t have enough time for the project’s upkeep, but I’m sure he’d be interested in hearing from you, if you have time to spare. You can contact him at https://github.com/ytti/oxidized#help-needed
Further, devops tools are beginning to emerge to address operator network automation issues. Ansible and Salt are among the most frequently used tools in production. Today our team mainly uses Ansible, written in Python. Therefore, the release of Eric Chou’s most recent book “Mastering Python Networking” comes at the perfect time, and is particularly interesting for InterCloud. Chou also writes regularly about these topics on his blog.
In their latest Priority Queue Episode, published on Nov. 9, Packet Pushers present a podcast on the book “Mastering Python Networking,” and lets the author share his insights on various aspects of Python network programming.
With Ethan Banks as the host, the informative podcast discusses the many flavors of software-defined (or “driven”) networks and addresses such issues as the main challenges of network operation, highlighting in particular a variety of proprietary APIs supported (or not) by hardware vendors. The podcast raises the omnipresent question of standardization vs. innovation, as illustrated by the lack of progress in vendors agreeing to accept an industry standard for MIBs on top of the venerable SNMP MIB-II, or by the slow pace of implementation of YANG data models. More information is available here.
The duo discusses the emergence of NAPALM as a unitary abstraction layer to support a variety of equipment implemented in operator network production. One of its main interests, according to the two speakers in the podcast, seems to be the generalization of the configuration “commit/ rollback.” It is a feature that is supported by more recent network operating systems, such as JunOS or IOS-XR and is quite appealing in terms of configuration management. Providing tools to emulate it on older systems is extremely useful. I am very curious to see if — and how — NAPALM is able to implement these automatic rollback features.
Another aspect highlighted by Chou concerning the richness of Python libraries is the abstraction layer of SQLAlchemy databases. Chou took Banks by surprise on the matter: Banks wanted to talk about “Time Series Databases” to briefly introduce this family of databases and their applications. InterCloud implements InfluxDB and Telegraph and, like Banks, we do not use SQLAlchemy to interact with these tools. However, we are going to study this possibility with great interest. Take a look at https://github.com/amancevice/influxalchemy and https://github.com/jjmalina/pyinfluxql. Both seem to demonstrate that it’s not as simple as Chou claims.
Banks and Chou subsequently discuss Python 3 vs 2.
At InterCloud, we standardized our developments on the latest version of Python early on. We saw this as the only option since there is no new functionality implemented in Python 2. In addition, Python 3 consistently processes internationalized strings coming from the network, and prevents wrong assumptions from the programmer, which is always useful when you develop an API for use in non ASCII-centric environments. Another benefit is that Python 3 handles IPv4 and IPv6 address manipulation and semantics through its IP address standard module.
Banks asks Chou his opinion on several hot network automation topics, such as use of server-less computing or Raspberry PIs. Chou doesn’t seem to have any experience of using these technologies in production environments but nevertheless shares an enthusiastic point of view.
As far as InterCloud is concerned, some of our developers are very keen on using AWS Lambda and we have already extensively deployed PIs to monitor hundreds of points in our network, get a summary view of performance and to be alerted in the event of an incident. There is no comparison available between the costs associated with this type of deployment and traditional network monitoring solutions.
Chou recommends developing on PI 3 models and deploying Zero Ws. It seems like a great idea and would certainly make a good topic for another blog entry. Banks further illustrates the convenience of prototyping with Raspberry PIs using the example of a Kubernetes cluster with four Raspberry PIs as described in different book he covered in another this episode.
Chou also expresses several “strong opinions” during the interview, for example, when touching on the topic of “the best web framework in Python,” claiming network engineers should not hesitate for long before settling for Flask. We wait anxiously to receive and read the book to determine if his opinions are substantiated. Either way, it’s always useful to learn from various experiences.
The podcast successfully provides us with an overview of the book: The first chapters cover automation topics such as Expect for Python, Ansible and API REST automation with Flask, while the final chapters describe “real” SDN with OpenFlow.
The publication will definitely be an excellent addition to our library. It clearly covers areas relevant to our engineering team’s daily activities as well our research about routing hybrid networks at the application layer, using the OpenFlow protocol with the POX orchestrator and an SDN domain specific language extending Python, called Pyretic.
By the way, if any of these subject matters are of interest to you, InterCloud is quickly expanding our engineering team (engineers, architects, leads, etc.). If you’d like to work with InterCloud, drop us a message!