Co-Engineered Together: OpenStack Platform and Red Hat Enterprise Linux

OpenStack is not a software application that just runs on top of any random Linux. OpenStack is tightly coupled to the operating system it runs on and choosing the right Linux  operating system, as well as an OpenStack platform, is critical to provide a trusted, stable, and fully supported OpenStack environment.

OpenStack is an Infrastructure-as-a-Service cloud management platform, a set of software tools, written mostly in Python, to manage hosts at large scale and deliver an agile, cloud-like infrastructure environment, where multiple virtual machine Instances, block volumes and other infrastructure resources can be created and destroyed rapidly on demand.

ab 1

In order to implement a robust IaaS platform providing API access to low level infrastructure components, such as block volumes, layer 2 networks, and others, OpenStack leverages features exposed by the underlying operating system subsystems, provided by Kernel space components, virtualization, networking, storage subsystems, hardware drivers, and services that rely on the operating system’s capabilities.

Exploring how OpenStack Flows

OpenStack operates as the orchestration and operational management layer on top of many existing features and services. Let’s first examine how Nova Compute service is implemented to better understand the OpenStack design concept. Nova is implemented through 4 Linux services: nova-api which is responsible to accept Nova’s API calls, nova-scheduler which implements a weighting and filtering mechanism to schedule creation of new instances, nova-conductor which makes database operations, and nova-compute which is responsible to create and destroy the actual instances. A message-bus, implemented through Oslo Messaging and instantiated since Red Hat Enterprise Linux OpenStack Platform 5 using  RabbitMQ, is used for services inner-communication.

To create and destroy instances, which are usually implemented as virtual machines, nova-compute service uses a supportive backend driver to make libvirt API calls, while Libvirt manages qemu-kvm virtual machines on the host.

All OpenStack services are implemented in a similar manner using integral operating system components. Each OpenStack distribution may decide on using different implementations. Here we will focus on the implementation choices made with Red Hat Enterprise Linux OpenStack Platform 6. For example the DHCP service is implemented using dnsmasq service, Security Groups are implemented using Linux IPTables, Cinder commonly uses LVM Logical Volumes for block volumes and scsi-target-utils to share tgt volumes over iSCSI protocol.ab 2

This is an oversimplified description of the complete picture and many additional sub-systems are also at play, such as SELinux with corresponding SELinux policies for each service and files in use, Kernel namespaces, hardware drivers and many others.

When deploying OpenStack in a highly available configuration, which is commonly found in real-world production environments, the story becomes even more complex with HAProxy load-balancing traffic, Pacemaker active-active clusters that use multicast for heartbeat, Bonding the Network Interfaces using Kernel’s LACP bonding modules, Galera which implements a database multi-master replication mechanism across the controller nodes, and RabbitMQ message broker which uses an internal queue mirroring mechanism across controller nodes.

Co-engineering the Operating system with OpenStack

Red Hat’s OpenStack technologies are purposefully co-engineered with the Red Hat Enterprise Linux operating system, and integrated with all its subsystems, drivers, and supportive components – to help deliver a trusted, long-term stability, and a fully supported, production-ready, OpenStack environment.

Red Hat is uniquely positioned to support customers effectively across the entire stack, we maintain an engineering presence that proactively works together with each of the communities across the entire stack, starting with the Linux Kernel, all the way up to the hypervisor and virtualized guest operating system. In addition Red Hat Enterprise Linux OpenStack Platform maintains the largest OpenStack-certified partner ecosystem, working closely with  OpenStack vendors to certify 3rd party solutions, and work through support cases when external solutions are involved.ab 3

Red Hat Enterprise Linux OpenStack Platform also benefits from the rich hardware certification ecosystem Red Hat Enterprise Linux offers working with major hardware vendors to provide driver compatibility. For example, the Neutron single root I/O virtualization (SR-IOV) feature is built on the top of the SR-IOV certified Kernel driver. Similarly, the support for tunneling protocols (VXLAN, NVGRE) to offload, which is key for performance, is derived from the Red Hat Enterprise Linux driver support.

We are doing this not only to deliver world-class, production-ready support for the whole platform stack, but also to drive new features requested by customers –  since adding new functionality to OpenStack often requires invasive changes to a large portion of the stack,  from OpenStack APIs, down to the kernel.

Introducing New NFV Features – NUMA, CPU Pinning, Hugepages

The Network Functions virtualization (NFV) use case, which required adding support for NUMA, CPU Pinning, and Hugepages, is a good example of this implementation. To support these features, work began at the kernel level, both for memory management and KVM. Red Hat Enterprise Linux 7.0 kernel added support for 2M and 1G huge pages for KVM and virtual machines, and IOMMU support with huge pages. Work on the Kernel continued and with the Red Hat Enterprise Linux 7.1 kernel, adding support for advanced NUMA placement and dynamic huge pages per NUMA node.

In parallel with the features in Red Hat Enterprise Linux Kernel, changes were made to qemu-kvm-rhev to utilize these features and were exposed via the libvirt API and XML.

Red Hat engineers worked on getting the needed changes into the OpenStack Nova project to determine availability of huge pages on the host as well as assign them to individual VMs when requested. Nova was then enhanced so that users could define hugepages as a requirement for all VMs booted from a given image, via image properties, or for all VMs booted with a given flavor, via flavor extra specs. The scheduler was enhanced to track the availability of huge pages as reported by the compute service and then confirm that the VM is scheduled to a host capable of fulfilling the hugepages requirement specified.

Coordinating the support for these features across the entire stack, kernel -> qemu-kvm -> libvirt -> Nova-compute -> nova-schedule -> nova-api , required several different teams, working in several upstream communities, to work closely together. Thanks to Red Hat’s strong engineering presence in each of the respective communities, and the fact that most of these engineers were all within the same company, we were able to drive each of these features into the upstream code bases and coordinate backporting them to Red Hat Enterprise Linux and Red Hat Enterprise Linux OpenStack Platform so that they could be utilized together with the combination of RHEL 7.1 used as the base operating system for Red Hat Enterprise Linux OpenStack Platform 6 which is based on the upstream Juno release.

Supporting Changes  

Red Hat Enterprise Linux 7.0 and 7.1 also included numerous enhancements to better support Red Hat Enterprise Linux OpenStack Platform 6. Some of these  enhancements include Kernel changes in the core networking stack to better support VXLAN with  TCP Segmentation Offloading (TSO) and Generic Segmentation Offloading (GSO) which used to lead to guest crashes, fixed issued with dhcpclient sending requests over VXLAN Interfaces, SELinux policy fixes and enhancements for Glance image files and other services, enhancements fixing issues in qemu-kvm for librdb (Ceph), changes in iscsi-initiator-utils preventing hosts from potential hangs while reboot, and much more.

Conclusion

In order to implement an IaaS solution and provide API access to low level infrastructure components, OpenStack needs to be tightly integrated with the operating system it runs on, making the operating system a crucial factor for long-term OpenStack stability. Red Hat Enterprise Linux OpenStack Platform is co-engineered and integrated with various RHEL services and subsystems leading to an IaaS Cloud environment enterprise customers can trust. To provide the world class support Red Hat customers are used to, Red Hat is actively participating in upstream communities across all OpenStack projects, positioning Red Hat to be able to support OpenStack effectively across all components in use. Red Hat’s active participation in the upstream communities also enables Red Hat to introduce and drive new OpenStack features and functionalities requested by customers.