RDO Community News

See also blogs.rdoproject.org

EasyFix: Getting started contributing to RDO

It can be intimidating trying to get involved in an open source project. Particularly one as huge and complicated as OpenStack. But we want you to join us on the RDO project, so we're trying to make it as easy as possible to get started.

To that end, we've started the EasyFix initiative. EasyFix is a collection of "easy" tickets that you should be able to get started on without having a deep knowledge of the RDO project. And in the process, you'll learn what you need to know to move on to more complex things.

These tickets cover everything from documentation, to packaging, to helping with events, to writing tools to make things easier. And each ticket comes with a mentor - someone who has volunteered to help you get through that first commit.

There's also a general mentors list, categorized by area of expertise, if you just have a question and you're not sure who to ping on IRC.

We've been running the EasyFix program for about 3 weeks now, and in that time, four new contributors have started on the RDO project.

We're very pleased to welcome these new contributors, and hope they'll be around for a long time to come.

Christopher "snecklifter" Brown - I'm Christopher Brown, a HPC Cloud Engineer based in Sheffield in the UK. I've worked on OpenStack since June 2015. I used to work on packaging for the Fedora project so I am transferring and updating those skills to help out with RDO when I can.

Anthony Chow - I am a software developer for legacy networking equipment wanting to be a Developer Advocate. I have been venturing into other technologies such as cloud, container, and configuration management tools. Passionate in learning and sharing technologies related topics.

Treva Williams - T. Nichole Williams is an RHCSA 7 Certified Linux and OpenStack engineer and content author for LinuxAcademy.com, & an OpenStack active technical contributor in the Magnum project. She is actively involved in several OpenStack, OpenShift, RDO, and Ceph communities and groups. When not OpenStacking or Cephing, she enjoys doggos, candy, cartoons, and playing "So You Think You're a Marine Biologist" on Google.

Ricardo Arguello - I am an Open Source enthusiast that tries to collaborate as much as I can. I have helped in the past with patches for WildFly, and as a Fedora packager. The RDO project is very complex an intimidating, but collaborators are welcome and there are easy to fix bugs for newbies to make their first contributions! That makes RDO a great community if you are interested in helping the project while learning about Open Stack internals in the process.

If you want to participate in the RDO project, we encourage you to find something on the EasyFix list and get started. And please consider attending EasyFix office hours, Tuesdays at 13:30 UTC.

View article »

What's new in ZuulV3

Zuul is a program used to gate a project's source code repository so that changes are only merged if they pass integration tests. This article presents some of the new features in the next version: ZuulV3

Distributed configuration

The configuration is distributed accross projects' repositories, for example, here is what the new zuul main.yaml configuration will look like:

- tenant:
    name: downstream
          - config
          - restfuzz
          - openstack-infra/zuul-jobs:
              include: job
              shadow: config

This configuration describes a downstream tenant with two sources. Gerrit is a local gerrit instance and openstack.org is the review.openstack.org service. For each sources, there are 2 types of projects:

  • config-projects hold configuration information such as logserver access. Jobs defined in config-projects run with elevated privileges.
  • untrusted-projects are projects being tested or deployed.

The openstack-infra/zuul-jobs has special settings discussed below.

Default jobs with openstack-infra/zuul-jobs

The openstack-infra/zuul-jobs repository contains common job definitions and Zuul only imports jobs that are not already defined (shadow) in the local config.

This is great news for Third Party CIs that will easily be able to re-use upstream jobs such as tox-docs or tox-py35 with their convenient post-processing of unittest results.

In-repository configuration

The new distributed configuration enables a more streamlined workflow. Indeed, pipelines and projects are now defined in the project's repository which allows changes to be tested before merging.

Traditionaly, projects' CI needed to be configured in two steps: first, the jobs were defined, then a test change was rechecked until the job was working. This is no longer needed because the jobs and configurations are directly set in the repository and CI change undergoes the CI workflow.

After being registered in the main.yaml file, a project author can submit a .zuul.yaml file (along with any other changes needed to make the test succeed). Here is a minimal zuul.yaml setting:

- project:
    name: restfuzz
        - tox-py35

Zuul will look for a zuul.yaml file or a zuul.d directory as well as hidden versions prefixed by a '.'. The project can also define its own jobs.

Ansible job definition

Jobs are now created in Ansible, which brings many advantages over the Jenkins Jobs Builder format:

  • Multi-node architecture where tasks are easily distributed,
  • Ansible module ecosystem simplify complex task, and
  • Manual execution of jobs.

Here is an example:

- job:
    name: restfuzz-rdo
    parent: base
    run: playbooks/rdo
      - name: cloud
        label: centos
      - name: fuzzer
        label: fedora

Then the playbook can be written like this:

- hosts: cloud
    - name: "Deploy rdo"
      command: packstack --allinone
      become: yes
      become_user: root

    - name: "Store openstackrc"
      command: "cat /root/keystonerc_admin
      register: openstackrc
      become: yes
      become_user: root

- hosts: fuzzer
    - name: "Setup openstackrc"
        content: "{{ hostvars['cloud']['openstackrc'].stdout }}"
        dest: "{{ zuul_work_dir }}/.openstackrc"

    - name: "Deploy restfuzz"
      command: python setup.py install
        chdir: "{{ zuul_work_dir }}"
      become: yes
      become_user: root

    - name: "Run restfuzz"
      command: "restfuzz --target {{ hostvars['cloud']['ansible_eth0']['ipv4']['address'] }}"

The base parent from the config project manages the pre phase to copy the sources to the instances and the post phase to publish the job logs.

Nodepool drivers

This is still a work in progress but it's worth noting that Nodepool is growing a driver based design to support non-openstack providers. The primary goal is to support static node assignements, and the interface can be used to implement new providers. A driver needs to implement a Provider class to manage access to a new API, and a RequestHandler to manage resource creation.

As a Proof Of Concept, I wrote an OpenContainer driver that can spawn thin instances using RunC:

  - name: container-host-01
    driver: oci
    hypervisor: fedora.example.com
      - name: main
        max-servers: 42
          - name: fedora-26-oci
            path: /
          - name: centos-6-oci
            path: /srv/centos6
          - name: centos-7-oci
            path: /srv/centos7
          - name: rhel-7.4-oci
            path: /srv/rhel7.4

This is good news for operators and users who don't have access to an OpenStack cloud since Zuul/Nodepool may be able to use new providers such as OpenShift for example.

In conclusion, ZuulV3 brings a lot of new cool features to the table, and this article only covers a few of them. Check the documentation for more information and stay tuned for the upcoming release.

View article »

rdopkg-0.44 ChangeBlog

I'm happy to annouce version 0.44.2 of rdopkg RPM packaging automation tool has been released.

While a changelog generated from git commits is available in the original 0.44 release commit message, I think it's also worth a human readable summary of the work done by the rdopkg community for this release. I'm not sure about the format yet, so I'll start with a blog post about the changes - a ChangeBlog ;)

41 commits from 7 contributors were merged over the course of 4 months since last release with average time to land of 6 days. More stats

For more information about each change, follow the link to inspect related commit on github.

Software Factory migration

Migrate to softwarefactory-project.io


Adopt pbr for version and setup.py management

Include minor version 0.44 -> 0.44.0 as pbr dictates

Python 3 compatibility

Add some Python 3 compatibility fixes

More python 3 compatibility fixes


Add BDD feature tests using python-behave

  • Unit tests sucked for testing high level behavior so I tried an alternative. I'm quite pleased with python-behave, see one of first new-version scenarios written in Gherkin:

    Scenario: rdopkg new-version with upstream patches
        Given a distgit at Version 0.1 and Release 0.1
        Given a patches branch with 5 patches
        Given a new version 1.0.0 with 2 patches from patches branch
        When I run rdopkg new-version -lntU 1.0.0
        Then .spec file tag Version is 1.0.0
        Then .spec file tag Release is 1%{?dist}
        Then .spec file doesn't contain patches_base
        Then .spec file has 3 patches defined
        Then .spec file contains new changelog entry with 1 lines
        Then new commit was created

    It also looks reasonable on the pyton side.

Avoid test failure due to git hooks

tests: include pep8 in test-requirements.txt

tests: enable nice py.test diffs for common test code

New Features

pkgenv: display color coded hashes for branches

  • You can now easily tell the state of branches just by looking at color:

distgit: new -H/–commit-header-file option

patch: new -B/–no-bump option to only sync patches

Add support for buildsys-tags in info-tags-diff

Add options to specify user and mail in changelog entry

allow patches remote and branch to be set in git config

new-version: handle RHCEPH and RHSCON products

guess: return RH osdist for eng- dist-git branches


distgit: Use NVR for commit title for multiple changelog lines

Improve %changelog handling

Improve patches_ignore detection

Avoid prompt on non interactive console

Update yum references to dnf

Switch to pycodestyle (pep8 rename)


Use absolute path for repo_path

  • This caused trouble when using rdopkg info -l.

Use always parse_info_file in get_info

Fix output of info-tags-diff for new packages


refactor: merge legacy rdopkg.utils.exception

  • There is only one place for exceptions now \o/

core: refactor unreasonable default atomic=False

make git.config_get behave more like dict.get

specfile: fix improper naming: get_nvr, get_vr

fixed linting


document new-version's –bug argument

Update doc to reflect output change in info-tags-diff

Happy rdopkging!

View article »

Upcoming events

There's a number of upcoming events where RDO enthusiasts will be present. Mark your calendar!

Join us for Test Day!

Milestone 3 of the Pike cycle was released last week, and so it's time to test the RDO packages. Join us on Thursday and Friday of next week (August 10th and 11th) for the Pike M3 test day. We'll be on the #RDO channel on the Freenode IRC network to test, help, answer questions, and propose solutions.


We encourage you to attend your local OpenStack meetup. There's hundreds of them, all over the world, every day. We post a list of upcoming meetups to the RDO mailing list each Monday, so that you can mark your calendar. Or you can search meetup.com for events near you.

If you're going to be speaking at an OpenStack meetup group, and you'd like to have some RDO swag to take along with you, please contact me - rbowen@redhat.com - or Rain - rain@redhat.com - with your request, at least two weeks ahead of time.

And if you're a meetup organizer who is looking for speakers, we can sometimes help you with that, too.

OpenStack Days

The following OpenStack days are coming up, and each of them will have RDO enthusiasts in attendance:

If you're speaking at any of these events, please get in touch with me, so that I can help promote your presence there.

Other Events

The week of September 11 - 15th, the OpenStack PTG will be held in Denver, Colorado. At this event, project teams will meet to determine what features will be worked on for the upcoming Queens release. Rich Bowen will be there conducting video interviews, like last time, where OpenStack developers will be talking about what they worked on in Pike, and what to expect for Queens. If you'll be there, watch the announcements from the OpenStack foundation for how to sign up for an interview slot.

On October 20th, we'll be joining up with the CentOS community at the CentOS Dojo at CERN. Details of that event may be found on the CERN website. CERN is located just north of Geneva. The event is expected to have a number of RDO developers in attendance, as CERN has one of the largest OpenStack deployments in the world, running on RDO.

Other RDO events, including the many OpenStack meetups around the world, are always listed on the RDO events page. If you have an RDO-related event, please feel free to add it by submitting a pull request on Github.

View article »

Recent blog posts from the community

Here's some of the great blogs from the RDO community which you may have missed in recent weeks:

Using NFS for OpenStack (glance,nova) with selinux by Fabian Arrotin

As announced already, I was (between other things) playing with Openstack/RDO and had deployed some small openstack setup in the CentOS Infra. Then I had to look at our existing DevCloud setup. This setup was based on Opennebula running on CentOS 6, and also using Gluster as backend for the VM store. That's when I found out that Gluster isn't a valid option anymore : Gluster is was deprecated and was now even removed from Cinder. Sad as one advantage of gluster is that you could (you had to ! ) user libgfapi so that qemu-kvm process could talk directly to gluster through ligbfapi and not accessing VM images over locally mounted gluster volumes (please, don't even try to do that, through fuse).

Read more at https://arrfab.net/posts/2017/Jul/28/using-nfs-for-openstack-glancenova-with-selinux/

Nested quota models by Tim Bell

At the Boston Forum, there were many interesting discussions on models which could be used for nested quota management (https://etherpad.openstack.org/p/BOS-forum-quotas).Some of the background for the use has been explained previously in the blog (http://openstack-in-production.blogspot.fr/2016/04/resource-management-at-cern.html), but the subsequent discussions have also led to further review.

Read more at http://openstack-in-production.blogspot.com/2017/07/nested-quota-models.html

Understanding ceph-ansible in TripleO by Giulio Fidente

One of the goals for the TripleO Pike release was to introduce ceph-ansible as an alternative to puppet-ceph for the deployment of Ceph.

Read more at http://giuliofidente.com/2017/07/understanding-ceph-ansible-in-tripleo.html

Tuning for Zero Packet Loss in Red Hat OpenStack Platform – Part 3 by m4r1k

In Part 1 of this series Federico Iezzi, EMEA Cloud Architect with Red Hat covered the architecture and planning requirements to begin the journey into achieving zero packet loss in Red Hat OpenStack Platform 10 for NFV deployments. In Part 2 he went into the details around the specific tuning and parameters required. Now, in Part 3, Federico concludes the series with an example of how all this planning and tuning comes together!

Read more at http://redhatstackblog.redhat.com/2017/07/18/tuning-for-zero-packet-loss-in-red-hat-openstack-platform-part-3/

View article »

OpsTools on RDO

OpsTools for RDO


In the CentOS community there are Special Interest Groups (SIG) that focus on specific issues such as cloud, storage, virtualization, or operational tools (OpsTools). These special interest groups are created to either to create awareness or to tackle the development of that subject with focus. Among the groups there is the Operational tools group (OpsTools) that focus on

  • Performance Monitoring
  • Availability Monitoring
  • Centralized Logging

OpenStack Operational Tools

While the OpsTools are created for the CentOS community, it is also applicable and available for RDO. More information can be found at GitHub.

Centralized Logging

The centralized logging has the following components:

  • A Log Collection Agent (Fluentd)
  • A Log relay/transformer (Fluentd)
  • A Data store (Elasticsearch)
  • An API/Presentation Layer (Kibana)

With the minimum hardware requirement:

  • 8GB of Memory
  • Single Socket Xeon Class CPU
  • 500GB of Disk Space

Detailed instruction to install can be found here.

Availability Monitoring

The Availability Monitoring has the following components:

  • A Monitoring Agent (sensu)
  • A Monitoring Relay/proxy (rabbitmq)
  • A Monitoring Controller/Server (sensu)
  • An API/Presentation Layer (uchiwa)

With the minimum hardware requirement:

  • 4GB of Memory
  • Single Socket Xeon Class CPU
  • 100GB of Disk Space

Detailed instruction to install can be found here.

Performance Monitoring

The Performance Monitoring has the following components:

  • A Collection Agent (collectd)
  • A Collection Aggregator/Relay (graphite)
  • A Data Store (whisperdb)
  • An API/Presentation Layer (grafana)

With the minimum hardware requirement:

  • 4GB of Memory
  • Single Socket Xeon Class CPU
  • 500GB of Disk Space

Detailed instruction to install can be found here.

Ansible playbooks for deploying OpsTools

Besides manually install the OpsTools, there are Ansible roles and playbooks to automate the installation process and instructions can be found here.

View article »