During the year of 2017, 10 contributors managed to merge
146 commits into
3771 lines of code were added and 1975 lines deleted across 107 files.
54 unit tests were added on top of existing 32 tests - an increase of 169 % to total of 86 unit tests.
33 scenarios for 5 core rdopkg features were added in new feature tests spanning total of 228 test steps.
3 minor releases increased version from
Let's talk about the most significant improvements.
rdopkg started as a developers' tool, basically a central repository to
accumulate RPM packaging automation in a reusable manner. Quickly adding new
features was easy, but making sure existing functionality works consistently
as code is added and changed proved to be much greater challenge.
rdopkg started shifting from developers' powertool to a module used in
other automation systems, unevitable breakages started to become a problem and
prompted me to adapt development accordingly. As a first step, I tried to practice
Test-Driven Development (TDD)
as opposed to writing tests after a breakage to prevent specific case. Unit
tests helped discover and prevent various bugs introduced by new code, but
testing complex behaviors was a frustrating experience where most of
development time was spent on writing units tests for cases they weren't meant
Sounds like using a wrong tool for the job, right? And so I opened a rather
rdopkg RFE: test actions in a way that doesn't
started researching what cool kids use to develop and test python software
It would seem that
cucumber started quite a revolution of
Behavior-Driven Development (BDD)
and I really like
the Business Readable, Domain Specific Language that lets you describe
software's behaviour without detailing how that behaviour is implemented.
Gherkin serves two purposes — documentation and automated tests.
After some more research on python BDD tools, I liked
behave's implementation, documentation
and community the most so I integrated it into
rdopkg and started using
feature tests. They make it easy to describe and define expected behavior before
writing code. New features now start with feature scenario which can be
reviewed before writing any code. Covering existing behavior with
feature tests helps ensuring they are both preserved and well
defined/explained/documented. Big thanks goes to Jon Schlueter who
contributed huge number of initial feature tests for core
Here is an example of
rdopkg fix scenario:
Scenario: rdopkg fix Given a distgit When I run rdopkg fix When I add description to .spec changelog When I run rdopkg --continue Then spec file contains new changelog entry with 1 lines Then new commit was created Then rdopkg state file is not present Then last commit message is: """ foo-bar-1.2.3-3 Changelog: - Description of a change """
rdopkg change now needs to pass following
automatic gate tests before it can be merged:
- unit tests (python 2, python 3, Fedora, EPEL, CentOS)
- feature tests (python 2, python 3, Fedora, EPEL, CentOS)
- integration tests
- code style check
In other words,
master is now significantly harder to break!
Tests are managed as individual
tox targets for convenience.
Paying back the Technical Debt
I tried to write
rdopkg code with reusability and future extension in mind,
yet in one point of development with big influx of new
rdopkg approached critical mass of technical debt
where it got into spiral of new functionality breaking existing functionality
and with each fix two new bugs surfaced. This kept happening so I stopped
adding new stuff and focused on ensuring
rdopkg keeps doing what people use
it for before extending(breaking) it further. This required quite a few core
code refactors, proper integration of features that were hacked in on the
clock, as well as leveraging new tools like
software factory CI pipeline,
behave described above. But I think it was a success and
its technical debt in 2017 and is ready to face whatever community throws at
it in near and far future.
Join Software Factory project
Software Factory is an open source, software development forge with an emphasis on collaboration and ensuring code quality through Continuous Integration (CI). It is inspired by OpenStack's development workflow that has proven to be reliable for fast-changing, interdependent projects driven by large communities. Read more in Introducing Software Factory.
rdopkg leverages following Software Factory features:
- code reviews: gerrit
- Continuous Integration: zuulV3 (bye Jenkins)
- code metricts: repoXplorer
Did I meantion you can login to Software Factory using github account?
Finally, big thanks to Javier Peña, who paved the way towards Software Factory
rdopkg has been using human
for quite some time, and it proved very useful even though I often +2/+1 my
own reviews due to lack of reviewers. However, people unevitably make
mistakes. There are decent unit and feature tests now
to detect mistakes, so we fight human error with computing power and
Each review and thus each code change to
rdopkg is gated - all unit tests,
feature tests, integration tests and code style checks need to pass before
human reviewers consider accepting the change.
Instead of setting up machines and testing environments and installing requirements and waiting for tests to pass, this boring process is now automated on supported distributions and humans can focus on the changes themselves.
Integration with Fedora, EPEL and CentOS
rdopkg is now finally available directly from Fedora/EPEL repositories, so
install instructions on Fedora 25+ systems boiled down to:
dnf install rdopkg
On CentOS 7+, EPEL is needed:
yum install epel-release yum install rdopkg
Fun fact: to update Fedora
rdopkg package, I use
fedpkg clone rdopkg cd rdopkg rdopkg new-version -bN fedpkg mockbuild # testing fedpkg push fedpkg build fedpkg update
rdopkg is officially packaging itself while also being packaged by
jruzicka/rdopkg copr if you were using it previously, it is now
rdopkg documentation was cleand up, proof-read, extended with more details
and updated with latest information and links.
Feature scenarios are now available as man pages thanks to
Packaging and Distribution
Python 3 compatibility
By popular demand,
rdopkg now supports Python 3. There are Python 3 unit
python3-rdopkg RPM package.
Adopt pbr for Versioning
Most of initial patches
rdopkg was handling in the very beginning were
pbr, the OpenStack packaging meta-library,
specifically making it work on a distribution with integrated package
management and old conservative packages.
pbr was integrated into
rdopkg (well, it actually does solve
some problems aside from creating new ones) and in order to release the new
rdopkg version with
pbr on CentOS/EPEL 7, I had to disable hardcoded
pbr>=2.1.0 checks on update of
python-pymod2pkg because older version of
pbr is available from EPEL 7. I removed the check (in two different places)
as I did so many times before and it works fine.
As a tribute to all the fun I had with
distutils, here is a link
to my first
nuke bogus requirements patch
Aside from being consistent with OpenStack related projects,
strict sematic versioning that
pbr uses, which means that releases are
always going to have 3 version numbers from now on:
0.45 -> 0.45.0 1.0 -> 1.0.0
Aside from the big changes mentioned above, large amount of new feature tests and numerous not-so-exciting fixes, here is a list of changes might be worth mentioning:
rdopkg update-patchesand use alias
rdopkg pkgenvshows more information and better color coding for easy telling of a distgit state and branches setup
- preserve Change-Id when amending a commit
- allow fully unattended runs of core actions.
- commit messages created by all
rdopkgactions are now clearer, more consistent and can be overriden using
- better error messages on missing patches in all actions
- git config can be used to override patches remote, pranch, user name and email
- improved handling of
- improved handling of %changelog
- improved new/old patcehs detection
- improved packaging as suggested in Fedora review
- improved naming in
- properly handle state files
- linting cleanup and better code style checks
- python 3 support
- improve unicode support
- handle VX.Y.Z tags
- split bloated
- merge legacy
rdopkg.utils.exceptionso there is only single module for exceptions now
- refactor unreasonable default
atomic=Falseaffecting action definitions
- remove legacy