Bug 20380 - Compare two local RPMs

Bug 20270 is also fixed.

This patch allows developer to compare two local RPMs in form

    fedabipkgdiff some/place/foo.rpm another/place/bar.rpm

But, network is still needed to talk with Koji.

This patch also introduces new terms for libabigail, that is the
subject, ancillary package, and comparison half. Subject represents a
package that is subject of the ABI comparison, a subject could be a RPM
and maybe it would be a DEB or some other kind of "package". A subject
may have several ancillary packages that are used to compare the
subject.  Generally, a subject may have devel, debuginfo, or both.

	* configure.ac: add dependent mimetype module.
	* doc/manuals/fedabipkgdiff.rst: Update to add document for the
	new use case of comparing two local RPMs.
	* tests/data/Makefile.am: Include new RPMs for tests.
	* tests/data/test-fedabipkgdiff/dbus-glib/dbus-glib-0.100.2-2.fc20.x86_64.rpm:
	New RPM for running test.
	* tests/data/test-fedabipkgdiff/dbus-glib/dbus-glib-0.106-1.fc23.x86_64.rpm:
	Likewise.
	* tests/data/test-fedabipkgdiff/nss-util/nss-util-3.12.6-1.fc14.x86_64.rpm:
	Likewise.
	* tests/data/test-fedabipkgdiff/nss-util/nss-util-3.24.0-2.0.fc25.x86_64.rpm:
	Likewise.
	* tests/data/test-fedabipkgdiff/nss-util/nss-util-devel-3.24.0-2.0.fc25.x86_64.rpm:
	Likewise.
	* tests/data/test-fedabipkgdiff/test4-glib-0.100.2-2.fc20.x86_64.rpm-glib-0.106-1.fc23.x86_64.rpm-report-0.txt:
	Rename filename by adding .rpm extension.
	* tests/data/test-fedabipkgdiff/test5-same-dir-dbus-glib-0.100.2-2.fc20.x86_64--dbus-glib-0.106-1.fc23.x86_64-report-0.txt:
	New reference output for testing comparing local RPMs.
	* tests/data/test-fedabipkgdiff/test6-missing-devel-debuginfo-nss-util-3.12.6-1.fc14.x86_64--nss-util-3.24.0-2.0.fc25.x86_64-report-0.txt:
	New reference output for testing comparison without non-existent
	debuginfo or development package.
	* tests/runtestfedabipkgdiff.py.in (FEDABIPKGDIFF_TEST_SPECS):
	Rename filename for test4. Add two new test cases.
	(run_fedabipkgdiff_tests): Remove semicolon and trailing
	whitespaces.
	(main): Likewise.
	(ensure_output_dir_created): Likewise.
	* tools/fedabipkgdiff: Require some new modules.
	Fix of return code.
	(PkgInfo): Renamed to ComparisonHalf.
	(match_nvr): New method to determine if a string matches format
	of N-V-R.
	(match_nvra): New method to determine if a string matches format
	of N-V-R.A.
	(is_rpm_file): New method to guess if a file is a RPM file.
	(RPM.is_peer): New method to determine if current RPM is a peer
	of another.
	(RPM.filename): Use Koji module API to construct the filename.
	(RPM.nvra): Get nvra from filename instead of constructing
	manually that is duplicated with Koji module API.
	(RPMCollection): New class to represent a set of RPMs.
	(generate_pkg_info_pair_for_abipkgdiff): New method working as a
	generator to yeild comparison halves for running abipkgdiff.
	(Brew.getRPM): Fix string format with incorrect argument.
	(Brew.select_rpms_from_a_build): Return instance of
	RPMCollection.
	(abipkgdiff): If there is no debuginfo or development package,
	just ignore it and leave a warning. If --error-on-warning is
	specified, raise an exception instead. Arguments are modified
	to represent the new name ComparisonHalf, and relative docstring
	is also updated.
	(magic_construct): Removed.
	(run_abipkgdiff): Rewrite.
	(make_rpms_usable_for_abipkgdiff): Removed.
	(diff_local_rpm_with_latest_rpm_from_koji): Rewrite by using
	RPMCollection.
	(diff_latest_rpms_based_on_distros): Likewise.
	(diff_two_nvras_from_koji): Likewise.
	(diff_from_two_rpm_files): New method to compare two local RPMs.
	(build_commandline_args_parser): Add new option
	--error-on-warning.
	(main): Add support to compare local RPMs.

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Signed-off-by: Dodji Seketeli <dodji@redhat.com>
This commit is contained in:
Chenxiong Qi 2016-08-11 21:48:00 +08:00 committed by Dodji Seketeli
parent 85318068cd
commit 783099cd25
13 changed files with 450 additions and 201 deletions

View File

@ -335,7 +335,7 @@ if test x$CHECK_DEPS_FOR_FEDABIPKGDIFF = xyes; then
REQUIRED_PYTHON_MODULES_FOR_FEDABIPKGDIFF="\
argparse logging os re subprocess sys urlparse \
xdg koji mock rpm imp tempfile"
xdg koji mock rpm imp tempfile mimetypes"
if test x$ENABLE_FEDABIPKGDIFF != xno; then
AX_CHECK_PYTHON_MODULES([$REQUIRED_PYTHON_MODULES_FOR_FEDABIPKGDIFF],

View File

@ -180,7 +180,15 @@ Below are some usage examples currently supported by
$ fedabipkgdiff --from fc23 ./httpd-2.4.18-2.fc24.x86_64.rpm
2. Compare the ABI of binaries in the latest build of the ``httpd``
2. Compare the ABI of binaries in two local packages.
Suppose you have built two versions of package httpd, and you want to see
what ABI differences between these two versions of RPM files. The
commandline invocation would be::
$ fedabipkgdiff path/to/httpd-2.4.23-3.fc23.x86_64.rpm another/path/to/httpd-2.4.23-4.fc24.x86_64.rpm
3. Compare the ABI of binaries in the latest build of the ``httpd``
package in ``Fedora 23`` against the ABI of the binaries in the
latest build of the same package in 24.
@ -192,7 +200,7 @@ Below are some usage examples currently supported by
$ fedabipkgdiff --from fc23 --to fc24 httpd
3. Compare the ABI of binaries of two builds of the ``httpd``
4. Compare the ABI of binaries of two builds of the ``httpd``
package, designated their versions and releases.
If we want to do perform the ABI comparison for all the processor
@ -207,7 +215,7 @@ Below are some usage examples currently supported by
$ fedabipkgdiff httpd-2.8.14.fc23.x86_64 httpd-2.8.14.fc24.x86_64
4. If the use wants to also compare the sub-packages of a given
5. If the use wants to also compare the sub-packages of a given
package, she can use the --all-subpackages option. The first
command of the previous example would thus look like: ::

View File

@ -1271,4 +1271,10 @@ test-default-supprs/dirpkg-1-dir1/libobj-v0.so \
test-default-supprs/dirpkg-1-dir1/obj-v0.cc \
test-default-supprs/dirpkg-1-dir2/dir.abignore \
test-default-supprs/dirpkg-1-dir2/libobj-v0.so \
test-default-supprs/dirpkg-1-dir2/obj-v0.cc
test-default-supprs/dirpkg-1-dir2/obj-v0.cc \
\
test-fedabipkgdiff/dbus-glib/dbus-glib-0.100.2-2.fc20.x86_64.rpm \
test-fedabipkgdiff/dbus-glib/dbus-glib-0.106-1.fc23.x86_64.rpm \
test-fedabipkgdiff/nss-util/nss-util-devel-3.24.0-2.0.fc25.x86_64.rpm \
test-fedabipkgdiff/nss-util/nss-util-3.12.6-1.fc14.x86_64.rpm \
test-fedabipkgdiff/nss-util/nss-util-3.24.0-2.0.fc25.x86_64.rpm

View File

@ -0,0 +1,26 @@
Comparing the ABI of binaries between dbus-glib-0.100.2-2.fc20.x86_64.rpm and dbus-glib-0.106-1.fc23.x86_64.rpm:
================ changes of 'dbus-binding-tool'===============
Functions changes summary: 2 Removed, 0 Changed, 0 Added functions
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
2 Removed functions:
'function BaseInfo* base_info_ref(BaseInfo*)' {base_info_ref}
'function void base_info_unref(BaseInfo*)' {base_info_unref}
================ end of changes of 'dbus-binding-tool'===============
================ changes of 'libdbus-glib-1.so.2.2.2'===============
Functions changes summary: 0 Removed, 0 Changed, 2 Added functions
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
2 Added functions:
'function DBusGConnection* dbus_g_connection_open_private(const gchar*, GMainContext*, GError**)' {dbus_g_connection_open_private}
'function DBusGConnection* dbus_g_method_invocation_get_g_connection(DBusGMethodInvocation*)' {dbus_g_method_invocation_get_g_connection}
================ end of changes of 'libdbus-glib-1.so.2.2.2'===============

View File

@ -0,0 +1,29 @@
Comparing the ABI of binaries between dbus-glib-0.100.2-2.fc20.x86_64.rpm and dbus-glib-0.106-1.fc23.x86_64.rpm:
================ changes of 'dbus-binding-tool'===============
Functions changes summary: 0 Removed, 0 Changed, 0 Added function
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
Function symbols changes summary: 2 Removed, 0 Added function symbols not referenced by debug info
Variable symbols changes summary: 0 Removed, 0 Added variable symbol not referenced by debug info
2 Removed function symbols not referenced by debug info:
base_info_ref
base_info_unref
================ end of changes of 'dbus-binding-tool'===============
================ changes of 'libdbus-glib-1.so.2.2.2'===============
Functions changes summary: 0 Removed, 0 Changed, 0 Added function
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
Function symbols changes summary: 0 Removed, 2 Added function symbols not referenced by debug info
Variable symbols changes summary: 0 Removed, 0 Added variable symbol not referenced by debug info
2 Added function symbols not referenced by debug info:
dbus_g_connection_open_private
dbus_g_method_invocation_get_g_connection
================ end of changes of 'libdbus-glib-1.so.2.2.2'===============

View File

@ -0,0 +1,51 @@
Comparing the ABI of binaries between nss-util-3.12.6-1.fc14.x86_64.rpm and nss-util-3.24.0-2.0.fc25.x86_64.rpm:
================ changes of 'libnssutil3.so'===============
Functions changes summary: 0 Removed, 0 Changed, 0 Added function
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
Function symbols changes summary: 0 Removed, 37 Added function symbols not referenced by debug info
Variable symbols changes summary: 0 Removed, 0 Added variable symbol not referenced by debug info
37 Added function symbols not referenced by debug info:
NSSUTIL_ArgDecodeNumber@@NSSUTIL_3.14
NSSUTIL_ArgFetchValue@@NSSUTIL_3.14
NSSUTIL_ArgGetLabel@@NSSUTIL_3.14
NSSUTIL_ArgGetParamValue@@NSSUTIL_3.14
NSSUTIL_ArgHasFlag@@NSSUTIL_3.14
NSSUTIL_ArgIsBlank@@NSSUTIL_3.14
NSSUTIL_ArgParseCipherFlags@@NSSUTIL_3.14
NSSUTIL_ArgParseModuleSpec@@NSSUTIL_3.14
NSSUTIL_ArgParseModuleSpecEx@@NSSUTIL_3.21
NSSUTIL_ArgParseSlotFlags@@NSSUTIL_3.14
NSSUTIL_ArgParseSlotInfo@@NSSUTIL_3.14
NSSUTIL_ArgReadLong@@NSSUTIL_3.14
NSSUTIL_ArgSkipParameter@@NSSUTIL_3.14
NSSUTIL_ArgStrip@@NSSUTIL_3.14
NSSUTIL_DoModuleDBFunction@@NSSUTIL_3.14
NSSUTIL_DoubleEscape@@NSSUTIL_3.14
NSSUTIL_DoubleEscapeSize@@NSSUTIL_3.14
NSSUTIL_Escape@@NSSUTIL_3.14
NSSUTIL_EscapeSize@@NSSUTIL_3.14
NSSUTIL_GetVersion@@NSSUTIL_3.13
NSSUTIL_MkModuleSpec@@NSSUTIL_3.14
NSSUTIL_MkNSSString@@NSSUTIL_3.14
NSSUTIL_MkSlotString@@NSSUTIL_3.14
NSSUTIL_Quote@@NSSUTIL_3.14
NSSUTIL_QuoteSize@@NSSUTIL_3.14
NSS_InitializePRErrorTable@@NSSUTIL_3.13
PORT_DestroyCheapArena@@NSSUTIL_3.24
PORT_InitCheapArena@@NSSUTIL_3.24
PORT_RegExpSearch@@NSSUTIL_3.12.7
SECITEM_AllocArray@@NSSUTIL_3.15
SECITEM_DupArray@@NSSUTIL_3.15
SECITEM_FreeArray@@NSSUTIL_3.15
SECITEM_ReallocItemV2@@NSSUTIL_3.15
SECITEM_ZfreeArray@@NSSUTIL_3.15
_NSSUTIL_EvaluateConfigDir@@NSSUTIL_3.14
_NSSUTIL_GetSecmodName@@NSSUTIL_3.14
_SGN_VerifyPKCS1DigestInfo@@NSSUTIL_3.17.1
================ end of changes of 'libnssutil3.so'===============

View File

@ -49,7 +49,7 @@ OUTPUT_DIR = '@abs_top_builddir@/tests/output/test-fedabipkgdiff'
# the first element of the tuple) the result of the comparison is
# going to be compared against the reference output file, and both
# must be equal.
#
#
# If a user wants to add a new test, she must add a new tuple to the
# variable below, and also add a new reference output to the source
# distribution.
@ -72,8 +72,24 @@ FEDABIPKGDIFF_TEST_SPECS = [
(['dbus-glib-0.100.2-2.fc20.i686', 'dbus-glib-0.106-1.fc23.i686'],
'data/test-fedabipkgdiff/test3-dbus-glib-0.100.2-2.fc20.i686--dbus-glib-0.106-1.fc23.i686-report-0.txt',
'output/test-fedabipkgdiff/test3-dbus-glib-0.100.2-2.fc20.i686--dbus-glib-0.106-1.fc23.i686-report-0.txt'),
([os.path.join(INPUT_DIR, 'packages/dbus-glib/0.100.2/2.fc20/x86_64/dbus-glib-0.100.2-2.fc20.x86_64.rpm'),
os.path.join(INPUT_DIR, 'packages/dbus-glib/0.106/1.fc23/x86_64/dbus-glib-0.106-1.fc23.x86_64.rpm')],
'data/test-fedabipkgdiff/test4-glib-0.100.2-2.fc20.x86_64.rpm-glib-0.106-1.fc23.x86_64.rpm-report-0.txt',
'output/test-fedabipkgdiff/test4-glib-0.100.2-2.fc20.x86_64.rpm-glib-0.106-1.fc23.x86_64.rpm-report-0.txt'),
([os.path.join(INPUT_DIR, 'dbus-glib/dbus-glib-0.100.2-2.fc20.x86_64.rpm'),
os.path.join(INPUT_DIR, 'dbus-glib/dbus-glib-0.106-1.fc23.x86_64.rpm')],
'data/test-fedabipkgdiff/test5-same-dir-dbus-glib-0.100.2-2.fc20.x86_64--dbus-glib-0.106-1.fc23.x86_64-report-0.txt',
'output/test-fedabipkgdiff/test5-same-dir-dbus-glib-0.100.2-2.fc20.x86_64--dbus-glib-0.106-1.fc23.x86_64-report-0.txt'),
([os.path.join(INPUT_DIR, 'nss-util/nss-util-3.12.6-1.fc14.x86_64.rpm'),
os.path.join(INPUT_DIR, 'nss-util/nss-util-3.24.0-2.0.fc25.x86_64.rpm')],
'data/test-fedabipkgdiff/test6-missing-devel-debuginfo-nss-util-3.12.6-1.fc14.x86_64--nss-util-3.24.0-2.0.fc25.x86_64-report-0.txt',
'output/test-fedabipkgdiff/test6-missing-devel-debuginfo-nss-util-3.12.6-1.fc14.x86_64--nss-util-3.24.0-2.0.fc25.x86_64-report-0.txt'),
]
def ensure_output_dir_created():
'''Create output dir if it's not already created.'''
@ -83,7 +99,8 @@ def ensure_output_dir_created():
pass
if not os.path.exists(OUTPUT_DIR):
sys.exit(1);
sys.exit(1)
def run_fedabipkgdiff_tests():
"""Run the fedabipkgdiff tests
@ -102,7 +119,7 @@ def run_fedabipkgdiff_tests():
FEDABIPKGDIFF_TEST_SPECS succeed.
"""
result = True;
result = True
for args, reference_report_path, output_path in FEDABIPKGDIFF_TEST_SPECS:
reference_report_path = os.path.join(TEST_SRC_DIR, reference_report_path)
output_path = os.path.join(TEST_BUILD_DIR, output_path)
@ -119,12 +136,13 @@ def run_fedabipkgdiff_tests():
else:
diffcmd = ['diff', '-u', reference_report_path, output_path]
return_code = subprocess.call(diffcmd)
if return_code:
if return_code:
sys.stderr.write('fedabipkgdiff test failed for ' +
reference_report_path + '\n')
result &= not return_code
return result;
return result
def main():
"""The main entry point of this program.
@ -135,10 +153,11 @@ def main():
"""
ensure_output_dir_created()
result = 0;
result = 0
result = run_fedabipkgdiff_tests()
if not result:
return result;
return result
if __name__ == '__main__':
exit_code = main()

View File

@ -23,14 +23,16 @@
# Author: Chenxiong Qi
import argparse
import glob
import logging
import mimetypes
import os
import re
import subprocess
import sys
from collections import namedtuple
from itertools import groupby
from itertools import chain
import xdg.BaseDirectory
@ -86,17 +88,23 @@ ABIDIFF_ABI_CHANGE = 1 << 2
# /path/to/package1.rpm \
# /path/to/package2.rpm
#
# PkgInfo is a three-elements tuple in format
# ComparisonHalf is a three-elements tuple in format
#
# (/path/to/package1.rpm, /path/to/package1-debuginfo.rpm /path/to/package1-devel.rpm)
#
# - the first element is the subject representing the package to compare.
# - the rest are ancillary packages used for the comparison. So, the second one
# is the debuginfo package, and the last one is the package containing API of
# the ELF shared libraries carried by subject.
#
# So, before calling abipkgdiff, fedabipkgdiff must prepare and pass
# the following information
#
# (/path/to/package1.rpm, /path/to/package1-debuginfo.rpm /path/to/package1-devel.rpm)
# (/path/to/package2.rpm, /path/to/package2-debuginfo.rpm /path/to/package1-devel.rpm)
#
PkgInfo = namedtuple('PkgInfo', 'package debuginfo_package devel_package')
ComparisonHalf = namedtuple('ComparisonHalf',
['subject', 'ancillary_debug', 'ancillary_devel'])
global_config = None
@ -106,8 +114,7 @@ session = None
# There is no way to configure the log format so far. I hope I would have time
# to make it available so that if fedabipkgdiff is scheduled and run by some
# service, the logs logged into log file is muc usable.
logging.basicConfig(format='[%(levelname)s] %(message)s',
level=logging.CRITICAL)
logging.basicConfig(format='[%(levelname)s] %(message)s', level=logging.CRITICAL)
logger = logging.getLogger(os.path.basename(__file__))
@ -155,6 +162,22 @@ def is_distro_valid(distro):
return re.match(r'^(fc|el)\d{1,2}$', distro) is not None
def match_nvr(s):
"""Determine if a string is a N-V-R"""
return re.match(r'^([^/]+)-(.+)-(.+)$', s) is not None
def match_nvra(s):
"""Determine if a string is a N-V-R.A"""
return re.match(r'^([^/]+)-(.+)-(.+)\.(.+)$', s) is not None
def is_rpm_file(filename):
"""Return if a file is a RPM"""
return os.path.isfile(filename) and \
mimetypes.guess_type(filename)[0] == 'application/x-rpm'
def cmp_nvr(left, right):
"""Compare function for sorting a sequence of NVRs
@ -233,13 +256,20 @@ class RPM(object):
else:
raise AttributeError('No attribute name {0}'.format(name))
def is_peer(self, another_rpm):
"""Determine if this is the peer of a given rpm"""
return self.name == another_rpm.name and \
self.arch == another_rpm.arch and \
self.release != another_rpm.release
@property
def nvra(self):
"""Return a RPM's N-V-R-A representation
An example: libabigail-1.0-0.8.rc4.1.fc23.x86_64
"""
return '%(name)s-%(version)s-%(release)s.%(arch)s' % self.rpm_info
nvra, _ = os.path.splitext(self.filename)
return nvra
@property
def filename(self):
@ -247,7 +277,7 @@ class RPM(object):
An example: libabigail-1.0-0.8.rc4.1.fc23.x86_64.rpm
"""
return '{0}.rpm'.format(self.nvra)
return os.path.basename(pathinfo.rpm(self.rpm_info))
@property
def is_debuginfo(self):
@ -338,6 +368,152 @@ class LocalRPM(RPM):
return self._find_rpm(filename)
class RPMCollection(object):
"""Collection of RPMs
This is a simple collection containing RPMs collected from a
directory on the local filesystem or retrieved from Koji.
A collection can contain one or more sets of RPMs. Each set of
RPMs being for a particular architecture.
For a given architecture, a set of RPMs is made of one RPM and its
ancillary RPMs. An ancillary RPM is either a debuginfo RPM or a
devel RPM.
So a given RPMCollection would (informally) look like:
{
i686 => {foo.i686.rpm, foo-debuginfo.i686.rpm, foo-devel.i686.rpm}
x86_64 => {foo.x86_64.rpm, foo-debuginfo.x86_64.rpm, foo-devel.x86_64.rpm,}
}
"""
def __init__(self, rpms=None):
# Mapping from arch to a list of rpm_infos.
# Note that *all* RPMs of the collections are present in this
# map; that is the RPM to consider and its ancillary RPMs.
self.rpms = {}
# Mapping from arch to another mapping containing index of debuginfo
# and development package
# e.g.
# self.ancillary_rpms = {'i686', {'debuginfo': foo-debuginfo.rpm,
# 'devel': foo-devel.rpm}}
self.ancillary_rpms = {}
if rpms:
map(self.add, rpms)
@classmethod
def gather_from_dir(cls, rpm_file, all_rpms=None):
"""Gather RPM collection from local directory"""
dir_name = os.path.dirname(os.path.abspath(rpm_file))
filename = os.path.basename(rpm_file)
nvra = koji.parse_NVRA(filename)
rpm_files = glob.glob(os.path.join(
dir_name, '*-%(version)s-%(release)s.%(arch)s.rpm' % nvra))
rpm_col = cls()
if all_rpms:
selector = lambda rpm: True
else:
selector = lambda rpm: local_rpm.is_devel or \
local_rpm.is_debuginfo or local_rpm.filename == filename
found_debuginfo = 1
for rpm_file in rpm_files:
local_rpm = LocalRPM(rpm_file)
if local_rpm.is_debuginfo:
found_debuginfo <<= 1
if found_debuginfo == 4:
raise RuntimeError(
'Found more than one debuginfo package in '
'this directory. At the moment, fedabipkgdiff '
'is not able to deal with this case. '
'Please create two separate directories and '
'put an RPM and its ancillary debuginfo and '
'devel RPMs in each directory.')
if selector(local_rpm):
rpm_col.add(local_rpm)
return rpm_col
def add(self, rpm):
"""Add a RPM into this collection"""
self.rpms.setdefault(rpm.arch, []).append(rpm)
devel_debuginfo_default = {'debuginfo': None, 'devel': None}
if rpm.is_debuginfo:
self.ancillary_rpms.setdefault(
rpm.arch, devel_debuginfo_default)['debuginfo'] = rpm
if rpm.is_devel:
self.ancillary_rpms.setdefault(
rpm.arch, devel_debuginfo_default)['devel'] = rpm
def rpms_iter(self, arches=None, default_behavior=True):
"""Iterator of RPMs to go through RPMs with specific arches"""
arches = self.rpms.keys()
arches.sort()
for arch in arches:
for _rpm in self.rpms[arch]:
yield _rpm
def get_sibling_debuginfo(self, rpm):
"""Get sibling debuginfo package of given rpm"""
if rpm.arch not in self.ancillary_rpms:
return None
return self.ancillary_rpms[rpm.arch].get('debuginfo')
def get_sibling_devel(self, rpm):
"""Get sibling devel package of given rpm"""
if rpm.arch not in self.ancillary_rpms:
return None
return self.ancillary_rpms[rpm.arch].get('devel')
def get_peer_rpm(self, rpm):
"""Get peer rpm of rpm from this collection"""
for _rpm in self.rpms[rpm.arch]:
if _rpm.is_peer(rpm):
return _rpm
return None
def generate_comparison_halves(rpm_col1, rpm_col2):
"""Iterate RPM collection and peer's to generate comparison halves"""
for _rpm in rpm_col1.rpms_iter():
if _rpm.is_debuginfo:
continue
if _rpm.is_devel and not global_config.check_all_subpackages:
continue
rpm2 = rpm_col2.get_peer_rpm(_rpm)
if rpm2 is None:
logger.warning('Peer RPM of {0} is not found.'.format(_rpm.filename))
continue
debuginfo1 = rpm_col1.get_sibling_debuginfo(_rpm)
devel1 = rpm_col1.get_sibling_devel(_rpm)
debuginfo2 = rpm_col2.get_sibling_debuginfo(rpm2)
devel2 = rpm_col2.get_sibling_devel(rpm2)
yield (ComparisonHalf(subject=_rpm,
ancillary_debug=debuginfo1,
ancillary_devel=devel1),
ComparisonHalf(subject=rpm2,
ancillary_debug=debuginfo2,
ancillary_devel=devel2))
class Brew(object):
"""Interface to Koji XMLRPC API with enhancements specific to fedabipkgdiff
@ -425,7 +601,7 @@ class Brew(object):
"""
rpm = self.session.getRPM(rpminfo)
if rpm is None:
raise RpmNotFound('Cannot find RPM {0}'.format(args[0]))
raise RpmNotFound('Cannot find RPM {0}'.format(rpminfo))
return rpm
@log_call
@ -596,7 +772,7 @@ class Brew(object):
By default, fedabipkgdiff requires the RPM package, as well as
its associated debuginfo and devel packages. These three
packages are selected, and noarch and src are excluded.
:param int build_id: from which build to select rpms.
:param str package_name: which rpm to select that matches this name.
:param arches: which arches to select. If arches omits, rpms with all
@ -623,7 +799,7 @@ class Brew(object):
rpm_infos = self.listRPMs(buildID=build_id,
arches=arches,
selector=selector)
return [RPM(rpm_info) for rpm_info in rpm_infos]
return RPMCollection((RPM(rpm_info) for rpm_info in rpm_infos))
@log_call
def get_latest_built_rpms(self, package_name, distro, arches=None):
@ -722,7 +898,7 @@ def build_path_to_abipkgdiff():
@log_call
def abipkgdiff(pkg_info1, pkg_info2):
def abipkgdiff(cmp_half1, cmp_half2):
"""Run abipkgdiff against found two RPM packages
Construct and execute abipkgdiff to get ABI diff
@ -735,30 +911,68 @@ def abipkgdiff(pkg_info1, pkg_info2):
called synchronously. fedabipkgdiff does not return until underlying
abipkgdiff finishes.
:param PkgInfo pkg_info1: the first package information provided for
abipkgdiff package1 paramter.
:param PkgInfo pkg_info2: the second package information provided for
abipkgdiff package2 paramter.
:param ComparisonHalf cmp_half1: the first comparison half.
:param ComparisonHalf cmp_half2: the second comparison half.
:return: return code of underlying abipkgdiff execution.
:rtype: int
"""
abipkgdiff_tool = build_path_to_abipkgdiff()
devel_pkg1 = '' if global_config.no_devel_pkg else \
'--devel-pkg1 {0}'.format(pkg_info1.devel_package.downloaded_file)
devel_pkg2 = '' if global_config.no_devel_pkg else \
'--devel-pkg2 {0}'.format(pkg_info2.devel_package.downloaded_file)
if global_config.no_devel_pkg:
devel_pkg1 = ''
devel_pkg2 = ''
else:
if cmp_half1.ancillary_devel is None:
msg = 'Development package for {0} does not exist.'.format(cmp_half1.subject.filename)
if global_config.error_on_warning:
raise RuntimeError(msg)
else:
devel_pkg1 = ''
logger.warning('{0} Ignored.'.format(msg))
else:
devel_pkg1 = '--devel-pkg1 {0}'.format(cmp_half1.ancillary_devel.downloaded_file)
if cmp_half2.ancillary_devel is None:
msg = 'Development package for {0} does not exist.'.format(cmp_half2.subject.filename)
if global_config.error_on_warning:
raise RuntimeError(msg)
else:
devel_pkg2 = ''
logger.warning('{0} Ignored.'.format(msg))
else:
devel_pkg2 = '--devel-pkg2 {0}'.format(cmp_half2.ancillary_devel.downloaded_file)
if cmp_half1.ancillary_debug is None:
msg = 'Debuginfo package for {0} does not exist.'.format(cmp_half1.subject.filename)
if global_config.error_on_warning:
raise RuntimeError(msg)
else:
debuginfo_pkg1 = ''
logger.warning('{0} Ignored.'.format(msg))
else:
debuginfo_pkg1 = '--d1 {0}'.format(cmp_half1.ancillary_debug.downloaded_file)
if cmp_half2.ancillary_debug is None:
msg = 'Debuginfo package for {0} does not exist.'.format(cmp_half2.subject.filename)
if global_config.error_on_warning:
raise RuntimeError(msg)
else:
debuginfo_pkg2 = ''
logger.warning('{0} Ignored.'.format(msg))
else:
debuginfo_pkg2 = '--d2 {0}'.format(cmp_half2.ancillary_debug.downloaded_file)
cmd = [
abipkgdiff_tool,
'--show-identical-binaries' if global_config.show_identical_binaries else '',
'--no-default-suppression' if global_config.no_default_suppr else '',
'--dso-only' if global_config.dso_only else '',
'--d1', pkg_info1.debuginfo_package.downloaded_file,
'--d2', pkg_info2.debuginfo_package.downloaded_file,
debuginfo_pkg1,
debuginfo_pkg2,
devel_pkg1,
devel_pkg2,
pkg_info1.package.downloaded_file,
pkg_info2.package.downloaded_file,
cmp_half1.subject.downloaded_file,
cmp_half2.subject.downloaded_file,
]
cmd = filter(lambda s: s != '', cmd)
@ -769,7 +983,7 @@ def abipkgdiff(pkg_info1, pkg_info2):
logger.debug('Run: %s', ' '.join(cmd))
print 'Comparing the ABI of binaries between {0} and {1}:'.format(
pkg_info1.package.filename, pkg_info2.package.filename)
cmp_half1.subject.filename, cmp_half2.subject.filename)
print
proc = subprocess.Popen(' '.join(cmd), shell=True,
@ -777,8 +991,7 @@ def abipkgdiff(pkg_info1, pkg_info2):
stdout, stderr = proc.communicate()
is_ok = proc.returncode == ABIDIFF_OK
is_internal_error = proc.returncode & ABIDIFF_ERROR or \
proc.returncode & ABIDIFF_USAGE_ERROR
is_internal_error = proc.returncode & ABIDIFF_ERROR or proc.returncode & ABIDIFF_USAGE_ERROR
has_abi_change = proc.returncode & ABIDIFF_ABI_CHANGE
if is_internal_error:
@ -789,83 +1002,22 @@ def abipkgdiff(pkg_info1, pkg_info2):
return proc.returncode
def magic_construct(rpms):
"""Construct RPMs into a magic structure
Convert list of
foo-1.0-1.fc22.i686
foo-debuginfo-1.0-1.fc22.i686
foo-devel-1.0-1.fc22.i686
to list of
(foo-1.0-1.fc22.i686, foo-debuginfo-1.0-1.fc22.i686,
foo-devel-1.0-1.fc22.i686)
and if to check all subpackages
(foo-devel-1.0-1.fc22.i686, foo-debuginfo-1.0-1.fc22.i686,
foo-devel-1.0-1.fc22.i686)
:param rpms: a sequence of RPM packages.
:type rpms: list or tuple
:return: list of two-element tuple where the first element is a RPM package
and the second one is the debuginfo package.
:rtype: list
"""
debuginfo = None
devel_package = None
packages = []
for rpm in rpms:
if rpm.is_debuginfo:
debuginfo = rpm
elif rpm.is_devel:
devel_package = rpm
if global_config.check_all_subpackages:
packages.append(rpm)
else:
packages.append(rpm)
return [PkgInfo(package, debuginfo, devel_package) for package in packages]
@log_call
def run_abipkgdiff(pkg1_infos, pkg2_infos):
def run_abipkgdiff(rpm_col1, rpm_col2):
"""Run abipkgdiff
If one of the executions finds ABI differences, the return code is the
return code from abipkgdiff.
:param dict pkg1_infos: a mapping from arch to list of RPMs
:param RPMCollection rpm_col1: a collection of RPMs
:param RPMCollection rpm_col2: same as rpm_col1
:return: exit code of the last non-zero returned from underlying abipkgdiff
:rtype: number
:rtype: int
"""
arches = pkg1_infos.keys()
arches.sort()
return_code = 0
for arch in arches:
pkg_infos = magic_construct(pkg1_infos[arch])
for pkg_info in pkg_infos:
rpms = pkg2_infos[arch]
package = [rpm for rpm in rpms
if rpm.name == pkg_info.package.name][0]
debuginfo = [rpm for rpm in rpms
if rpm.name == pkg_info.debuginfo_package.name][0]
devel_package = [rpm for rpm in rpms
if rpm.name == pkg_info.devel_package.name][0]
ret = abipkgdiff(pkg_info,
PkgInfo(package=package,
debuginfo_package=debuginfo,
devel_package=devel_package))
if ret > 0:
return_code = ret
return return_code
return_codes = [
abipkgdiff(cmp_half1, cmp_half2) for cmp_half1, cmp_half2
in generate_comparison_halves(rpm_col1, rpm_col2)]
return max(return_codes)
@log_call
@ -892,72 +1044,13 @@ def diff_local_rpm_with_latest_rpm_from_koji():
raise ValueError('{0} does not exist.'.format(local_rpm_file))
local_rpm = LocalRPM(local_rpm_file)
local_debuginfo = local_rpm.find_debuginfo()
local_devel = local_rpm.find_devel()
if local_debuginfo is None:
raise ValueError(
'debuginfo rpm {0} does not exist.'.format(local_debuginfo))
if local_devel is None and global_config.no_devel_pkg is not None:
raise ValueError(
'development package {0} does not exist.'.format(local_devel))
rpm_col1 = session.get_latest_built_rpms(local_rpm.name,
from_distro,
arches=local_rpm.arch)
rpm_col2 = RPMCollection.gather_from_dir(local_rpm_file)
rpms = session.get_latest_built_rpms(local_rpm.name,
from_distro,
arches=local_rpm.arch)
download_rpms(rpms)
pkg_infos = make_rpms_usable_for_abipkgdiff(rpms)
rpms = pkg_infos.values()[0]
package, debuginfo, devel_package = sorted(rpms, key=lambda rpm: rpm.name)
return abipkgdiff(PkgInfo(package=package,
debuginfo_package=debuginfo,
devel_package=devel_package),
PkgInfo(package=local_rpm,
debuginfo_package=local_debuginfo,
devel_package=local_devel))
@log_call
def make_rpms_usable_for_abipkgdiff(rpms):
"""Prepare package information structure for running abipkgdiff
So far, RPMs input to this method are queried from Koji and abipkgdiff will
run against these RPMs. For convenience, these RPMs should be restructured
into a mapping so that subsequent operations could easily find RPMs from
arch.
For example, input RPMs are
[RPM(arch='x86_64', name='httpd'),
RPM(arch='i686', name='httpd'),
RPM(arch='x86_64', name='httpd-devel'),
RPM(arch='i686', name='http-debuginfo'),
RPM(arch='x86_64', name='httpd-debuginfo'),
]
it is converted into mapping
{
'x86_64': [RPM(arch='x86_64', name='httpd'),
RPM(arch='x86_64', name='httpd-devel'),
RPM(arch='x86_64', name='httpd-debuginfo')],
'i686': [RPM(arch='i686', name='httpd'),
RPM(arch='i686', name='http-debuginfo')],
}
The order RPMs in the mapping is unpredictable. So, if they must be in a
particular order, caller is responsible for this.
:param list rpms: a list of RPMs
:return: a mapping from an arch to corresponding list of RPMs
:rtype: dict
"""
result = {}
rpms_iter = groupby(sorted(rpms, key=lambda rpm: rpm.arch),
key=lambda item: item.arch)
for arch, rpms in rpms_iter:
result[arch] = list(rpms)
return result
download_rpms(rpm_col1.rpms_iter())
return run_abipkgdiff(rpm_col1, rpm_col2)
@log_call
@ -981,17 +1074,14 @@ def diff_latest_rpms_based_on_distros():
package_name = global_config.NVR[0]
rpms = session.get_latest_built_rpms(package_name,
distro=global_config.from_distro)
download_rpms(rpms)
pkg1_infos = make_rpms_usable_for_abipkgdiff(rpms)
rpm_col1 = session.get_latest_built_rpms(package_name,
distro=global_config.from_distro)
rpm_col2 = session.get_latest_built_rpms(package_name,
distro=global_config.to_distro)
rpms = session.get_latest_built_rpms(package_name,
distro=global_config.to_distro)
download_rpms(rpms)
pkg2_infos = make_rpms_usable_for_abipkgdiff(rpms)
download_rpms(chain(rpm_col1.rpms_iter(), rpm_col2.rpms_iter()))
return run_abipkgdiff(pkg1_infos, pkg2_infos)
return run_abipkgdiff(rpm_col1, rpm_col2)
@log_call
@ -1028,20 +1118,27 @@ def diff_two_nvras_from_koji():
right_rpm['arch'])
build_id = session.get_rpm_build_id(*params1)
rpms = session.select_rpms_from_a_build(
rpm_col1 = session.select_rpms_from_a_build(
build_id, params1[0], arches=params1[3],
select_subpackages=global_config.check_all_subpackages)
download_rpms(rpms)
pkg1_infos = make_rpms_usable_for_abipkgdiff(rpms)
build_id = session.get_rpm_build_id(*params2)
rpms = session.select_rpms_from_a_build(
rpm_col2 = session.select_rpms_from_a_build(
build_id, params2[0], arches=params2[3],
select_subpackages=global_config.check_all_subpackages)
download_rpms(rpms)
pkg2_infos = make_rpms_usable_for_abipkgdiff(rpms)
return run_abipkgdiff(pkg1_infos, pkg2_infos)
download_rpms(chain(rpm_col1.rpms_iter(), rpm_col2.rpms_iter()))
return run_abipkgdiff(rpm_col1, rpm_col2)
@log_call
def diff_from_two_rpm_files(from_rpm_file, to_rpm_file):
"""Diff two RPM files"""
rpm_col1 = RPMCollection.gather_from_dir(from_rpm_file)
rpm_col2 = RPMCollection.gather_from_dir(to_rpm_file)
download_rpms(chain(rpm_col1.rpms_iter(), rpm_col2.rpms_iter()))
return run_abipkgdiff(rpm_col1, rpm_col2)
def build_commandline_args_parser():
@ -1052,8 +1149,8 @@ def build_commandline_args_parser():
parser.add_argument(
'NVR',
nargs='*',
help='RPM package N-V-R, N-V-R-A, N, or a local RPM '
'file name with relative or absolute path.')
help='RPM package N-V-R, N-V-R-A, N, or local RPM '
'file names with relative or absolute path.')
parser.add_argument(
'--dry-run',
required=False,
@ -1142,6 +1239,12 @@ def build_commandline_args_parser():
action='store_true',
dest='show_identical_binaries',
help='Show information about binaries whose ABI are identical')
parser.add_argument(
'--error-on-warning',
required=False,
action='store_true',
dest='error_on_warning',
help='Raise error instead of warning')
return parser
@ -1166,26 +1269,33 @@ def main():
if global_config.from_distro and global_config.to_distro is None and \
global_config.NVR:
returncode = diff_local_rpm_with_latest_rpm_from_koji()
return diff_local_rpm_with_latest_rpm_from_koji()
elif global_config.from_distro and global_config.to_distro and \
if global_config.from_distro and global_config.to_distro and \
global_config.NVR:
returncode = diff_latest_rpms_based_on_distros()
return diff_latest_rpms_based_on_distros()
elif global_config.from_distro is None and \
global_config.to_distro is None and len(global_config.NVR) > 1:
returncode = diff_two_nvras_from_koji()
if global_config.from_distro is None and global_config.to_distro is None:
if len(global_config.NVR) > 1:
left_one = global_config.NVR[0]
right_one = global_config.NVR[1]
else:
print >>sys.stderr, 'Unknown arguments. Please refer to --help.'
returncode = 1
if is_rpm_file(left_one) and is_rpm_file(right_one):
return diff_from_two_rpm_files(left_one, right_one)
return returncode
both_nvr = match_nvr(left_one) and match_nvr(right_one)
both_nvra = match_nvra(left_one) and match_nvra(right_one)
if both_nvr or both_nvra:
return diff_two_nvras_from_koji()
print >>sys.stderr, 'Unknown arguments. Please refer to --help.'
return 1
if __name__ == '__main__':
try:
main()
sys.exit(main())
except KeyboardInterrupt:
if global_config.debug:
logger.debug('Terminate by user')