Commit Graph

101368 Commits

Author SHA1 Message Date
Yuval Lifshitz
9945a9c084
Merge pull request #29877 from yuvalif/pubsub_regression_issue2
rgw/pubsub: service reordering issue
2019-08-27 16:18:33 +03:00
Nathan Cutler
808d394beb
Merge pull request #29797 from smithfarm/wip-backport-resolve-issue
script: add backport-resolve-issue

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-08-27 15:14:01 +02:00
Nathan Cutler
91d62cf8fa
Merge pull request #29904 from smithfarm/wip-bci-resolve-parent
script/backport-create-issue: add --resolve-parent feature

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-08-27 15:07:51 +02:00
Kefu Chai
e0f1ed20b9
Merge pull request #29912 from tchaikov/wip-wheels-only-for-make-check
install-deps.sh: only prepare wheels for "make check"

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2019-08-27 20:49:01 +08:00
Nathan Cutler
70ebffca56 script: add backport-resolve-issue
This script processes GitHub backport PRs, checking for proper cross-linking
with a Redmine Backport tracker issue and, if a PR is merged and properly
cross-linked, it can optionally resolve the tracker issue and correctly
populate the "Target version" field.

The script takes a single positional argument, which is optional. If the
argument is an integer, it is assumed to be a GitHub backport PR ID (e.g. "28549").
In this mode ("single PR mode") the script processes a single GitHub backport
PR and terminates.

If the argument is not an integer, or is missing, it is assumed to be a
commit (SHA1 or tag) to start from. If no positional argument is given, it
defaults to the tag "BRI", which might have been added by the last run of the
script. This mode is called "scan merge commits mode".

In both modes, the script scans a local git repo, which is assumed to be
in the current working directory. In single PR mode, the script will work
only if the PR's merge commit is present in the current branch of the local
git repo. In scan merge commits mode, the script starts from the given SHA1
or tag, taking each merge commit in turn and attempting to obtain the GitHub
PR number for each.

For each GitHub PR, the script interactively displays all relevant information
(NOTE: this includes displaying the GitHub PR and Redmine backport issue in
web browser tabs!) and prompts the user for her preferred disposition.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
2019-08-27 14:42:37 +02:00
Kefu Chai
b9bdd34d0a
Merge pull request #29790 from hjwsm1989/fix-unittest-osdmap
tests/osd: fix typo in unittest_osdmap

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-08-27 19:42:21 +08:00
Yuval Lifshitz
e91489f443 rgw/pubsub: service reordering issue
Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
2019-08-27 14:36:08 +03:00
Kefu Chai
ff631a19b4 install-deps.sh: only prepare wheels for "make check"
* move `for_make_check` to the beginning of script, as FreeBSD will also
  use this variable
* extract `preload_wheels_for_tox()` function out to improve readability
* call `preload_wheels_for_tox()` only if `for_make_check` is true

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-27 17:53:28 +08:00
Kefu Chai
b0aec38341
Merge pull request #29090 from liewegas/wip-40792
mon/MonClient: ENXIO when sending command to down mon

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-08-27 17:34:13 +08:00
Kefu Chai
a08f4a30f9
Merge pull request #28177 from jecluis/wip-noreply-failures
mon: ensure prepare_failure() marks no_reply on op

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
2019-08-27 17:32:54 +08:00
Nathan Cutler
e4d6312fa2 script/backport-create-issue: add --resolve-parent feature
When --resolve-parent is provided on the command line, the script
will check the status of backport issues for those parent issues
all of whose backport issues already exist. If all the backport
issues are in status "Resolved", the parent issue's status is set
to "Resolved" as well.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
2019-08-27 11:22:46 +02:00
Kefu Chai
a342158bb7
Merge pull request #29862 from tchaikov/wip-41416
rgw,bluestore: fixes to address failures from check-generated.sh

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2019-08-27 17:16:41 +08:00
Xie Xingguo
6f74759cf4
Merge pull request #29893 from xiexingguo/wip-inc-recovery-5
osd/osd_types: inc-recovery - add special handler for lost_revert

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-08-27 13:16:20 +08:00
Kefu Chai
bf0b161115 test/encoding/check-generated.sh: show diff if cmp fails
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-27 13:00:41 +08:00
Kefu Chai
2c5e59c1fb test/encoding/check-generated.sh: run loop in the same shell
set parent scope variables in the same shell.

foo | while read ....

in the above statement, `while read ...` is executed in a subshell. so
it cannot change the bash variables in its parent shell.

in this change, the output of `foo` is redirected to the stdin of `while
read` statement. so we can capture the test failures.

before this change, the test always succeed, even if there are failures.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-27 13:00:41 +08:00
Kefu Chai
c1b47b62e4 common/DecayCounter: do not decay when testing denc
check-generated.sh tries to verify the invariance of a type after
encoding and decoding. so we should keep the value of `DecayCounter` the
same when encoding/decoding/dumping.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-27 13:00:41 +08:00
Kefu Chai
123f42d0aa rgw: rgw_log_entry always initialize total_time
zero-initialize rgw_log_entry::total_time

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-27 12:41:58 +08:00
Kefu Chai
3e65551d0a mgr/ServiceMap: use plain gid
* partially revert 5c25a018. which is not backward compatible.
* change `ServiceMap::get_daemon()` so it returns a
  `pair<Daemon*,bool>`.

git is an `optional<uint64_t>`, so we cannot dump it without checking.

Fixes: https://tracker.ceph.com/issues/41424
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-27 12:41:58 +08:00
Kefu Chai
1726bf21ab osd: init pg_pool_t::pg_autoscale_mode in ctor
so its value is determined

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-27 12:41:58 +08:00
Kefu Chai
4c9e142b77 osd: use scoped enum for pg_pool_t::pg_autoscale_mode
for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-27 12:41:57 +08:00
Kefu Chai
0c1836d3d3 osd: use in-class initialization in pg_pool_t
less repeat this way

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-27 12:41:57 +08:00
Kefu Chai
8bc4352725 os/bluestore: implement copy ctor and assignment for bluestore_blob_use_tracker_t
to address the `check-generated.sh` test failure.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-27 12:41:57 +08:00
Kefu Chai
6833234fd2 rgw: RGWObjManifest::obj_iterator::seek always update locations
we should update explicit loc and location even if
`ofs >= manifest->obj_size`. there is chance that we are updating an end
iterator whose ofs is equal to obj_size. before being updated, the end
iterator points to an implicit location, while after being updated, the
manifest could be using an explicit location, so we should update the
end iterator as well.

Fixes: https://tracker.ceph.com/issues/41416
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-27 12:41:57 +08:00
Kefu Chai
690267208d rgw: RGWObjManifest::obj_iterator: init member variables in-class
* remove the init() member function which is solely used by the
  constructors.
* use constructor delegation to minimize the repeating

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-27 12:41:57 +08:00
Kefu Chai
1efc7b1d74 rgw: update_iterator() in RGWObjManifest::set_explicit()
we should update the end iterator when updating RGWObjManifest with its
parts

Fixes: https://tracker.ceph.com/issues/41416
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-27 12:41:57 +08:00
Kefu Chai
bad4ccc7bf
Merge pull request #29913 from tchaikov/wip-debian/control
debian/control: fix Build-Depends

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
2019-08-27 10:28:00 +08:00
Kefu Chai
f6d7fd28af debian/control: fix Build-Depends
it's a regression introduced by 5d6d770e

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-27 10:05:17 +08:00
Patrick Donnelly
babd27c9e8
Merge PR #29867 into master
* refs/pull/29867/head:
	doc: default values for mon_health_to_clog_* were flipped

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2019-08-26 14:09:39 -07:00
Neha Ojha
04b5c38ce4
Merge pull request #29822 from vumrao/wip-vumrao-41399
os/bluestore/BlueFS: Move bluefs alloc size initialization log message to log level 1.

Reviewed-by: Sage Weil <sage@redhat.com>
2019-08-26 12:57:58 -07:00
Vikhyat Umrao
96d48af325 os/bluestore/BlueFS: Move bluefs alloc size initialization log message to log level 1
Fixes: https://tracker.ceph.com/issues/41399

Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>
2019-08-26 12:40:36 -07:00
Kefu Chai
f3d3e37c7e
Merge pull request #29903 from tchaikov/wip-pip-download
install-deps.sh: download wheel using 'pip wheel'

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2019-08-27 00:44:28 +08:00
Kefu Chai
3562504afd install-deps.sh: download wheel using 'pip wheel'
otherwise we will fail to install the build dependencies of
`lazy-object-proxy` from the wheelhouse. as `lazy-object-proxy` does not
add `setuptools_scm` in its `setup.py`, instead it lists
`setuptools_scm` in `setup.cfg` and `pyproject.toml` as a `build-system`
requires. but unfortunately, `pip download` only downloads the
install/run-time dependencies at this moment. and `lazy-object-proxy`
does not offer binary package for at least python2.7.

ideally, `pip download` should collects its dependencies like

Collecting setuptools_scm>=3.3.1 (from lazy-object-proxy->astroid<3,>=2.2.0->pylint->-r requirements-lint.txt (line 1))

so we need to use `pip wheel` do download build-time dependencies

see also https://github.com/pypa/pip/issues/6222

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-26 23:06:39 +08:00
Kefu Chai
35dbbf9a9b
Merge pull request #29854 from tchaikov/wip-sloppy-cond
common/condition_variable_debug: do not assert() if sloppy

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
2019-08-26 22:38:50 +08:00
Kefu Chai
00ccadff26
Merge pull request #29866 from joscollin/wip-fix-stupid
os/bluestore: silence StupidAllocator reorder warning

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-08-26 22:36:41 +08:00
Alfredo Deza
846d6c775a
Merge pull request #29890 from rhcs-dashboard/wip-signed-master
github: display phrase for signed-off check

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>
2019-08-26 08:20:57 -04:00
Kefu Chai
a3fdfe6d67
Merge pull request #29831 from tchaikov/wip-python-int
mgr/BaseMgrModule: use PyInt_Check() to compatible with py2

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-08-26 20:06:21 +08:00
Lenz Grimmer
0301fc9061
mgr/dashboard: Edit Pool Placement Group E2E Test (#29779)
mgr/dashboard: Edit Pool Placement Group E2E Test

Reviewed-by: Tiago Melo <tmelo@suse.com>
2019-08-26 11:36:53 +00:00
Lenz Grimmer
fb63142ab0
Merge pull request #29707 from rafaelquint/dashdoc
mgr/dashboard: Extend "Writing End-to-End Tests" section (describe vs it)

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
2019-08-26 11:33:41 +00:00
Ernesto Puerta
dcf40c7d01
github: display phrase for signed-off check
https://github.com/ceph/ceph-build/pull/1379 will introduce a
trigger-phrase for pr-commit check (DCO/signed-off).

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
2019-08-26 11:16:25 +02:00
Kefu Chai
20bebe242b
Merge pull request #29840 from tchaikov/wip-install-deps.sh-without-python-saml
install-deps.sh,deb,rpm: move python-saml deps into debian/control an…

Reviewed-by: Ricardo Marques <rimarques@suse.com>
2019-08-26 16:40:02 +08:00
xie xingguo
c165c5286a osd/osd_types: add_next_event - add special handler for lost_revert
For unfound objects, we might append LOST_REVERT log entries,
which shall allow these objects to be reverted to the newest
available version later.
However, we are currently lack of support to rewind the clean_regions
portion too when marking unfound objects as lost with inc-recovery mode
enabled. Hence we must mark these unfound-revert objects as fully dirty
to make sure they can be correctly recovered.

E.g.,:
- primary is pulling object A from replica 1
- object A is corrupted on replica 1
- object A is now unfound
- mark object A as lost, replica 1 will persist a wrong
  missing item for object A..

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-08-26 13:13:27 +08:00
xie xingguo
81c4ed2b1d osd/osd_types: drop 'new_object' from constructor
There is no consumer.
Actually, I think this field is only meaningful to be used
to indicate whether we should initiate an inc-recovery or not.
If not, then we shall fall back to triggering a full-recovery
instead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-08-26 13:13:27 +08:00
xie xingguo
e70f8ddb5a osd/osd_types: drop 'new_object' from missing.add
because below here we know we'll always mark object as fully dirty.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-08-26 13:13:27 +08:00
xie xingguo
679f0777a7 osd/osd_types: always call mark_fully_dirty for missing.add
In general we shall build missing set (and hence clean_regions)
based on pg log. However, currently there are still 5 cases we might call
missing.add to add a new pg_missing_item into the missing set
explicitly (or replace an existing pg_missing_item entirely):

1. we explicitly build missing set on startup, in which case
we know we are trying to be compatiable with pre-kraken versions,
so it should be ok for us to disable inc-recovery.

2. we are currently processing authoritative log, and there are
some divergent objects detected. For simplicity (and correctness),
we should disable inc-recovery entirly for these objects.

3. we are re-building missing set, e.g., due to the global
CEPH_OSDMAP_RECOVERY_DELETES policy changing.
In this case we know we are at the end of upgrading from a
pervious version that is lack of CEPH_OSDMAP_RECOVERY_DELETES support.
Hence it should be the recommended option to disable inc-recovery
simultaneously since these objects should be lack of inc-recovery support
too.

4. we are adding or re-adding missing object into primary's missing_loc.
It doesn't matter whether we have a correct clean_regions there
since we never actually refer to that field from missing_loc
when we actually start to perform object recovery later.

5. we are auto-repairing a corrupted object and hence the need of
adding it to the corresponding missing set first, e.g, by leveraging
the existing recovery procedure. In this case, we always disable
inc-recovery to make sure this object can be fully (and correctly)
recovered later.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-08-26 13:12:55 +08:00
Mykola Golub
732fe579b6
Merge pull request #29808 from dillaman/wip-41356
rbd-mirror: ignore errors relating to parsing the cluster config file

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>
2019-08-25 20:23:51 +03:00
James McClune
820ada1e58 doc: default values for mon_health_to_clog_* were flipped
On a freshly installed nautilus cluster (i.e. 14.2.2), the default values are:

mon_health_to_clog_interval = 3600
mon_health_to_clog_tick_interval = 60.000000

Fixes: https://tracker.ceph.com/issues/41403
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>
2019-08-25 11:35:41 -04:00
Kefu Chai
a576f9bbc2
Merge pull request #29651 from rzarzynski/wip-crimson-cls
crimson, osd: add support for Ceph Classes, part 1

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-08-25 15:25:41 +08:00
Kefu Chai
083c3a5d1f
Merge pull request #29595 from bk201/wip-41151
mgr/test_orchestrator: Allow initializing dummy data

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
2019-08-25 15:13:33 +08:00
Kefu Chai
0d2891a058
Merge pull request #29787 from NancySu05/fix-osd
osd: add log information to record the cause of do_osd_ops failure

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-08-25 15:11:51 +08:00
Jos Collin
510679f61a
os/bluestore: silence StupidAllocator reorder warning
warning: ‘StupidAllocator::last_alloc’ will be initialized after [-Wreorder]

Signed-off-by: Jos Collin <jcollin@redhat.com>
2019-08-25 10:36:53 +05:30