Commit Graph

87537 Commits

Author SHA1 Message Date
Sage Weil
6c4d76ca0c common/options: dump flags
Also document.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-06-18 07:01:24 -05:00
Kefu Chai
9551fc8585
Merge pull request #22577 from ErwanAliasr1/evelu-parallel
src/test: Using gtest-parallel to speedup unittests

Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-06-16 00:15:30 +08:00
Lenz Grimmer
93e7717cfe
Merge pull request #21998 from p-na/osd-flags-wip
mgr/dashboard: Add ability to list,set and unset cluster-wide OSD flags to the backend

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2018-06-15 16:49:51 +02:00
Patrick Donnelly
c2311e1aeb
Merge PR #22340 into master
* refs/pull/22340/head:
	doc: update description of mount options for cephfs

Reviewed-by: Zheng Yan <zyan@redhat.com>
2018-06-15 07:31:34 -07:00
Patrick Donnelly
4c3bf7132f
Merge PR #22464 into master
* refs/pull/22464/head:
	mds: print dir decay counters on hit
	DecayCounter: removed unused velocity
	DecayCounter: remove unnecessary delta member
	mds: use monotonic time for DecayCounter

Reviewed-by: Zheng Yan <zyan@redhat.com>
2018-06-15 07:05:40 -07:00
Patrick Donnelly
8db4bedabd
Merge PR #22512 into master
* refs/pull/22512/head:
	client: fix bug #24491 _ll_drop_pins may access invalid iterator

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
2018-06-15 07:05:36 -07:00
Patrick Donnelly
35a7c01d51
mds: print dir decay counters on hit
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-06-15 07:04:42 -07:00
Patrick Donnelly
201cc8ea01
DecayCounter: removed unused velocity
It's not actually used anywhere and it's not accurate.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-06-15 07:04:42 -07:00
Patrick Donnelly
50c573910b
DecayCounter: remove unnecessary delta member
We can just add/sub to val directly (especially now that decay can be called on
a const DecayCounter).

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-06-15 07:04:42 -07:00
Patrick Donnelly
bc7cb57985
mds: use monotonic time for DecayCounter
This commit has a few side-effects:
- Decaying the DecayCounter is more accurate, we do not need to "skip" decaying
  the counter if it's been less than a second since the last decay. The time
  delta is now at the granularity of the monotonic clock.
- Any check of the DecayCounter results in updating the current value, even
  const checks.
- DecayRate is now established when the DecayCounter is created. There's no longer
  a need to pass the DecayRate whenever interacting with the counter.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-06-15 07:04:42 -07:00
Patrick Donnelly
9a3f8a14bb
Merge PR #22357 into master
* refs/pull/22357/head:
	common/DecayCounter: set last_decay to current time when decoding decay counter

Reviewed-by: Zheng Yan <zyan@redhat.com>
2018-06-15 07:02:02 -07:00
Patrick Donnelly
c7951d8f36
Merge PR #22289 into master
* refs/pull/22289/head:
	mds: put Capability in map container

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-06-15 07:01:57 -07:00
Patrick Donnelly
88bf596a35
Merge PR #22266 into master
* refs/pull/22266/head:
	client:  update inode fields according to issued caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-06-15 07:01:52 -07:00
Patrick Donnelly
94445e64cf
Merge PR #22234 into master
* refs/pull/22234/head:
	mds: add magic to header of open file table

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-06-15 07:01:48 -07:00
Erwan Velu
434589a320 ctest: Removing unittest_alloc_bench
unittest_alloc_bench is very cpu consuming and can take up to 20mn to
run. As per a discussion with the original author of this code, this
test is only about testing the performance while there is no validation
code in it.

To save time from people doing make check often or even the CI itself,
removing this test will save a lot of time while not reducing the test
coverage.

This commit is only removing the test from the make check but keep the
binary being compiled for those who want to run it manually.

Signed-off-by: Erwan Velu <erwan@redhat.com>
2018-06-15 15:59:51 +02:00
Kefu Chai
eb615410d3 run-make-check: enable WITH_GTEST_PARALLEL
Signed-off-by: Kefu Chai <kchai@redhat.com>
2018-06-15 15:55:27 +02:00
Kefu Chai
4c0a2b9373 cmake: add WITH_GTEST_PARALLEL option
and remove src/test/gtest-parallel submodule, because gtest-parallel is
only useful for running tests. and not all end-users are interested in
running test not to mention running them in parallel. so, to avoid
including gtest-parallel scripts in the dist tarball. it'd be better to
make it optional, and an external project.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2018-06-15 15:55:15 +02:00
Erwan Velu
13bc625b5f src/test: Using gtest-parallel to speedup unittests
Unittests are run sequentially and could take a long while to run.

This commit is about using gtest-parallel on some of them which are
known to be very slow due to this sequentiality.

To enable the parallel features, the 'parallel' argument just have to be
added to the add_ceph_unittest() call like in :
    -add_ceph_unittest(unittest_throttle)
    +add_ceph_unittest(unittest_throttle parallel)

This commit impact the following tests :

Test name                          Before   After (in seconds)
unittest_erasure_code_shec_all:       212      43
unittest_throttle                      15       5
unittest_crush                          9       6
unittest_rbd_mirror                    79      21

Total                                 315      75

This commit saves 240 seconds (4 minutes) per build.

Note it exist several other long tests but can't be parallelized since
there is explicit dependencies in the order to run the subtests.
Those stay sequential.

Signed-off-by: Erwan Velu <erwan@redhat.com>
2018-06-15 15:53:30 +02:00
Orit Wasserman
42ff13c776
Merge pull request #22390 from votdev/rgw_metadata_self
rgw: Get the user metadata of the user used to sign the request
2018-06-15 12:21:41 +03:00
Patrick Nawracay
ef5756b5ff mgr/dashboard: Extend dev scripts for API testing
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
2018-06-15 10:26:30 +02:00
Patrick Nawracay
0d8176acb6 mgr/dashboard: Make noise when AuthRequired used incorrectly
Don't fail silently when AuthRequired is used incorrectly, namely
without parentheses.

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
2018-06-15 10:26:30 +02:00
Patrick Nawracay
cce9a0f229 mgr/dashboard: Extend RestController.bulk_set test
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
2018-06-15 10:26:30 +02:00
Patrick Nawracay
d7de1fa067 mgr/dashboard/backend: Enable get/set of cluster-wide OSD settings
Add ability to list, set and unset cluster-wide OSD flags.

Flags can be listed and changed through the `/api/osd/flags` API
resource. By using a GET request, the list is retrieved. By using a PUT
request, the flags are updated (all at once). Flags not contained in the
data of the PUT are removed, additional once are added. Note that the
PUT requests require a JSON body with the data contained as value of the
'flags' key like so:

    {"flags": ["flag1", "flag2", ...]}

Fixes: http://tracker.ceph.com/issues/24056

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
2018-06-15 10:26:29 +02:00
Lenz Grimmer
ac8f5c75ec
Merge pull request #22459 from p-na/grafana-token-auth
mgr/dashboard: Add token authentication to Grafana proxy

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2018-06-15 09:44:46 +02:00
Lenz Grimmer
85908e280f
Merge pull request #22014 from Devp00l/wip-task-wrapper
mgr/dashboard: Task wrapper service

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2018-06-15 09:40:17 +02:00
Kefu Chai
3234db0649
Merge pull request #22524 from tchaikov/wip-history-les-bound
osd/PG: unset history_les_bound if local-les is used

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
2018-06-15 10:08:20 +08:00
Kefu Chai
a20ca03042
Merge pull request #22472 from majianpeng/bluestore-misc-cleanup
os/bluestore: misc cleanup

Reviewed-by: Sage Weil <sage@redhat.com>
2018-06-15 10:07:12 +08:00
Kefu Chai
3071cb8997
Merge pull request #22538 from alimaredia/wip-vstart-cmake-cls-opt
cmake: Add cls_opt for vstart target

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-06-15 09:24:19 +08:00
Ali Maredia
f824161a3a cmake: Add cls_opt for vstart target
Adding a dependency on cls_opt for the radosgw so
that when the vstart target is made, `radosgw-admin
mfa` commands work.

Signed-off-by: Ali Maredia <amaredia@redhat.com>
2018-06-14 14:40:07 -04:00
Sage Weil
70ba58e9e8 Merge PR #22554 into master
* refs/pull/22554/head:
	qa/standalone/ceph-helpers.sh: Fixing comment for wait_for_health()
	tests: Protecting rados bench against endless loop
	qa/standalone/ceph-helpers.sh: Defining custom timeout for wait_for_clean()

Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
2018-06-14 13:04:56 -05:00
Kefu Chai
4040e2acad
Merge pull request #22491 from tchaikov/wip-seastar-msgr
crimson: port messenger to seastar

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
2018-06-14 23:24:42 +08:00
Kefu Chai
ae5fde0240
Merge pull request #22547 from tchaikov/wip-spdk
spdk: update to latest spdk-18.05 branch

Tested-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
2018-06-14 23:23:01 +08:00
Stephan Müller
f62ab0ed5b mgr/dashboard: Executing messages for tasks
Signed-off-by: Stephan Müller <smueller@suse.com>
2018-06-14 16:46:35 +02:00
Stephan Müller
fbfdbdc0d9 mgr/dashboard: Use task wrapper in RBD list
Signed-off-by: Stephan Müller <smueller@suse.com>
2018-06-14 16:46:35 +02:00
Stephan Müller
260dad1729 mgr/dashboard: Use task wrapper in RBD form
Signed-off-by: Stephan Müller <smueller@suse.com>
2018-06-14 16:46:35 +02:00
Stephan Müller
3201ba6f7d mgr/dashboard: Task wrapper service
Has a method to wrap an API call into a task.

Fixes: https://tracker.ceph.com/issues/24134

Signed-off-by: Stephan Müller <smueller@suse.com>
2018-06-14 16:46:35 +02:00
Lenz Grimmer
d785010a35
Merge pull request #22303 from ricardoasmarques/wip-help-menu
mgr/dashboard: Add help menu entry

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2018-06-14 15:56:39 +02:00
Patrick Nawracay
add29e27f4 mgr/dashboard/cleanup: Remove unnecessary parentheses
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
2018-06-14 13:54:33 +02:00
Patrick Nawracay
bd2196b6ca mgr/dashboard: Add token authentication to Grafana proxy
Enables token authentication for the Grafana proxy as additional option
to username/password authentication. The authentication method has to be
set, too.

    $ ceph dashboard set-grafana-api-token <token>	   # default: ''
    $ ceph dashboard set-grafana-api-auth-method <method>  # default: ''

Possible values for the authentication method are 'password' and
'token'.

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
2018-06-14 13:54:00 +02:00
Lenz Grimmer
27f6012ff4
Merge pull request #21460 from s0nea/wip-dashboard-config-options
mgr/dashboard: Config options integration (read-only) depends on #22422

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
2018-06-14 12:59:47 +02:00
Kefu Chai
ff81eb924e
Merge pull request #22557 from tchaikov/wip-rbd-ggate
rbd-ggate: tag "level" with need_dynamic

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
2018-06-14 17:17:31 +08:00
Erwan Velu
2ce480b8fd qa/standalone/ceph-helpers.sh: Fixing comment for wait_for_health()
wait_for_health doesn't check if the cluster is making progress. So
let's adjust the comment accordingly.

Signed-off-by: Erwan Velu <erwan@redhat.com>
2018-06-14 11:06:52 +02:00
Erwan Velu
e6e10246c6 tests: Protecting rados bench against endless loop
If the cluster dies during the rados bench, the maximum running time is
no more considered and all emitted aios are pending.

rados bench never quits and the global testing timeout (3600 sec : 1
hour) have to be reach to get a failure.

This situation is dramatic for a background test or a CI run as it locks
the whole job for too long for an event that will never occurs.

This ideal solution would be having 'rados bench' considering a failure
once the timeout is reached when aios are pending.

A possible workaround here is to put use the system command 'timeout'
before calling rados bench and fail if rados didn't completed on time.

To avoid side effects, this patch is doubling rados timeout. If rados
didn't completed after twice the expected time, it have to fail to avoid
locking the whole testing job.

Please find below the way it worked on a real test case.
We can see no IO after t>2 but despite timeout=4 the bench continue.
Thanks to this patch, the bench is stopped at t=8 and return 1.

5: /home/erwan/ceph/src/test/smoke.sh:55: TEST_multimon:  timeout 8 rados -p foo bench 4 write -b 4096 --no-cleanup
5: hints = 1
5: Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 4 seconds or 0 objects
5: Object prefix: benchmark_data_mr-meeseeks_184960
5:   sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
5:     0       0         0         0         0         0           -           0
5:     1      16      1144      1128   4.40538   4.40625  0.00412965   0.0141116
5:     2      16      2147      2131   4.16134   3.91797  0.00985654   0.0109079
5:     3      16      2147      2131   2.77424         0           -   0.0109079
5:     4      16      2147      2131    2.0807         0           -   0.0109079
5:     5      16      2147      2131   1.66456         0           -   0.0109079
5:     6      16      2147      2131   1.38714         0           -   0.0109079
5:     7      16      2147      2131   1.18897         0           -   0.0109079
5: /home/erwan/ceph/src/test/smoke.sh:55: TEST_multimon:  return 1
5: /home/erwan/ceph/src/test/smoke.sh:18: run:  return 1

Signed-off-by: Erwan Velu <erwan@redhat.com>
2018-06-14 11:06:52 +02:00
Erwan Velu
62d2646c30 qa/standalone/ceph-helpers.sh: Defining custom timeout for wait_for_clean()
The wait_for_clean() is using the default timeout aka 300sec = 5mn.

wait_for_clean() is trying to find a clean status within that timeout
_or_ reset its counter if any progress got made in between loops.

In a case where the cluster is sane, the recovery should be made in
shorter than 5mn but it the cluster died, waiting for 5mn for nothing is
unefficient.

This patch is about defining a custom timeout for a wait_for_clean() not
to wait much more that 1m30 (90sec). If no progress is made in that
period, there is very few chance this will read the a valid state
anyhow.

Signed-off-by: Erwan Velu <erwan@redhat.com>
2018-06-14 11:06:52 +02:00
Lenz Grimmer
be406bf515
Merge pull request #22526 from Devp00l/wip-new-validators
mgr/dashboard: Added new validators

Reviewed-by: Volker Theile <vtheile@suse.com>
2018-06-14 11:05:26 +02:00
Kefu Chai
9065f4edae rbd-ggate: tag "level" with need_dynamic
otherwise compiler will fail to figure out the right should_gather()
variant to use.
see also 1a3e9357

Signed-off-by: Kefu Chai <kchai@redhat.com>
2018-06-14 10:41:20 +08:00
Kefu Chai
c2026b7473 cmake: update BuildSPDK for spdk-18.05
in spdk v18.05, libuuid is linked by libspdk_util.a, in which,
it is used by lib/util/uuid.c. and libspdk_vol.a uses the wrapper
function exposed by libspdk_util.a, so update the CMakefile script to
reflect the change.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2018-06-14 09:34:01 +08:00
Kefu Chai
8098e54964 spdk: update to latest spdk-18.05 branch
also bump dpdk to to spdk-18.05 branch. this should fix the build
failure on Fedora 28

Signed-off-by: Kefu Chai <kchai@redhat.com>
2018-06-14 09:34:01 +08:00
Josh Durgin
8b83d44c90
Merge pull request #22500 from thinkercui/bugfix
osd: fix memory leak in EC fast and error read

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
2018-06-13 16:57:57 -04:00
Jason Dillaman
ad89b795f0
Merge pull request #21969 from zhongyimao/interlock_fast-diff/object-map
rbd: interlock object-map/fast-diff features together

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2018-06-13 16:22:04 -04:00