Commit Graph

71047 Commits

Author SHA1 Message Date
John Spray
131d1bd570 qa: add log whitelists for MDS health messages
Now that we send these to the cluster log, we must
whitelist them in the tests that exercise those
unhealthy states.

Fixes: http://tracker.ceph.com/issues/19551
Signed-off-by: John Spray <john.spray@redhat.com>
2017-04-14 05:47:43 -04:00
Kefu Chai
27dc8a1685 mon/OSDMonitor: transit creating_pgs from pgmap when upgrading
there could be some pg(s) still being created when we are upgrading to
luminous, and the pools holding them are not changed in the sense of
pg_pool_t::last_change after the upgrade and before we scan for
creating pgs. in that case, the existing update_pending_creatings()
will fail to collect the pgs being created before the upgrade.

with this change, the creating_pgs in pgmap are also used for updating
the OSDMonitor's creating_pgs if it's updated.

but we should stopupdating the pgmap once the upgrade completes. i.e.
stop dispatching MSG_PGSTATS messages to PGMonitor if the quorum and all
osds are luminous.

Fixes: http://tracker.ceph.com/issues/19584
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-04-14 16:45:23 +08:00
Gui Hecheng
accf8cfd22 Revert "rgw_file.h: fix potential data race condition"
This reverts commit 3f7b411e1e.
Causing hung up due to double lock.

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>
2017-04-14 16:17:01 +08:00
Kefu Chai
125f27e3e8 Merge pull request #10851 from atheism/rgw-s3-doc-update
doc/radosgw/s3/cpp.rst: update usage of libs3 APIs to make the examples work

Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-04-14 10:18:10 +08:00
Brad Hubbard
7aa5792d19 Merge pull request #14469 from badone/wip-return-error-on-missing-client-id
mds: Return error message instead of asserting

Reviewed-by: John Spray <john.spray@redhat.com>
2017-04-14 08:10:59 +10:00
Sage Weil
ef5e1d31e7 mon/OSDMonitor: slightly better debug output
Signed-off-by: Sage Weil <sage@redhat.com>
2017-04-13 17:11:19 -04:00
Sage Weil
7959760d91 mon/OSDMonitor: fix creating pg trimming during upgrade
The creating pgs are in the PGMap with the CREATING
state bit set.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-04-13 17:11:19 -04:00
Sage Weil
8fd450e986 qa/suites/rados: require_luminous_osds at end of run, then scrub
Signed-off-by: Sage Weil <sage@redhat.com>
2017-04-13 17:11:19 -04:00
Sage Weil
5ca72c1193 qa/tasks/exec_on_cleanup.py: add
Signed-off-by: Sage Weil <sage@redhat.com>
2017-04-13 17:11:19 -04:00
Willem Jan Withagen
ddf44e948b src/ceph-disk/ceph_disk/main.py: raise exeception if zfs is not found
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
2017-04-13 22:05:26 +02:00
Loic Dachary
6e20d648e9 Merge pull request #14508 from smithfarm/wip-doc-dev
doc: dev guide: how to run s3-tests locally against vstart

Reviewed-by: Loic Dachary <ldachary@redhat.com>
2017-04-13 20:54:46 +02:00
Sage Weil
9e88a71105 os/bluestore: tune deferred_batch_ops separately for hdd and ssd
Signed-off-by: Sage Weil <sage@redhat.com>
2017-04-13 14:09:14 -04:00
Sage Weil
9967fceb62 Merge pull request #14420 from optimistyzy/329
os/bluestore/NVMEDevice: Add multiple thread support for SPDK I/O thread

Reviewed-by: Haomai Wang <haomai@xsky.com>
2017-04-13 12:34:51 -05:00
Nathan Cutler
0dd2dfce1d doc: dev guide: how to run s3-tests locally against vstart
Add a bunch of verbiage to the Developer Guide

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>
2017-04-13 19:14:52 +02:00
Matt Benjamin
9a5cdf3b48 Merge pull request #14458 from linuxbox2/wip-rgw_file-readahead
rgw_file: support readdir cb type hints (plus fixes)
2017-04-13 13:12:27 -04:00
John Spray
177a97d5c5 mds: shut down finisher before objecter
Some of the finisher contexts would try to call into Objecter.
We mostly are protected from this by mds_lock+the stopping
flag, but at the Filer level there's no mds_lock, so in the
case of file size probing we have a problem.

Fixes: http://tracker.ceph.com/issues/19204
Signed-off-by: John Spray <john.spray@redhat.com>
2017-04-13 12:43:39 -04:00
John Spray
ccd6c81742 Merge pull request #13966 from jcsp/wip-client-logmsg
client: specify inode in get_caps log message

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
2017-04-13 17:43:10 +01:00
John Spray
e1b4327d67 Merge pull request #14087 from jcsp/wip-doc-fscid
doc: how to specify filesystem for cephfs clients

 Reviewed-by: Yan, Zheng <zyan@redhat.com>
2017-04-13 17:42:13 +01:00
John Spray
c0bff51ef4 mds: include advisory path field in damage
This will just be whatever path we were looking
at at the point that damage was notified -- no
intention whatsoever of providing any up to date
path or resolution when there are multiple paths
to an inode.

Fixes: http://tracker.ceph.com/issues/18509
Signed-off-by: John Spray <john.spray@redhat.com>
2017-04-13 12:41:11 -04:00
John Spray
c39aaf90ed mds: validate prealloc_inos on sessions after load
Mitigates http://tracker.ceph.com/issues/16842

Signed-off-by: John Spray <john.spray@redhat.com>
2017-04-13 12:32:07 -04:00
John Spray
0f89787d83 mds: operator<< for Session
Use this to get a nice human readable name
when available (also including the session id in
parentheses)

Signed-off-by: John Spray <john.spray@redhat.com>
2017-04-13 12:32:07 -04:00
John Spray
0f7988d4f0 mds: pretty json from tell commands
The overhead of the whitespace is trivial and
makes the output somewhat human readable.  Previously
I was always taking `damage ls` into a file and
parsing it out with python.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-04-13 12:28:20 -04:00
John Spray
e05493fe54 PendingReleaseNotes: recent cephfs changes
Signed-off-by: John Spray <john.spray@redhat.com>
2017-04-13 12:27:07 -04:00
John Spray
3fccc2372f mds: ignore ENOENT on writing backtrace
We get ENOENT when a pool doesn't exist.  This can
happen because we don't prevent people deleting
former cephfs data pools whose files may not have
had their metadata flushed yet.

http://tracker.ceph.com/issues/19401
Signed-off-by: John Spray <john.spray@redhat.com>
2017-04-13 12:20:25 -04:00
Ali Maredia
b31b84529e rgw multisite: use get_config_master_client for radosgw_admin task
Signed-off-by: Ali Maredia <amaredia@redhat.com>
2017-04-13 12:15:50 -04:00
Ali Maredia
c5956790e6 rgw: multisite enabled over multiple clusters
Added '--cluster' to all necessary commands
ex: radosgw-admin, rados, ceph, made sure
necessary checks were in place so that clients
can be read with our without a cluster_name
preceeding them

Made master_client defined in the config for
radosgw-admin task

Signed-off-by: Ali Maredia <amaredia@redhat.com>
2017-04-13 12:15:50 -04:00
Kefu Chai
dc7ff92e52 mgr: always free allocated MgrPyModule
use unique_ptr to manage the lifecycle of MgrPyModule and ServeThread,
it's easier and safer. without this chance, we don't free allocated
MgrPyModule if it fails to load().

Fixes: http://tracker.ceph.com/issues/19590
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-04-13 23:58:43 +08:00
Matt Benjamin
007b7451c2 rgw_file: don't expire directories being read
If a readdir expire event turns out to be older than last_readdir,
just reschedule it (but actually, we should just discard it, as
another expire event must be in queue.

Fixes: http://tracker.ceph.com/issues/19625

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2017-04-13 11:30:31 -04:00
John Spray
01f79a748b mon: emit cluster log messages on MDS health changes
Previously, when we got a beacon that updated the health
metrics for an MDS, the user would just see mysterious-looking
cluster log messages indicating a rising fsmap epoch number.

It would be good to do this for health messages in general at
some point, but for now just do it for the MDS ones.

Fixes: http://tracker.ceph.com/issues/19551
Signed-off-by: John Spray <john.spray@redhat.com>
2017-04-13 11:06:13 -04:00
John Spray
fc28d29c0c mds: fix mgrc shutdown
Were previously only tearing MgrClient down when not
holding a rank, leading to it trying to continue
to run after monclient was shut down.

Fixes: http://tracker.ceph.com/issues/19566
Signed-off-by: John Spray <john.spray@redhat.com>
2017-04-13 11:00:46 -04:00
Matt Benjamin
e0191d74e3 rgw_file: chunked readdir
Adjust readdir callback path for new nfs-ganesha chunked readdir,
including changes to respect the result of callback to not
continue.

Pending introduction of offset name hint, our caller will just be
completely enumerating, so it is possible to remove the offset map
and just keep a last offset.

Fixes: http://tracker.ceph.com/issues/19624

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2017-04-13 10:58:22 -04:00
Matt Benjamin
2e66c7a7cc rgw_file: introduce rgw_lookup type hints
The new type hints optimize object type deduction, when the
rgw_lookup is called from an rgw_readdir callback.

Fixes: http://tracker.ceph.com/issues/19623

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2017-04-13 10:35:37 -04:00
Amir Vadai
972c7416de msg/async/rdma: Use RDMA resources only after fork
Thanks to previous patch [1], no need to access RDMA resources before
the fork. Initialize Infiniband class only before a connection is
established or a listener is created. [1] is making sure that the call
to RDMAWorker::listen() is postponed till after the fork.

[1] - 7393db45644d ("msg/async: Postpone bind if network stack is not ready")

Issue: 995322
Change-Id: I8ea246b2e03c8c9533bc324b2b8d142eb3d1ed4d
Signed-off-by: Amir Vadai <amir@vadai.me>
2017-04-13 14:29:03 +00:00
Haomai Wang
a5b87e2fb8 msg/async: Postpone bind if network stack is not ready
RDMAStack shouldn't access hardware from the parent process.
The only reason to do so, is because bind is called before the fork.
After this patch the bind is postponed until the NetworkStack reports
that it is ready to bind.
For NetworkStack types will always return true, except the RDMAStack
which will return true only after the fork (after
AsyncMessenger::ready() is called).

This patch is based on a patch by Haomai Wang <haomai@xsky.com>

Issue: 995322
Change-Id: I1d0d0d52db0a339b9319680c18ee05cde87b2b64
Signed-off-by: Amir Vadai <amir@vadai.me>
2017-04-13 14:28:48 +00:00
Haomai Wang
22b0312b4f msg/async: remove duplicate NetworkStack::start which exists in constructor
Issue: 995322
Change-Id: Ib8f7f25b9b360bea97147ce27c369c6baefcc6f3
Signed-off-by: Haomai Wang <haomai@xsky.com>
2017-04-13 14:28:36 +00:00
Kefu Chai
8442cbb521 Merge pull request #14468 from tchaikov/wip-show-me-fake-plugin
qa/workunits/ceph-helpers: display rejected string

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
2017-04-13 22:24:03 +08:00
Sage Weil
20e5bff7fc mon/OSDMonitor: fix initial map when require_luminous_osds not set on mkfs
If we don't set the luminous flag, we should not set the new luninous
fields or else we'll get a crc mismatch.  (Funnily that happens in the
epoch where the flag is eventually set and the encoded map finally includes
the field we have set in memory.)

Signed-off-by: Sage Weil <sage@redhat.com>
2017-04-13 10:10:26 -04:00
Sage Weil
10ea8e1d7d mon/OSDMonitor: ignore health checks about require_luminous if debug flag is set
This makes it tedious for teuthology health checks to proceed when we
deliberately run luminous osds without this flag.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-04-13 10:10:26 -04:00
Yang Honggang
eb915d0eec cephfs: fix write_buf's _len overflow problem
After I have set about 400 64KB xattr kv pair to a file,
mds is crashed. Every time I try to start mds, it will crash again.
The root reason is write_buf._len overflowed when doing
Journaler::append_entry().

This patch try to fix this problem through the following changes:

 1. limit file/dir's xattr size
 2. throttle journal entry append operations

Fixes: http://tracker.ceph.com/issues/19033
Signed-off-by: Yang Honggang joseph.yang@xtaotech.com
2017-04-13 20:09:07 +08:00
John Spray
b81e4b70a4 Merge pull request #14234 from jcsp/wip-19406-dirhash
tools/cephfs: set dir_layout when injecting inodes

Reviewed-by: Yan, Zheng <zyan@redhat.com>
2017-04-13 12:01:28 +01:00
Willem Jan Withagen
291815aae1 ceph-disk/tests/test_main.py: Some partition tests are no longer valid under FreeBSD
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
2017-04-13 12:53:35 +02:00
Willem Jan Withagen
c8fda3f04c src/ceph-disk/ceph_disk/main.py: Make ceph-disk list work on FreeBSD
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
2017-04-13 11:59:06 +02:00
Kefu Chai
051f9138df Merge pull request #14472 from joscollin/wip-warning-test
test: add explicit braces to avoid ambiguous ‘else’ and to silence warnings

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-04-13 16:28:12 +08:00
Kefu Chai
097d00cb81 Merge pull request #14495 from tchaikov/wip-cmake-ctags-without-repo
cmake: do not try to add submodule to exclude list if .git is not around

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
2017-04-13 15:35:09 +08:00
Kefu Chai
9acd868857 Merge pull request #14404 from tchaikov/wip-another-mgr-command
mon,mgr: move reweight-by-* to mgr

Reviewed-by: Sage Weil <sage@redhat.com>
2017-04-13 13:57:42 +08:00
Kefu Chai
4847a88556 cmake: do not try to add submodule to exclude list if .git is not around
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-04-13 13:49:54 +08:00
fang yuxiang
dcaaea5dc8 rgw: bucket stats display bucket index type
Signed-off-by: fang yuxiang fang.yuxiang@eisoo.com
2017-04-13 13:47:02 +08:00
Mykola Golub
b9522ef915 Merge pull request #14445 from dillaman/wip-rbd-import-diff
rbd: import-diff should discard any zeroed extents

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
2017-04-13 08:00:05 +03:00
Matt Benjamin
335cf9a23e Merge pull request #14474 from Wilhelmshaven/fix_non-exist_string
rgw: avoid using null pointer in rgw_file.cc
2017-04-12 21:00:59 -04:00
Brad Hubbard
465ab063ad mds: Return error message instead of asserting
For "session evict" admin socket command return an error message when we
receive an invalid/missing client_id parameter rather than asserting.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
2017-04-13 10:05:19 +10:00