Commit Graph

101465 Commits

Author SHA1 Message Date
Radoslaw Zarzynski
9c01ab59be osd: share PGLSPlainFilter with crimson.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 03:33:59 +02:00
Radoslaw Zarzynski
2c0f089710 objclass, osd: dissect PGLSFilter into the osd_types.h header.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 03:33:59 +02:00
Radoslaw Zarzynski
10fca231e2 crimson/osd: implement PGLSFilter.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 03:33:59 +02:00
Radoslaw Zarzynski
50f042d661 crimson/osd: OpsExecuter differentiates read and modifying operations.
This is necessary to pass ClsHello.BadMethods.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 03:33:59 +02:00
Radoslaw Zarzynski
d2640e5b45 crimson/osd: implement CEPH_OSD_OP_GETXATTR and cls_cxx_getxattr().
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 03:33:59 +02:00
Radoslaw Zarzynski
7631691f9f crimson/osd: differentiate ENOENT during do_osd_call().
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 03:33:58 +02:00
Radoslaw Zarzynski
e374b22086 crimson/osd: implement cls_cxx_read2().
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 03:33:58 +02:00
Radoslaw Zarzynski
6c7e7ad7ce crimson/osd: implement cls_cxx_setxattr().
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 03:33:58 +02:00
Radoslaw Zarzynski
edced75b15 crimson/osd: implement CEPH_OSD_OP_SETXATTR.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 03:33:58 +02:00
Radoslaw Zarzynski
ca3ad1dbd1 crimson/os: port get_max_attr_name_length() from MemStore.
This method is an enabler for CEPH_OSD_OP_SETXATTR.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 03:33:58 +02:00
Radoslaw Zarzynski
71ccd4f2b1 objclass, osd: drop unused variable.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 03:33:57 +02:00
Radoslaw Zarzynski
eee4b843cf crimson/osd: implement cls_cxx_write_full().
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 03:33:57 +02:00
Radoslaw Zarzynski
44e14775a0 crimson/osd: introduce OpsExecuter to uniform calling CEPH_OSD_OPS.
OSD has two entry points for executing CEPH_OSD_OP_*:
  1. the MOSDOp message handler,
  2. the Object Class API (cls_* and cls_cxx_* functions).

We definitely want to address these two users without code
duplication. However, exposing the entire PG to Obj Class
would break encapsulation. Moreover, there is difference
in life times between PG and sequence-of-operations-from-
MOSDOp.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 03:33:57 +02:00
Radoslaw Zarzynski
aaadbe6df7 crimson/osd: implement cls_cxx_stat().
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 03:33:57 +02:00
Radoslaw Zarzynski
3d8b28bfa0 crimson/osd: implement CEPH_OSD_OP_STAT.
This commit is enabler of cls_cxx_stat(), and thus the RecordHello
and ClsHello unit test.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 01:27:44 +02:00
Radoslaw Zarzynski
c76ed81343 crimson/osd: implement CEPH_OSD_OP_CALL.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 01:27:44 +02:00
Radoslaw Zarzynski
ed8eee05cd crimson/osd: rename crimson_error and introduce namespace.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 01:27:44 +02:00
Radoslaw Zarzynski
2407f576ef crimson/osd: exceptions derive from std::system_error now.
This change is be useful especially for CEPH_OSD_OP_CALL
which will be brought by further commits. The issue here
is that the Ceph Classes will be based on existing iface
handling errors with usual, int-based ret codes. These
codes needs to be glued with the existing handling  in
pg::do_osd_ops().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 01:27:44 +02:00
Radoslaw Zarzynski
a9c29ae5ac crimson: put extra debugs into do_osd_op().
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 01:27:44 +02:00
Radoslaw Zarzynski
57330088d3 osd: switch ClassHandler::get_method() to std::string.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 01:27:43 +02:00
Radoslaw Zarzynski
5a8acb6868 crimson/osd: mock OSD-specific parts of the objclass API.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 01:27:40 +02:00
Radoslaw Zarzynski
4bb662948d objclass, crimson: initialize the ClassHandler.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 01:21:59 +02:00
Radoslaw Zarzynski
42fbda5d1c objclass, osd: dissect the OSD-specific parts from class_api.cc.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 01:21:59 +02:00
Radoslaw Zarzynski
125508f864 osd: improve formatting and refactor ClassHandler.h a bit.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-24 01:18:34 +02:00
Patrick Donnelly
f13b3483e7
Merge PR #28855 into master
* refs/pull/28855/head:
	doc: document scrub summary in ceph status output
	test: extend scrub control test to validate mds task status
	mds: send scrub state changes to cluster log.
	mds: periodically sent mds scrub status to ceph manager
	mgr, mon: allow normal ceph services to register with manager

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2019-08-23 16:16:16 -07:00
Patrick Donnelly
7ac9f243cd
Merge PR #29167 into master
* refs/pull/29167/head:
	client: return -eio when sync file which unsafe reqs has been dropped

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2019-08-23 16:10:40 -07:00
Patrick Donnelly
f68c087e37
Merge PR #29572 into master
* refs/pull/29572/head:
	mds: Reorganize class members in FSMap header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2019-08-23 16:06:51 -07:00
Jason Dillaman
1c40cfc82e rbd-nbd: shutdown should wait for pending IOs to complete
Previously it was incorrectly waiting for finished IOs to complete.
The finished IO would have already been cleared by the writer thread
at this point. Any remaining pending or finished IOs implies a
failure / shutdown of the NBD device.

Fixes: https://tracker.ceph.com/issues/41409?
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-08-23 15:05:39 -04:00
Jason Dillaman
d1322dba2e test/librbd: set nbd timeout due to newer kernels defaulting it on
Fixes: https://tracker.ceph.com/issues/41409
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-08-23 15:05:31 -04:00
David Zafman
0115595c1d osd: Rename backfill reservation reject names to reflect too full use
The only reason for a reject is in a too full condition.  Now that we
have two types of revoke, let's make this clearer.

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-08-23 11:34:32 -07:00
David Zafman
7810ee584c osd: Rename MBackfillReserve::TOOFULL to what it does in particular (revoke)
Signed-off-by: David Zafman <dzafman@redhat.com>
2019-08-23 11:31:45 -07:00
David Zafman
fa569ecfec osd: Don't set backfill_toofull in RemoteReservationRevoked path
We shouldn't set backfill_toofull when a revoke occurs in the non-toofull
case.

Fixes: https://tracker.ceph.com/issues/41255

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-08-23 11:29:18 -07:00
Kefu Chai
5d6d770e2f install-deps.sh,deb,rpm: move python-saml deps into debian/control and ceph.spec.in
these dependencies are only used for building python-saml which is in
turn used for the SAML support. this feature is tested using
`test_sso.py` while performing dashboard tests. we do not package or
ship python-saml along with other Ceph packages. so let's move these
dependencies to the "make check" sections in ceph.spec.in and
debian/control for simplifying install-deps.sh.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-24 02:11:51 +08:00
Jason Dillaman
5bb807d1ae
Merge pull request #29796 from trociny/wip-journal-player-handle_cache_rebalanced2
journal: fix race between player shut down and cache rebalance

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-08-23 13:40:15 -04:00
Jason Dillaman
4e8a825777
Merge pull request #29775 from trociny/wip-41229
librbd: always try to acquire exclusive lock when removing image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-08-23 13:39:37 -04:00
Jason Dillaman
648b649848
Merge pull request #29459 from zy751713126/config_set
pybind/rbd: add config_set/get/remove api in rbd.pyx

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-08-23 13:39:07 -04:00
Patrick Donnelly
d30af45a54
Merge PR #29715 into master
* refs/pull/29715/head:
	qa: fix broken ceph.restart marking of OSDs down
	qa: add debugging failed osd-release setting

Reviewed-by: Sage Weil <sage@redhat.com>
2019-08-23 10:09:17 -07:00
Patrick Donnelly
b82e87bda4
Merge PR #29821 into master
* refs/pull/29821/head:
	qa: stop DaemonWatchdog for each cluster in daemon roles

Reviewed-by: Jos Collin <jcollin@redhat.com>
2019-08-23 10:00:52 -07:00
Kefu Chai
2a18b3bbd0 common/condition_variable_debug: do not assert() if sloppy
replicate the behavior of `Cond::SloppySignal()` to avoid crash like

 ceph version Development (no_version) octopus (dev)
  1: <ceph::__ceph_assert_fail(char const*, char const, int, char const)+0x1e0> at /home/jenkins/workspace/ceph-master36/build/bin/ceph-osd
  2: <ceph::__ceph_assert_fail(ceph::assert_data const&)+0x2f> at /home/jenkins/workspace/ceph-master36/build/bin/ceph-osd
  3: <ceph::condition_variable_debug::notify_all(bool)+0x59> at /home/jenkins/workspace/ceph-master36/build/bin/ceph-osd
  4: <FileJournal::check_for_full(unsigned long, long, long)+0x4e2> at /home/jenkins/workspace/ceph-master36/build/bin/ceph-osd
  5: <FileJournal::prepare_single_write(FileJournal::write_item&, ceph::buffer::v14_2_0::list&, long&, unsigned long&, long&)+0x71> at /home/jenkins/workspace/ceph-master36/build/bin/ceph-osd
  6: <FileJournal::prepare_multi_write(ceph::buffer::v14_2_0::list&, unsigned long&, ceph::buffer::v14_2_0::list&)+0x19d> at /home/jenkins/workspace/ceph-master36/build/bin/ceph-osd
  7: <FileJournal::write_thread_entry(void)+0x561> at /home/jenkins/workspace/ceph-master36/build/bin/ceph-osd
  8: <FileJournal::Writer::entry(void)+0x19> at /home/jenkins/workspace/ceph-master36/build/bin/ceph-osd
  9: <Thread::entry_wrapper(void)+0x84> at /home/jenkins/workspace/ceph-master36/build/bin/ceph-osd
  10: <Thread::_entry_func(void*)+0x15> at /home/jenkins/workspace/ceph-master36/build/bin/ceph-osd

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-24 00:31:09 +08:00
Sage Weil
af4500be86 Merge PR #29575 into master
* refs/pull/29575/head:
	objclass, osd: improve const-correctness of PGLSFilter.
	common: add bl::contents_equal() override for void* + size_t.
	osd: refactor manufacturing of PGLSFilter.
	osd: don't carry PGLSFilter between multiple ops in MOSDOp.

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-08-23 11:26:28 -05:00
Sage Weil
1f39b36b8e Merge PR #28727 into master
* refs/pull/28727/head:
	test/crimson: resolve name collision
	test: switch to ldout; let users specify mon debug level
	test: add new ElectionLogic unit test framework
	elector: const-ify a bunch of functions
	elector: swap order of parameters in ElectionLogic::receive_propose
	elector: Update Elector and ElectionLogic function documentation
	elector: persist the epoch in bump_epoch()
	elector: make some more ElectionLogic members private
	elector: fix privacy and restore dout in Elector
	elector: don't clear peer_info in bump_epoch()
	elector: split ElectionLogic into its own compilation unit
	elector: move all the elector callouts into the Elector
	elector: make ElectionLogic private to Elector; undo most public shenanigans
	elector: create declare_standlone_victory in Elector/Logic for Monitor
	elector: make ElectionLogic::declare_victory private
	elector: route _bump_epoch through the interface-to-be
	elector: rename handle_propose_logic -> receive_propose
	elector: hoist handle_victory into ElectionLogic
	elector: hoist handle_ack into ElectionLogic
	elector: hoist victory into ElectionLogic
	elector: hoist expire into ElectionLogic
	elector: hoist start into ElectionLogic
	elector: hoist participating into ElectionLogic
	elector: hoist init into ElectionLogic
	elector: hoist defer into ElectionLogic
	elector: split handle_propose in two and hoist into ElectionLogic
	elector: hoist bump_epoch into ElectionLogic
	elector: store accessors for ElectionLogic
	elector: hoist Elector data bits out into a new ElectionLogic class
	mon: Rearrange Paxos::dispatch to be a little cleaner

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2019-08-23 11:25:28 -05:00
Sage Weil
8876cf36c2 Merge PR #15183 into master
* refs/pull/15183/head:
	kv/rocksdb: support rmrange unconditionally
	cls/rgw: rgw_bi_log_trim() uses cls_cxx_map_remove_range()
	cls/log: cls_log_trim() uses cls_cxx_map_remove_range()
	test/cls: add cls_log.trim_by_marker test
	test/cls: test_cls_log doesn't allocate ObjectOperations
	test/cls: test_cls_log uses fixture for temporary pool
	test/cls: add cls_rgw.bi_log_trim test
	cls/rgw: expose cls_rgw_bilog_list/trim() for single shard
	test/cls: test_cls_rgw uses cls_rgw_obj_key
	test/cls: test_cls_rgw doesn't allocate ObjectOperations
	test/cls: test_cls_rgw uses fixture for temporary pool
	objclass: add cls_cxx_map_remove_range()
	librados: add rados_write_op_omap_rm_range2()
	osdc: add Objecter omap_rm_range()
	osd: add CEPH_OSD_OP_OMAPRMKEYRANGE to do_osd_ops()
	osd: add omap_rmkeyrange() to PGTransaction
	os: add bufferlist overload for omap_rmkeyrange()
	tracing: add do_osd_op_pre_omaprmkeyrange
	rados: add CEPH_OSD_OP_OMAPRMKEYRANGE

Reviewed-by: Sage Weil <sage@redhat.com>
2019-08-23 10:46:33 -05:00
Jason Dillaman
3930e9f903 test/cli-integration/rbd: fixed missing image and snap ids
Fixes: https://tracker.ceph.com/issues/41411
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-08-23 11:24:35 -04:00
Kefu Chai
b0ce0b4e13 mgr/BaseMgrModule: use PyInt_Check() to compatible with py2
in python2, we have both `PyLongObject` and `PyIntObject` for presenting
`long` and `int` types. and in python3, `PyIntObject` is dropped, `int`
is represented using `PyLongObject`. and the related functions like
`PyInt_Check()` and `PyInt_AsLong()` are removed along with
`PyIntObject`.

so we should use different set of functions for converting an `int` python
object to native type in C++. since we don't use `long` for "count" when
calling `MgrModule.set_health_checks()`, in this change, we just define
a macro to unify the difference between python2 and python3.

the same applies to "max_count" parameter of `MgrModule.add_osd_perf_query()`
call.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-23 22:30:22 +08:00
Casey Bodley
6b0f3ce4cb
Merge pull request #29778 from cbodley/wip-41212
vstart: move [client.rgw] config into [client]

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
2019-08-23 10:24:41 -04:00
Lenz Grimmer
862876d900
mgr/dashboard: User Management E2E tests (#29641)
mgr/dashboard: User Management E2E tests 

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
2019-08-23 14:00:16 +00:00
Jos Collin
f13f9f9fc1
qa/tasks: drop object inherit
Signed-off-by: Jos Collin <jcollin@redhat.com>
2019-08-23 15:29:27 +05:30
Lenz Grimmer
d373178c43
mgr/dashboard: run-backend-api-tests.sh CI improvements (#29504)
mgr/dashboard: run-backend-api-tests.sh CI improvements

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
2019-08-23 09:11:39 +00:00
Kefu Chai
e909031d3c
Merge pull request #29590 from Aran85/fix_proc_replica_log
osd: merge replica log on primary need according to replica log's crt

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-08-23 14:59:02 +08:00
Changcheng Liu
940a5b5ae3 msg/async/Stack: rename variable to improve readability
1. rename var i to be worker_id when creating Worker
"i" is assigned to be Worker::id, it means worker's id

2. rename EventCenter::idx to EventCenter::center_id
"idx" is EventCenter's index in global_centers obj.
rename it to be center_id.

3. rename EventCenter::init API's parameter n to be nevent
"n" is actually assigned to EventCenter::nevent. rename it
to be "nevent".

4. rename EventCenter::init API's paramter t to be type
"t" is corresponding to Epoll Driver's implementation's type.

5. rename EpollDriver::size to be EpollDriver::nevent
"size" is actually epoll events number, rename it to be "nevent"

6. use event_id as index name to get event instead of "j"

7. rename "nw" to be "nowait"

8. Processor::start unify variable name with Processor::accept & Processor::stop
==> auto &l to be auto &listen_socket

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
2019-08-23 14:36:05 +08:00