Commit Graph

134171 Commits

Author SHA1 Message Date
Samuel Just
b12feb3d38 crimson/os/FuturizedCollection: use thread_safe_counter
This should work for now, long term we probably want to get
rid of FuturizedCollection entirely from the interface.

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
2803358525 crimson/osd/heartbeat: drop ownership of messenger instances
Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
4e9dab8003 crimson/osd/pg_shard_manager: add shard_id asserts
Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
615ead85cc src/vstart.sh: add vstart --crimson-smp option
Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
b0b3c09962 crimson/net: Message::conn needs to be a foreign_ptr
There are two main consequences of this:
1. Messages can't be default copy constructed in crimson.  MMonCommand
   seems to be the only user, and we simply add a copy constructor that
   duplicates data portions of the message.
2. We can't casually copy-construct the conn into other structures.
   The main user here is watch/notify.  We use copy() explicitely
   to populate the object_context structures and avoid passing
   ConnectionFRef by value.

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
5f2c40d0c8 crimson/osd/osd_operations/client_request: remove OSD reference
Capture ShardServices reference for use in put_historic directly.

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
08758bef9e crimson/osd: move send_incremental_map to OSDSingletonState
Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
f152424631 crimson/net/Connection: let send and keepalive be called from foreign cores
Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
ea46c3e6c3 crimson/osd/shard_services: check core on each PerShardState method
Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
5451c93a27 crimson/osd/osd_operations/client_request: fix whitespace on do_process
Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
cb657b4221 crimson/osd: update_heartbeat_peers can iterate pgs syncronously
OSDSingletonState has a local inventory of all pgids.

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
a9684d4665 crimson/osd/osd_operations: use foreign_ptr for conn
Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
b692dfbad7 vstart: echo osd arguments
Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
3c632e45bb crimson/osd/osd_operations/peering_event.cc: whitespace fix
Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
998cb8c141 crimson/osd: remove compound_peering_request
The state shared between sub events creates problems for multicore.  The
only user is MOSDPGCreate2, so the optimization isn't really worth
salvaging.

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
6ea21eddba crimson/common/operation: release pipeline stages on the core they are on
Otherwise, tasks waiting on the stage will wake up on the wrong core.
Later, we may choose to statically enable this behavior only for stages
that can actually span cores, but this is ok for now.

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
752e8cf8f3 crimson/osd: move perf counter ownership to OSDSingletonState
Later, we'll want to create a properly sharded implentation that avoids
barriers.

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
ce65d7e6fa crimson/osd: shard PerShardState across cores
Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
582bbd5ee1 crimson/osd: move hb_stamps to PerShardState
We may later need to syncronize this across cores, perhaps during tick.
This should work for now, however.

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
b52990c626 crimson/osd: replace global tid with unique core-local tids
We don't really want a global counter here if we can avoid it.  Instead,
assign tids with core-local counters prefixed with the core id.  We
continue to ensure that tids are unique within an osd boot, but lose
the property that sucessive tids on different cores are ordered.  I
don't see anything relying on that property, however, so this should be
fine.

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
be1371762a crimson/osd: remove explicit pg counter
The main user of num_pgs was get_target_pg_log_entries(), but we
don't really want to be accessing a global counter there if we
can avoid it.  Instead, compute a core-local target and use the
core-local pg count.

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
1297b49c93 crimson/osd: clean up PerShardState/OSDSingletonState ownership/access
- move heartbeat messenger ownership to OSD
- move OSDSingletonState/ShardServices initialization to
  PGShardManager::start() method
- add accessor OSD::get_shard_services(), remove OSD::shard_services
  reference
- accessor/forwarding cleanups within PGShardManager

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
12042dcbc7 crimson/osd: move store responsiblity to PerShardState
Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
2b5c471913 crimson/os: consolidate context handling in FuturizedStore
Context handling is pretty uniform accross all implementations,
may as well do it in the same place.

ShardedStoreProxy would need to handle it otherwise, since
callbacks need to be handled on the core do_transaction is
invoked on.

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:41 -07:00
Samuel Just
2c1f213bfa crimson/os/futurized_store: create ShardedStoreProxy
For now, FuturizedStore implementations assume that methods are invoked
on core 0.  Later, we'll adapt each implementation to intelligently
support invocation on any pg core.  Until then, this wrapper converts
the existing implementations to a safe, if not particuarly performant,
proxy behavior.

AlienStore should be safe as is.

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:35:38 -07:00
Samuel Just
2b2f5b7009 crimson/osd: adjust get_pg_stats() and for_each_pg() to return futures
Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:33:04 -07:00
Samuel Just
a7296e5ea2 crimson/osd: refactor pg management for multicore
OSDSingletonState will now only be responsible for the spg_t->core
mapping for pgs, the individual PerShardState's will hold local
PGMap's.  PG management operations are now proxied from PGShardManager
to PerShardState.  Subsequent patches will shard PerShardState.

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:33:01 -07:00
Samuel Just
e94a12a8e3 crimson/osd: modify start_pg_operation to return op_id
The op will actually be run generally on a different core, so we don't
want to be returning a reference to it.

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:31:16 -07:00
Samuel Just
25a4612745 crimson/osd/pg_map: add PGShardMapping
Signed-off-by: Samuel Just <sjust@redhat.com>
2022-09-26 19:31:13 -07:00
Patrick Donnelly
efe41d2a9d
qa: ignore MDS_TRIM warnings when osd thrashing
Fixes: https://tracker.ceph.com/issues/57677
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2022-09-26 22:20:57 -04:00
Zack Cerza
af67a7152c tasks/cephadm: Add 'osd_method' config item
Since the default of None gets interpreted as 'lvm', the only useful
value would be 'raw'. This will enable us to use loop devices inside
containers for OSDs.

Signed-off-by: Zack Cerza <zack@redhat.com>
2022-09-26 15:35:08 -06:00
Zack Cerza
74c3446c25 cephadm: Avoid TypeError in download_cephadm()
Signed-off-by: Zack Cerza <zack@redhat.com>
2022-09-26 15:35:08 -06:00
Ken Dreyer
c2adf33ed5 Merge PR #48147 into main
* refs/pull/48147/head:
	win32: install wget when building on ubuntu
	win32: install deps with DEBIAN_FRONTEND=noninteractive

Reviewed-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Reviewed-by: Justin Caratzas <jcaratza@redhat.com>
2022-09-26 14:44:51 -04:00
Ken Dreyer
7cfe523c7f Merge PR #47153 into main
* refs/pull/47153/head:
	build: fix atomic linking with LTO on s390x

Reviewed-by: Justin Caratzas <jcaratza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2022-09-26 13:47:11 -04:00
Patrick Donnelly
e930a8b221
qa: ignore expected scrub error
The test_orphan_scan test deliberately removes a dentry which will cause
rstat damage. Ignore it.

Fixes: https://tracker.ceph.com/issues/57657
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2022-09-26 13:27:57 -04:00
Daniel Gryniewicz
036d034d66 RGW - Re-do admin API split
The mgr instantiation was re-added in the build re-org.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
2022-09-26 12:03:39 -04:00
Adam King
c68489988c
Merge pull request #48197 from phlogistonjohn/jjm-cephadm-cov-podman
cephadm: add test cases for container engine classes

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
2022-09-26 11:43:47 -04:00
Ernesto Puerta
7f41f15745
Merge pull request #47131 from aaryanporwal/vrt-doc-update
doc/dev: mgr/dashboard: Replace broken link

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2022-09-26 13:22:11 +02:00
Tobias Urdin
d6f5e49f3b qa/tasks/barbican: dont copy policy files
In newer versions the policies is inside the code
so we don't need these files for default policy
anymore.

Signed-off-by: Tobias Urdin <tobias.urdin@binero.se>
2022-09-26 06:27:40 +00:00
Yingxin
b73279e836
Merge pull request #48217 from cyx1231st/wip-crimson-buffer
crimson: create buffer from temporary_buffer with foreign-ptr by default

Reviewed-by: Samuel Just <sjust@redhat.com>
2022-09-26 09:23:35 +08:00
zdover23
1cb996d4b1
Merge pull request #48235 from zdover23/wip-doc-2022-09-25-dev-guide-basic-workflow-fork-note-formatting
doc/dev: improve presentation of note (git remote)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2022-09-25 14:36:11 +10:00
Zac Dover
b8472b98e2 doc/dev: improve presentation of note (git remote)
This commit corrects the formatting of a note that previously
did not make console output and commands as clear as this commit
makes them.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
2022-09-25 13:41:35 +10:00
Venky Shankar
5f2e47ba21
Merge pull request #47767 from batrick/i57249
mds: correct storage of multiple damaged dentries in DamageTable

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
2022-09-23 16:00:25 +05:30
Lucian Petrut
9921c8a355
Merge pull request #48148 from ktdreyer/win-remove-curl
win32: remove libcurl
2022-09-23 10:43:44 +03:00
Venky Shankar
9016dcf085
Merge pull request #47080 from dparmar18/wip-dparmar-MDS-56522
mds/Server: Do not abort MDS on unknown messages

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
2022-09-23 12:45:40 +05:30
Tobias Urdin
31e5d89377 qa/suites/rgw: blocklist object expiry tempest tests
Signed-off-by: Tobias Urdin <tobias.urdin@binero.se>
2022-09-23 06:31:06 +00:00
Tobias Urdin
7c460bd2fb qa/tasks/barbican: sync barbican db and secret stores
Signed-off-by: Tobias Urdin <tobias.urdin@binero.se>
2022-09-23 06:31:06 +00:00
Tobias Urdin
cd3313531f qa/tasks/barbican: copy barbican policy json or yaml
Signed-off-by: Tobias Urdin <tobias.urdin@binero.se>
2022-09-23 06:31:06 +00:00
Tobias Urdin
8310b179f3 qa/suites/rgw/tempest: bump openstack versions
Update the versions used for OpenStack Keystone,
Barbican and Tempest projects.

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>
2022-09-23 06:31:06 +00:00
Yingxin Cheng
6f44a21e6d crimson: create buffer from temporary_buffer with foreign-ptr by default
temporary_buffer is internally shareable with a thread-unsafe
ref-counter, we need to make sure it is released in the same core where
it is constructed.

Users that need the extra efficiency can swap to create_local as needed.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2022-09-23 14:03:42 +08:00