Commit Graph

99631 Commits

Author SHA1 Message Date
Sage Weil
81f7edc6bd osd: sync old purged_snaps on startup after upgrade or osd creation
This path only triggers after an upgrade or osd creation, when
purged_snaps_last < current_epoch.  When that happens, we slurp down the
old purged snaps so that we have a full history recorded locally.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:50 -05:00
Sage Weil
87b539c2b6 osd: record purged_snaps when we store new maps
When we get a new map, record the (new) purged_snaps.

Only do this if the OSD has purged_snaps that are in sync with the latest
OSDMap.  That means that after an upgrade, if the OSD didn't sync the
old purged_snaps on startup, it won't sync anything until it *next* starts
up.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:50 -05:00
Sage Weil
7315d3fdba mon/OSDMonitor: add messages to get past purged_snaps
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:50 -05:00
Sage Weil
4e5093cee3 mon/OSDMonitor: record pre-octopus purged snaps with first octopus map
When we public our first require_osd_release >= octopus osdmap, record
all prior purged snaps in a key linked to the previous osdmap.  We assume
this will encode and fit into a single key and transaction because the
even larger set of removed_snaps is already a member of pg_pool_t, which
is included in every osdmap.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:50 -05:00
Sage Weil
47fb89c072 mon/OSDMonitor: record purged_snaps for each epoch
Only do this if the mons are all running octopus (and thus there is also
a record for all the pre-octopus purged snaps).

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:50 -05:00
Sage Weil
847dc5fb78 mon/OSDMonitor: make_snap_epoch_key -> make_removed_snap_epoch_key
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:50 -05:00
Sage Weil
3217ca77cb osd/osd_types: add purged_snaps_last to OSDSuperblock
Make this 0 for old encodings or new superblocks.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:50 -05:00
Sage Weil
61d8407514 osd/osd_types: clean up initial values for OSDSuperblock
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:50 -05:00
Sage Weil
acd7e903d3 mon/OSDMonitor: make {removed,purged}_snap storage more efficient
Merge adjacent extents to keep the database smaller and lookups more
efficient.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:50 -05:00
Sage Weil
1e7718c37c mon/OSDMonitor: move (removed, purged) snap update into a helper
This is trivial at the moment, but we'll make it clever next.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:50 -05:00
Sage Weil
c0233801a8 mon/OSDMonitor: generalize/refactor lookup_*_snap
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:50 -05:00
Sage Weil
c0f3624345 mon/OSDMonitor: refactor snap key and value helpers
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:50 -05:00
Sage Weil
19a590e114 mon/OSDMonitor: make_snap_key -> make_removed_snap_key, make_purged_snap_key
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:50 -05:00
Sage Weil
441f42b8ad mon/OSDMonitor: fix lookup_purged_snap implementation
- look at purged, not removed snap keys
- fix the key check to look at the *key name* prefix, not the overall
  prefix (the one implemented by the KeyValueDB interface).

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:50 -05:00
Sage Weil
a12d80a813 mon/OSDMonitor: lookup_pruned_snap -> lookup_purged_snap
This confused me and sent me into the weeds.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:50 -05:00
Sage Weil
94ebe0eab9 osd: adjust snapmapper keys on first start as octopus
Convert snapmapper keys to the new form the first time we start up running
octopus.

This is an incompat feature--once you start as octopus you can't go back.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
84bea65d3e osd/SnapMapper: include poolid in snap index
We want to sort starting with (pool, snapid, ...) so that we align with
the structure of the purged_snaps.  Simply flattening all snaps across
pools is less than ideal because the purge records are intervals (with the
snap in the key the last snap for the interval); flattening means we'd have
to look at many records (across pools) to conclude anything.  Putting
these in the form we really want them simplifies things going forward.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
14fdb52c50 mon/OSDMonitor: document osd snap metadata format
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
e8a359814c osd/SnapMapper: document stored keys and values
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
ec4cd08261 mon/OSDMonitor: use structured binding for prepare_remove_snaps
No functional change

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
e457b73075 mon/OSDMonitor: send MRemoveSnaps back to octopus MDS
The octopus MDS wants an explicit ack when snaps are removed.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
ab405ab560 mds/SnapServer: handle MRemoveSnaps acks from mon
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
7eeee5b49b CMakeLists: include 'cephfs' (which includes libcephfs) in 'vstart' target
This ensures 'make vstart' will build libcephfs, which lets the mgr volumes
module start, such that we can successfully vstart.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
bc4c31a439 mon/PaxosService: add C_ReplyOp
Generic helper context to reply to an op on (successful) finish.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
75edd21c45 vnewosd.sh: add script to add a new osd to an existing vstart
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
dcc8b6b28a vstart.sh: remove useless auth add for osds
'osd new' installs the key.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
d056372e7b vstart.sh: wait for mgr volume module to start up
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
fa7655351c mon/OSDMonitor: make snap removal handle dups safely
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
b945de141f mon/OSDMonitor: only update removed_snaps when pre-octopus
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
7fe43cc4e4 ceph_test_rados: stop doing long object names
This was there to test filestore's long file name handling, which (1)
works, and (2) we don't care that much about anymore.  Meanwhile, the
long names make the OSD log files *really* painful to read.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
419251a401 ceph_test_rados_api_tier_pp: fix osd version checks
These need to exclude old releases but run the test for newer, unnamed
releases.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
f58f5cfc51 osd/PrimaryLogPG: use get_ssc_as_of for snapc for flushing clones
We will stop maintaining SnapSet::snaps shortly.  Instead, generate this
snapc using the existing SnapSet::get_ssc_as_of() method, which will now
derive the snap list from the clone_snaps member.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
d7c343eebf osd/PrimaryLogPG: only maintain SnapSet::snaps for pre-octopus compat
Once we have all octopus or newer OSDs, we can stop maintaining
SnapSet::snaps.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
de5d3f6a0a mon/OSDMonitor: only maintain pg_pool_t::removed_snaps for pre-octopus
- do not examine removed_snaps
- do not add new items to removed_snaps unless we need pre-octopus compat
- clear removed_snaps in first octopus epoch

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:49 -05:00
Sage Weil
8f714e2ba6 osd/osd_types: mark SnapSet::snaps as legacy
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:48 -05:00
Sage Weil
f43d214789 osd/osd_types: SnapSet::get_ssc_as_of: use clone_snaps
Fabricate a SnapContext from the clone_snaps instead of snaps (which we
are trying to kill).

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:48 -05:00
Sage Weil
a362fedc81 osd/PrimaryLogPG: change fabrication of promoted clone snaps
Instead of using SnapSet::snaps (which we are trying to kill), use the
clone_snaps[] snaps.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:48 -05:00
Sage Weil
0964498132 osd/PrimaryLogPG: only filter SnapSet::snaps for flush for pre-octopus compat
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:48 -05:00
Sage Weil
c88d860dce osd/PrimaryLogPG: trim_objects: only filter SnapSet::snaps for pre-octopus
For octopus and later, we don't need SnapSet::snaps.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:48 -05:00
Sage Weil
b1a5bff4a8 osd/PrimaryLogPG: make best effort to sanitize clones on copy-from
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:48 -05:00
Sage Weil
8c0c63c7e1 mds/SnapServer: int -> int32_t for encoded type
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:48 -05:00
Sage Weil
25b690fe06 messages/MRemoveSnaps: int -> int32_t on encoded type
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:48 -05:00
Sage Weil
a6f85089c0 osd/PrimaryLogPG: find_object_context: trust SnapSet's clone_snaps
Instead of checking the OSDMap pg_pool_t whether a snap exists, instead

1- Look at the clone_snaps more carefully.  If the snap didn't exist when
the clone was last touched (created or partially-trimmed) then it still
doesn't exist now (snaps aren't resurrected).

2- Check in the OSDMap's removed snaps queue.  This will catch anything
that is still being removed but hasn't been reflected by the clone_snaps
yet.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:48 -05:00
Sage Weil
cabc48c403 osd/PrimaryLogPG: use osdmap removed_snaps_queue for snap trimming
No need to use the pg_pool_t member now--the osdmap has a queue
specifically for the snaps we are in the process of trimming.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:48 -05:00
Sage Weil
3f9c28823a mon/OSDMonitor: avoid is_removed_snap()
Instead, consult the OSDMap's removed_snap_queue and the mon's record of
purged snaps.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:48 -05:00
Sage Weil
9b6acc2cad osd/PeeringState: drop some mimic conditionals
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:48 -05:00
Sage Weil
b59a25d9c9 osd/PG: drop pre-mimic snap_trimq code
This is no longer needed for octopus.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:48 -05:00
Sage Weil
e963ee6039 osd/PeeringState: removed pre-mimic removed snap tracking
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:48 -05:00
Sage Weil
58c4c8fc8e osd: move snap_interval_set_t to osd_types
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:48 -05:00
Sage Weil
99970bb795 mon: drop mon_debug_no_require_mimic
Signed-off-by: Sage Weil <sage@redhat.com>
2019-07-02 08:37:48 -05:00