OSD: await_reserved_maps() prior to calling mark_down

send_message_osd_cluster() et al are *trying* to protect their Connection
lookups (and not re-open zapped Connections) via map reservations, but
that only works if we know that we haven't already called mark_down() on
the entities they might be looking up. So we need to await_reserved_maps
before we do any mark_down calls.
Since the waiting might take some time (fast dispatch in progress), only do
so if we are actually going to mark somebody down.

Fixes: #8512

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

src/osd/OSD.cc

@@ -5944,12 +5944,17 @@ void OSD::handle_osd_map(MOSDMap *m)
     service.pre_publish_map(newmap);
 
     // kill connections to newly down osds
+    bool waited_for_reservations = false;
     set<int> old;
     osdmap->get_all_osds(old);
     for (set<int>::iterator p = old.begin(); p != old.end(); ++p) {
       if (*p != whoami &&
 	  osdmap->have_inst(*p) &&                        // in old map
 	  (!newmap->exists(*p) || !newmap->is_up(*p))) {  // but not the new one
+        if (!waited_for_reservations) {
+          service.await_reserved_maps();
+          waited_for_reservations = true;
+        }
 	note_down_osd(*p);
       }
     }