objecter: use ordered map<> for tracking tids to preserve order on resend

We are using a hash_map<> to map tids to Op*'s.  In handle_osd_map(),
we will recalc_op_target() on each Op in a random (hash) order.  These
will get put in a temp map<tid,Op*> to ensure they are resent in the
correct order, but their order on the session->ops list will be random.

Then later, if we reset an OSD connection, we will resend everything for
that session in ops order, which is be incorrect.

Fix this by explicitly reordering the requests to resend in
kick_requests(), much like we do in handle_osd_map().  This lets us
continue to use a hash_map<>, which is faster for reasonable numbers of
requests.  A simpler but slower fix would be to just use map<> instead.

This is one of many bugs contributing to #2947.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

src/osdc/Objecter.cc

@@ -758,21 +758,31 @@ void Objecter::kick_requests(OSDSession *session)
   ldout(cct, 10) << "kick_requests for osd." << session->osd << dendl;
 
   // resend ops
+  map<tid_t,Op*> resend;  // resend in tid order
   for (xlist<Op*>::iterator p = session->ops.begin(); !p.end();) {
     Op *op = *p;
     ++p;
     logger->inc(l_osdc_op_resend);
     if (op->should_resend) {
-      send_op(op);
+      resend[op->tid] = op;
     } else {
       cancel_op(op);
     }
   }
+  while (!resend.empty()) {
+    send_op(resend.begin()->second);
+    resend.erase(resend.begin());
+  }
 
   // resend lingers
+  map<uint64_t, LingerOp*> lresend;  // resend in order
   for (xlist<LingerOp*>::iterator j = session->linger_ops.begin(); !j.end(); ++j) {
     logger->inc(l_osdc_linger_resend);
-    send_linger(*j);
+    lresend[(*j)->linger_id] = *j;
+  }
+  while (!lresend.empty()) {
+    send_linger(lresend.begin()->second);
+    lresend.erase(lresend.begin());
   }
 }