PendingReleaseNotes: some notes about upgrade and ceph user

Signed-off-by: Sage Weil <sage@redhat.com>

PendingReleaseNotes

@@ -4,6 +4,34 @@ v9.1.0
 v9.0.4
 ======
 
+Upgrading
+---------
+
+* Ceph daemons now run as user and group ceph by default.  During upgrade,
+  administrators have two options:
+
+   #. Add the following line to ``ceph.conf`` on all hosts::
+
+        setuser match path = /var/lib/ceph/$type/$cluster-$id
+
+      This will make the daemon remain root (i.e., not drop privileges and
+      switch to user ceph) if the daemon's data directory is still owned by
+      root.  Newly deployed daemons will be created with data owned by user
+      ceph and will run with reduced privileges, but upgraded daemons will
+      continue to run as root.
+
+   #. Fix the data ownership during the upgrade.  This is the preferred option,
+      but is more work.  The process for each host would be to:
+
+      #. Upgrade the ceph package.  This creates the ceph user and group.
+      #. Stop the daemon(s)
+      #. Fix the ownership.  E.g.,::
+
+          chown -R ceph:ceph /var/lib/ceph/mon/ceph-foo
+	  ceph-disk chown /dev/sdb1
+
+      #. Restart the daemon(s)
+
 * The on-disk format for the experimental KeyValueStore OSD backend has
   changed.  You will need to remove any OSDs using that backend before you
   upgrade any test clusters that use it.