Commit Graph

72244 Commits

Author SHA1 Message Date
Sage Weil
209a4b8f3a Merge pull request #14958 from ovh/bp-fix-finisher-warning
common/Finisher: fix uninitialized variable warning
common/Finisher: fix uninitialized variable warning

Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-05-05 17:42:26 -05:00
Sage Weil
7f72100be5 Merge pull request #14793 from dmick/wip-prio-and-daemonperf
ceph: perfcounter priorities and daemonperf updates to use them
2017-05-05 17:41:55 -05:00
Sage Weil
7c8a2a1766 Merge pull request #14510 from liewegas/wip-bluefs-sync-write
os/bluestore/BlueFS: add bluefs_sync_write option

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
2017-05-05 17:41:22 -05:00
Sage Weil
425769fc10 Merge pull request #14305 from liewegas/wip-zipkin-partial
osd: zipkin tracing
2017-05-05 17:40:58 -05:00
Sage Weil
c5a46089f0 messages/MCommand: fix type on decode
Wow, this has been broken since v0.38, but apparently
the message never made it into the object corpus so
we never noticed!

In reality the bug is harmless: decode_message() will
set_header which clobbers whatever version the default
ctor fills in, so this only affects ceph-dencoder's
test.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 18:02:45 -04:00
Sage Weil
0eb0694002 ceph-object-corpus: kraken objects
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 18:02:45 -04:00
Dan Mick
b37d0b49da test/osd/osd-dup.sh: lower wb fd throttle limits
osd-dup.sh was losing OSDs to EMFILE, mostly on arm64 (not clear why).

Signed-off-by: Dan Mick <dan.mick@redhat.com>
2017-05-05 14:47:14 -07:00
Orit Wasserman
0c5b2a9026 Merge pull request #14981 from cbodley/wip-rgw-bi-list-plain
cls/rgw: list_plain_entries() stops before bi_log entries
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2017-05-06 00:04:11 +03:00
Sage Weil
b89dff472c test/encoding/generate-corpus-objects: fix typo
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 16:31:02 -04:00
Casey Bodley
b29a1633a5 cls/rgw: list_plain_entries() stops before bi_log entries
list_plain_entries() was using encode_obj_versioned_data_key() to set
its end_key, which gives a prefix of BI_BUCKET_OBJ_INSTANCE_INDEX[=2]

that range between start_key and end_key would not only span the
BI_BUCKET_OBJS_INDEX[=0] prefixes, but BI_BUCKET_LOG_INDEX[=1] prefixes
as well. this can result in list_plain_entries() trying and failing to
decode a rgw_bi_log_entry as a rgw_bucket_dir_entry

Fixes: http://tracker.ceph.com/issues/19876

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 15:48:20 -04:00
Sage Weil
b38b663a71 test/encoding/generate-corpus-objects.sh: simplify object corpus population
Script that generates corpus objects.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 15:20:18 -04:00
Sage Weil
40fe1d181c qa/workunits/rgw/run-s3tests.sh: run s3-tests
This works out of the box with a vstart environment and

 RGW=1 ../src/vstart.sh -n -l
 PATH=bin:$PATH ../qa/workunits/rgw/run-s3tests.sh

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 15:20:18 -04:00
Jason Dillaman
128f593fe8 Merge pull request #14051 from SpandanKumarSahu/bug#17195
rbd: stop indefinite thread waiting in krbd udev handling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2017-05-05 14:40:55 -04:00
Sage Weil
e8ab3fdc87 os/bluestore: remove useless IOContext::num_reading
If we are a syncrhonous read, we don't need this: we don't aio_wait for
sync reads.  If we are an aio_read, we are in the aio_running count anyway,
and there is also no purpose for this counter.

I'm a bit unsure about the NVME use of this counter; I switched it to use
num_running (pretty sure we aren't mixing reads and writes on a single
IOContext) *but* it might make more sense to switch to a private counter.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 14:39:30 -04:00
Sage Weil
1026593173 os/bluestore/BlockDevice: fix racy aio_{wake,wait}() on IOContext
Thread 1 (_do_read)                 Thread 2 (_aio_thread)
queues aio
ioc->aio_wait()
  locks ioc->lock
  num_waiting++
                                    finishes aios
                                    ioc->aio_wake
                                      if (num_waiting)
                                        lock ioc->lock (blocks)
  num_running == 0, continue
  ioc->unlock
                                        ioc->lock taken
do_read destroys ioc
                                        use after free, may block forever

The last bit of the race may vary; the key thing is that thread 2 is
taking a lock that thread 1 can destroy; thread 2 doesn't have it pinned
in memory.

Fix this by simplifying the aio_wake, aio_wait.  Since it is mutually
exclusive with a callback completion, we can avoid calling it at all when
a callback in present, and focus on keeping it simple.

Avoid use-after-free by making sure the last decrement happens under
the lock in the aio_wake/wait case.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 14:39:30 -04:00
Sage Weil
5457991143 os/bluestore/KernelDevice: remove weird aio_wake from sync read()
I'm not entirely sure why we were ever signaling the cond here; the read
is synchronous!

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 14:39:29 -04:00
Sage Weil
b4eab62c5e CMakeLists.txt: WITH_BLKIN off for now
We are unconditionally linking lttng-ust, which make selinux complain. We
should either

 - fix selinux rules and unconditionally link
 - dlopen at runtime based on an option (like we do for the current
   tracepoints)
 - ???

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 14:05:24 -04:00
Sage Weil
3eb554c1fc blkin: fix test
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 14:05:24 -04:00
Sage Weil
32d3119696 blkin: fix cmake compile options
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 14:05:24 -04:00
Sage Weil
90e52907be blkin: remove installed files
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 14:05:24 -04:00
Sage Weil
91d270abf8 blkin: fix -fPIC option
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 14:05:24 -04:00
Sage Weil
4ea342ca47 blkin: drop boost dependency
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 14:05:24 -04:00
Sage Weil
838dd2a603 common/zipkin_trace: fix blkin_trace_info encoding
- encode the same regardless of whether it is compiled in (!)
- encode in endian- and struct alignment-safe way.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 14:05:24 -04:00
Sage Weil
25eb14247d msg/Message: make {encode,decode}_trace unconditional
These are protocol features and cannot vary based on our compilation.
Encode and decode unconditionally.  The callers have already guarded these
field additions with a message version bump and are conditionally calling
decode_trace.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 14:05:24 -04:00
Sage Weil
60baa90c29 CMakeLists.txt: fix libcommon link to blkin
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 14:05:24 -04:00
Sage Weil
bdd6c3f382 CMakeLists.txt: default WITH_BLKIN=ON
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 14:05:24 -04:00
Casey Bodley
26678e673a cmake: build blkin instead of using find_package
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 14:05:24 -04:00
Sage Weil
bbe6d24184 src/blkin: add blkin submodule
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 14:05:24 -04:00
Victor Araujo
e0a1532bfb librados: Overload aio_operate with blkin trace info parameter
This will enable passing trace information from rbd to librados for
aio_reads and aio_writes.

Signed-off-by: Victor Araujo <ve.ar91@gmail.com>
2017-05-05 14:05:24 -04:00
vears91
046b2bde6d cmake: Add blkin libs to target_link_libraries
Signed-off-by: Victor Araujo <ve.ar91@gmail.com>
2017-05-05 14:05:24 -04:00
Casey Bodley
ffdfe6996f blkin: add config variable osdc_blkin_trace_all
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 14:05:24 -04:00
Casey Bodley
f453eafe8f blkin: add traces to librados and Objecter
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 14:05:24 -04:00
Casey Bodley
4516d8bf3c blkin: add config variable osd_blkin_trace_all
when set, Message::decode_trace() will always create a trace for
incoming messages, even if they didn't pass trace information

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 14:05:24 -04:00
Victor Araujo
e3524e6977 filestore: Initialize Zipkin traces when there is a store trace
Avoids creating orphaned traces.

Signed-off-by: Victor Araujo <ve.ar91@gmail.com>
2017-05-05 14:05:23 -04:00
Casey Bodley
e0a5b51fad blkin: add traces to FileStore/Journal
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 14:05:23 -04:00
Sage Weil
7d4af0531d blkin: add traces to ECBackend
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 14:05:23 -04:00
Casey Bodley
6586e724a0 blkin: add traces to ReplicatedBackend
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 14:04:51 -04:00
Casey Bodley
bf8772ff8e blkin: set up tracing in PGs
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 14:04:51 -04:00
Casey Bodley
a39b8c466e blkin: set up tracing in the OSD
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 14:04:51 -04:00
Casey Bodley
6159027fda blkin: osd op messages carry trace information
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 14:04:30 -04:00
Casey Bodley
679f8e6eaf blkin: add traces to AsyncMessenger
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 13:59:47 -04:00
Casey Bodley
5cc7c06245 blkin: add traces to XioMessenger
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 13:59:47 -04:00
Casey Bodley
82160fb591 blkin: add traces to SimpleMessenger
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 13:59:47 -04:00
Casey Bodley
612d15bc1b blkin: Messenger integration
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 13:59:46 -04:00
Casey Bodley
c8107a3e1f msg: pass Connection to decode_message
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 13:59:46 -04:00
Casey Bodley
6862811ee2 blkin: add header and initialization
zipkin_trace.h is a wrapper around ztracer.hpp, which provides a stub
implementation when WITH_BLKIN is not defined

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 13:59:46 -04:00
Casey Bodley
5ec55b9daa cmake: add module and WITH_BLKIN option for blkin
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 13:59:46 -04:00
Andrew Shewmaker
1d3255a971 Build support for BlkKin (LTTng + Zipkin) tracing
* Adds --with-blkin to autoconf (default without)
 * Adds BLKIN_LIBS to necessary automake files
 * Adds documentation for testing BlkKin tracing

Signed-off-by: Andrew Shewmaker <agshew@gmail.com>
Signed-off-by: Marios-Evaggelos Kogias <marioskogias@gmail.com>
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 13:59:46 -04:00
Sage Weil
7104a4e8da osd/PG: debug scrub range
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:39:14 -04:00
Sage Weil
73f8fb9976 qa/suites/upgarde/jewel-x: add final scrub and legacy snapset check
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:39:14 -04:00