Commit Graph

30563 Commits

Author SHA1 Message Date
Sage Weil
baa74d522b osd/OSDMap: implement propapage_snaps_to_tiers()
Tier pools mirror the base pool's snapshot metadata.

Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-22 09:49:24 -08:00
Sage Weil
cf34af8bd2 rgw: add -ldl for mongoose
/usr/bin/ld: mongoose/mongoose.o: undefined reference to symbol 'dlsym@@GLIBC_2.2.5'
/lib/x86_64-linux-gnu/libdl.so.2: error adding symbols: DSO missing from command line
error: collect2: ld returned 1 exit status

Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-22 09:48:28 -08:00
Sage Weil
e6f9fb5b32 Merge pull request #980 from ceph/port/misc
Misc portability patches

Reviewed-by: Sage Weil <sage@inktank.com>
2013-12-22 09:34:12 -08:00
Sage Weil
13df29e318 Merge pull request #981 from dachary/wip-rep-replicated
replace pool type REP with REPLICATED

Reviewed-by: Sage Weil <sage@inktank.com>
2013-12-21 23:43:38 -08:00
Sage Weil
9b182c700a ceph_test_rados_api_tier: more grace for HitSetTrim
Saw this test fail due to ill-timed thrashing:

 /a/teuthology-2013-12-20_23:00:02-rados-master-testing-basic-plana/10941

Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-21 23:40:14 -08:00
Sage Weil
2bb90b38e8 ceph_test_rados: update in-memory user_version on RemoveAttrsOp
Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-21 23:33:51 -08:00
Loic Dachary
a0d1521ddb replace pool type REP with REPLICATED
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-12-22 08:15:24 +01:00
Sage Weil
45449b880e doc/release-notes: missed a name
Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-21 21:42:43 -08:00
Sage Weil
b0e42d8bd4 doc/release-notes: v0.72.2
Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-21 21:33:23 -08:00
Noah Watkins
c10ba91ece pipe: add compat for TEMP_FAILURE_RETRY symbol
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2013-12-21 13:13:09 -06:00
Noah Watkins
f13709d28f linux_version: build on all platforms
This linux version check is used in FileJournal to check about write
caching behavior. This is a temporary fix that will result in the
failure path and a warning about writing caching being turned on until
methods for OSX/FreeBSD/Windows can be found to find the same
information.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2013-12-21 13:13:09 -06:00
Noah Watkins
79c5a2e8c8 make: add libcommon for missing symbols
On OSX without linking in libcommon at the end of these make targets
there is a missing reference to pipe_cloexec, even though the dependency
is present indirectly through libglobal.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2013-12-21 13:13:09 -06:00
Noah Watkins
ab19c5f9b2 buffer: remove darwin portability cruft
valloc conflicts with an existing call, and none of these macros are
actually used in buffer.h. The DARWIN check isn't valid either since
this is an installed header and that depends on acconfig.h

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2013-12-21 08:41:37 -06:00
Noah Watkins
cf9569f399 statfs: include headers for statfs structs
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2013-12-20 15:44:11 -08:00
Noah Watkins
3510400da9 compat: enable lseek64 alias
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2013-12-20 15:44:11 -08:00
Noah Watkins
9d72c4f8ff libcephfs: ignore missing offset64 definition
on apple.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2013-12-20 15:43:48 -08:00
Sage Weil
7b98862dd5 Merge pull request #977 from ceph/wip-kill-raid4
osd: remove remaining instances of raid4 pool types (never implemented)

Reviewed-by: Loic Dachary <loic@dachary.org>
2013-12-20 15:20:23 -08:00
Yehuda Sadeh
6afaf48674 Merge pull request #975 from BCLibCoop/bclibcoop/rgw_cors
RGW: CORS use the correct headers for checking, and validate headers as lowercase where needed

Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>
2013-12-20 14:26:41 -08:00
Sage Weil
d9299fd447 Merge pull request #979 from dachary/wip-wrapped-vstart-errors
vstart_wrapped_tests must fail if one test fail 

Reviewed-by: Sage Weil <sage@inktank.com>
2013-12-20 14:19:05 -08:00
Sage Weil
7bacc413f3 Merge pull request #818 from ceph/wip-rgw-standalone-2
Wip rgw standalone 2

Reviewed-by: Sage Weil <sage@inktank.com>
2013-12-20 13:41:58 -08:00
Sage Weil
cdc178fb9b Revert "Enable libs3 support for debian packages"
This reverts commit 8814265f08.

Or not!  This adds a build-time dependency which none of the gitbuilders
have, so scrap it.
2013-12-20 13:14:08 -08:00
Loic Dachary
1e238e6f95 mon: pool create will not fail if the type differs
It looked like it worked because the wrapper hide the error. The failing
tests are commented out so that the other tests can be used.

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-12-20 22:00:31 +01:00
Sage Weil
5f1957ded2 doc/release-notes: v0.67.5
Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-20 12:53:03 -08:00
Loic Dachary
98a1525307 unittests: fail if one test fail
vstart_wrapped_tests must return on error if one of the tests
fail.

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-12-20 21:13:00 +01:00
Sage Weil
f86c3c0775 Merge pull request #460 from toabctl/build-depends
Enable libs3 support for debian packages

Reviewed-by: Sage Weil <sage@inktank.com>
2013-12-20 10:48:11 -08:00
Sage Weil
eb7ed58c4c Merge pull request #916 from ceph/port/buffer
buffer: use int64_t instead of loff_t

Reviewed-by: Sage Weil <sage@inktank.com>
2013-12-20 10:29:03 -08:00
Noah Watkins
9ab947c7f6 buffer: use int64_t instead of loff_t
Because portability.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2013-12-20 08:21:43 -08:00
Loic Dachary
e04d7b8743 Merge pull request #933 from dachary/wip-erasure-code-benchmark
osd: erasure code benchmark tool

Reviewed-by: Andreas Peters <andreas.joachim.peters@cern.ch>
Reviewed-by: Christophe Courtaut <christophe.courtaut@gmail.com>
2013-12-20 03:24:10 -08:00
Loic Dachary
03693aceae osd: git ignore erasure code benchmark binary
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-12-20 12:15:44 +01:00
Loic Dachary
42b4fe1496 osd: erasure code benchmark is installed is part of ceph-test
Add to the packaging for RPMs and DEBs

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-12-20 12:15:44 +01:00
Loic Dachary
81dee1b67f osd: erasure code benchmark workunit
Display benchmark results for the default erasure code plugins, in a tab
separated CSV file. The first two column contain the amount of KB
that were coded or decoded, for a given combination of parameters
displayed in the following fields.

seconds	KB	plugin	k	m	work.	iter.	size	eras.
1.2	10	example	2	1	encode	10	1024	0
0.5	10	example	2	1	decode	10	1024	1

It can be used as input for a human readable report. It is also intented
to be used to show if a given version of an erasure code plugin performs
better than another.

The last column ( not shown above for brievety ) is the exact command
that was run to produce the result so it can be copy / pasted to
reproduce them or to profile.

Only the jerasure techniques mentionned in
https://www.usenix.org/legacy/events/fast09/tech/full_papers/plank/plank_html/
are benchmarked, the others are assumed to be less interesting.

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-12-20 12:15:44 +01:00
Loic Dachary
a619fe9869 osd: erasure code benchmark tool
Implement the ceph_erasure_code_benchmark utility to:

* load an erasure code plugin

* loop over the encode/decode function using the parameters from the
  command line

* print the number of bytes encoded/decoded and the time to process

When decoding, random chunks ( as set with --erasures ) are lost on each
run.

For instance:

    $ ceph_erasure_code_benchmark \
       --plugin jerasure \
       --parameter erasure-code-directory=.libs \
       --parameter erasure-code-technique=reed_sol_van \
       --parameter erasure-code-k=2 \
       --parameter erasure-code-m=2 \
       --workload decode \
       --erasures 2 \
       --iterations 1000
    0.964759	1048576

shows 1GB is decoded in 1second.

It is intended to be used by other scripts to present a human readable
output or detect performance regressions.

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-12-20 12:15:13 +01:00
Loic Dachary
a36bc5fabf osd: set erasure code packet size default to 2048
As shown in
https://www.usenix.org/legacy/events/fast09/tech/full_papers/plank/plank_html/
under "Impact of the Packet Size", the optimal for is in the order of 1k
rather than the current default of 8. Benchmarks are required to find
the actual optimum.

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-12-20 11:28:47 +01:00
Loic Dachary
c7d8ba7b6d osd: better performances for the erasure code example
The XOR based example is ten times slower than it could because it uses
the buffer::ptr[] operator. Use a temporary char * instead. It performs
as well as jerasure Reed Solomon when decoding with a single erasure:

$ ceph_erasure_code_benchmark \
   --plugin example  --parameter erasure-code-directory=.libs \
   --parameter erasure-code-technique=example \
   --parameter erasure-code-k=2 --parameter erasure-code-m=1 \
   --erasure 1 --workload decode --iterations 5000
8.095007	5GB

$ ceph_erasure_code_benchmark \
   --plugin jerasure  --parameter erasure-code-directory=.libs \
   --parameter erasure-code-technique=reed_sol_van \
   --parameter erasure-code-k=10 --parameter erasure-code-m=6 \
   --erasure 1 --workload decode --iterations 5000
7.870990	5GB

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-12-20 11:28:47 +01:00
Loic Dachary
ff9455baad osd: conditionally disable dlclose of erasure code plugins
When profiling, tools such as valgrind --tool=callgrind require that the
dynamically loaded libraries are not dlclosed so they can collect usage
information.

The public ErasureCodePluginRegistry::disable_dlclose boolean is introduced
for this purpose.

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-12-20 11:28:47 +01:00
David Zafman
8879e43921 osd: Fix assert which doesn't apply when compat_mode on
Signed-off-by: David Zafman <david.zafman@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit edaec9a836)
2013-12-19 21:24:07 -08:00
David Zafman
0bd5cb65ef Add backward comptible acting set until all OSDs updated
Add configuration variable to override compatible acting set handling.
Later we'll check the osdmap that all OSDs are updated to use new acting sets.

Fixes: #6990

Signed-off-by: David Zafman <david.zafman@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 19cff890eb)
2013-12-19 21:23:52 -08:00
Sage Weil
8d31f71b32 osd/ReplicatedPG: fix promote cancellation
The canceling caller cleans up the blocked objects for us; we simply need
to bail out early.

Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-19 16:40:02 -08:00
Sage Weil
923bff1dba osd/ReplicatedPG: drop RepGather::ondone callback
We kick the blocked contexts in the completion path of process_copy_chunk(),
after we have take the RWWRITE obc lock.  There is no need to delay the
unblocking until the RepGather finishes.

This also fixes a leak: the ondone wasn't getting cleaned up if a peering
interval change happens and the repgather is applied early in on_change().

Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-19 16:40:01 -08:00
Sage Weil
2a9c6fcda9 vstart.sh: go faster
Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-19 16:40:01 -08:00
Sage Weil
bc893f55d8 osd/ReplicatedPG: fix undirty on clean object
Return success, but do not screw up the stats.

Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-19 16:40:01 -08:00
Sage Weil
c561d5ea22 osd/ReplicatedPG: track dirty, whiteout stat counts
These counts will be useful (even necessary!) for the cache agent, and are
generally interesting to the admin as well.

Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-19 16:40:01 -08:00
Sage Weil
028bb0d1e2 osd/osd_types: include num_objects_dirty, num_whiteouts in object_stat_sum_t
Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-19 16:40:01 -08:00
Sage Weil
1403798327 osd/ReplicatedPG: EBUSY on cache-evict when watchers are present
Linger operations will follow the object to the cache pool when the pool
overlay process is set.  If we evict the object, the object_info_t will
go away along with the watch state and confusing things will happen.
Prevent that from happening by returning EBUSY when you try to evict a
watched object.

Note that you *can* flush a watched object, and the dirty flag will be
cleared.  But you still can't evict it.

Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-19 16:40:01 -08:00
Sage Weil
9ed6679a94 ceph_test_rados: test cache_flush, cache_try_flush, cache_evict
Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-19 16:40:00 -08:00
Sage Weil
1dcbb663d8 ceph_test_rados_api_tier: fix HitSet* test names
Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-19 16:40:00 -08:00
Sage Weil
99cee55c8a osd/osd_types: debug: include size in object_info_t operator<<
Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-19 16:40:00 -08:00
Sage Weil
1af6723cdc osd/ReplicatedPG: debug: clean up oi printout
Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-19 16:40:00 -08:00
Sage Weil
0d2d6a5faa osd/ReplicatedPG: debug: add an assert for copy-get
Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-19 16:40:00 -08:00
Sage Weil
2fda4c0161 osd/ReplicatedPG: fix locking for promote
After we get the copy-from data and unblock the obc, we still need to take
the RWWRITE lock on the object for the duration of the repop while we
actually apply the change locally.

Signed-off-by: Sage Weil <sage@inktank.com>
2013-12-19 16:40:00 -08:00