Commit Graph

55931 Commits

Author SHA1 Message Date
Ramana Raja
1c1d65a45f ceph_volume_client: version on-disk metadata
Version on-disk metadata with two attributes,
'compat version', the minimum CephFSVolume Client
version that can decode the metadata, and
'version', the version that encoded the metadata.

Signed-off-by: Ramana Raja <rraja@redhat.com>
2016-07-18 16:38:32 +05:30
Ramana Raja
46876fb2ce ceph_volume_client: add versioning
Add class attributes to CephFSVolumeClient to version
its capabilities.

'version' attribute stores the current version number
of CephFSVolumeClient.

'compat_version' attribute stores the earliest version
number of CephFSVolumeClient that the current version is
compatible with.

Fixes: http://tracker.ceph.com/issues/15406

Signed-off-by: Ramana Raja <rraja@redhat.com>
2016-07-18 16:38:32 +05:30
Ramana Raja
82445a20a2 ceph_volume_client: disallow tenants to share auth IDs
Restrict an auth ID to a single OpenStack tenant to enforce
strong tenant isolation of shares.

Signed-off-by: Ramana Raja <rraja@redhat.com>
2016-07-18 16:38:32 +05:30
Ramana Raja
ec2e6e37d0 ceph_volume_client: cleanup auth meta files
Remove auth meta files on last rule for an auth ID deletion

Signed-off-by: Ramana Raja <rraja@redhat.com>
2016-07-18 16:38:32 +05:30
Ramana Raja
7731287761 ceph_volume_client: fix log messages
Log the path of the volume during creation and deletion of volumes.

Signed-off-by: Ramana Raja <rraja@redhat.com>
2016-07-18 16:38:32 +05:30
Ramana Raja
37fbfc7aa8 ceph_volume_client: create/delete VMeta for create/delete volume
Create and delete volume meta files during creation and deletion of
volumes.

Signed-off-by: Ramana Raja <rraja@redhat.com>
2016-07-18 16:38:30 +05:30
Ramana Raja
f7c037229b ceph_volume_client: modify locking of meta files
File locks are applied on meta files before updating the meta
file contents. These meta files would need to be cleaned up
sometime, which could lead to locks being held on unlinked meta
files. Prevent this by checking whether the file had been deleted
after lock was acquired on it.

Signed-off-by: Ramana Raja <rraja@redhat.com>
2016-07-18 15:36:40 +05:30
Ramana Raja
f58403f3d1 cephfs.pyx: implement python bindings for fstat
Signed-off-by: Ramana Raja <rraja@redhat.com>
2016-07-18 15:36:40 +05:30
Ramana Raja
7f7d2a76ae ceph_volume_client: restrict volume group names
Prevent craftily-named volume groups from colliding with meta files.

Signed-off-by: Ramana Raja <rraja@redhat.com>
2016-07-18 15:36:40 +05:30
Ramana Raja
27eb51baab ceph_volume_client: use fsync instead of syncfs
Signed-off-by: Ramana Raja <rraja@redhat.com>
2016-07-18 15:36:40 +05:30
Ramana Raja
647a2447f0 ceph_volume_client: recover from dirty auth and auth meta updates
Check dirty flag after locking something and call recover() if we are
opening something dirty (racing with another instance of the driver
restarting after failure) -- only required if someone running multiple
manila-share instances with Ceph loaded.

Signed-off-by: Ramana Raja <rraja@redhat.com>
2016-07-18 15:36:40 +05:30
Ramana Raja
7c8a28a7e8 ceph_volume_client: modify data layout in meta files
Notable changes to data layout in auth meta and volume meta files:

In the auth meta files, add a 'dirty' flag to track the status of auth
updates to a single volume.

In the volume meta file, make the 'dirty' flag track the status of
auth updates for a single ID.

Optimize the recovery of partial auth update changes to auth meta,
volume meta, and the Ceph backend, facilitated by changes in the
data layout in the meta files.

Signed-off-by: Ramana Raja <rraja@redhat.com>
2016-07-18 15:36:40 +05:30
John Spray
d2e9eb55ca pybind: ceph_volume_client authentication metadata
Store a two-way mapping between auth IDs and volumes.

Enables us to record some metadata on auth ids (which
openstack tenant created it) so that we can avoid exposing
keys to other tenants who try to use the same ceph
auth id.

Enables us to expose the list of which auth ids have access
to a volume, so that Manila's update_access() can be
implemented efficiently.

DNM: see TODOs inline.

Fixes: http://tracker.ceph.com/issues/15615

Signed-off-by: John Spray <john.spray@redhat.com>
2016-07-18 15:36:40 +05:30
John Spray
5678584f41 pybind: enable integer flags to libcephfs open
The 'rw+' style flags are handy and convenient, but
they don't capture all possibilities.  Change to
optionally accept an integer here for advance users
who want to specify arbitrary combinations of
flags.

Signed-off-by: John Spray <john.spray@redhat.com>
2016-07-18 15:36:40 +05:30
Mark Nelson
cab254e924 os/bluestore: revert preferred csum behavior
This passes "ceph_test_objectstore --gtest_filter=*/2".
This restores 4K random read performance to previous levels when objects
are were previously written out using large IOs (4MB in this case):

pre-patch: 26MB/s
post-pated: 610MB/s

Closes #10320

Signed-off-by: Mark Nelson <mnelson@redhat.com>
2016-07-17 21:33:59 -05:00
Somnath Roy
bf70bcb6c5 Bluestore: Fixed a Bluestore crash
A bluestore race condition is been fixed by protecting txc structures
within _txc_state_poc with collection lock.

Mark's comments:

This fixes segfaults during random write tests with bluestore.
This passes "ceph_test_objectstore --gtest_filter=*/2".
This may introduce a small performance regresion, though there is enough
noise in the results to make it inconclusive.

Closes #10220

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>
2016-07-17 21:26:31 -05:00
Yehuda Sadeh
3e5775475b Merge pull request #10301 from cbodley/wip-rgw-meta-stack-wakeup
rgw: RGWMetaSyncCR holds refs to stacks instead of crs

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
2016-07-15 19:19:57 -07:00
Casey Bodley
e4bc16044e rgw: RGWMetaSyncCR holds refs to stacks for wakeup
because RGWCoroutine::wakeup() calls RGWCoroutinesStack::wakeup(), the
stack must also stay alive

Fixes: http://tracker.ceph.com/issues/16666

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2016-07-15 11:20:13 -04:00
Kefu Chai
4025a2eb80 Merge pull request #10306 from tchaikov/wip-no-mktemp-p
qa/workunits/cephtool/test.sh: s/TMPDIR/TEMP_DIR/

Reviewed-by: Haomai Wang <haomai@xsky.com>
2016-07-15 21:34:02 +08:00
Kefu Chai
bf58aeb6d5 Merge pull request #9972 from ceph/objclass-perm
osd: object class loading and execution permissions

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
2016-07-15 17:25:44 +08:00
Kefu Chai
316c15bc9d Merge pull request #9980 from gaowanlong/split_out_handle_pg_scrub
osd: small cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2016-07-15 17:09:14 +08:00
Kefu Chai
9103958688 Merge pull request #10126 from dx9/wip-fcntl-warns
test/libcephfs: fix gcc sys/fcntl.h warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>
2016-07-15 17:05:15 +08:00
Kefu Chai
56df5dcda4 Merge pull request #10166 from wjwithagen/wip-wjw-cmake-test_rados_tool.sh
test_rados_tool.sh: Make script work under ctest

Reviewed-by: Kefu Chai <kchai@redhat.com>
2016-07-15 17:03:59 +08:00
Kefu Chai
e0a5e67401 Merge pull request #9782 from Ved-vampir/zlib_cleanup
compressor: zlib compressor plugin  cleanup

Reviewed-by: Kefu Chai <kchai@redhat.com>
2016-07-15 17:03:38 +08:00
Kefu Chai
83e64fad87 test: run_seed_to_range.sh: check existance of a directory using [ -d "$dir" ]
sadly, sh evalutes `[ -d ]` to true. as it takes "-d" as a non-empty
string as true.

this fixes following failure
```
2016-07-12T23:22:02.839 INFO:teuthology.orchestra.run.mira084.stderr:cp:
missing destination file operand after ‘.’
2016-07-12T23:22:02.839 INFO:teuthology.orchestra.run.mira084.stderr:Try
'cp --help' for more information.
```
see
http://pulpito.ceph.com/kchai-2016-07-12_23:09:35-rados-wip-kefu-testing2---basic-mira/311334/

Signed-off-by: Kefu Chai <kchai@redhat.com>
2016-07-15 16:48:22 +08:00
Kefu Chai
7c1b456022 qa/workunits/cephtool/test.sh: s/TMPDIR/TEMP_DIR/
this fixes the test failure of
```
2016-07-12T23:29:40.935
INFO:tasks.workunit.client.0.mira101.stderr:/home/ubuntu/cephtest/workunit.client.0/cephtool/test.sh:
line 153: /CEPH_WATCH_9445: Permission denied
```
see
http://pulpito.ceph.com/kchai-2016-07-12_23:09:35-rados-wip-kefu-testing2---basic-mira/311333/

it's a regression introduced by e5c262b

Signed-off-by: Kefu Chai <kchai@redhat.com>
2016-07-15 16:20:37 +08:00
Kefu Chai
e5c262bfe6 qa/workunits/cephtool/test.sh: use mktemp $TEMP_DIR/XXX instead
mktemp -p is not supported on FreeBSD

Signed-off-by: Kefu Chai <kchai@redhat.com>
2016-07-15 14:21:52 +08:00
Kefu Chai
e0b94ff38d Merge pull request #10131 from badone/wip-peering-doc-fixes
doc: peering.rst, fix typo

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2016-07-15 12:15:55 +08:00
Kefu Chai
b342e4f06b Merge pull request #10292 from badone/wip-perf-counters-doc-fixes
doc: perf_counters.rst fix trivial typo

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2016-07-15 12:13:19 +08:00
Yuri Weinstein
3f6daa90e2 Merge pull request #9491 from Yan-waller/yj-wip-fixbuildpushop
osd/ReplicatedBackend: add sanity check during build_push_op()

Reviewed-by: Haomai Wang <haomai@xsky.com>
2016-07-14 15:16:22 -07:00
Yuri Weinstein
f748b95966 Merge pull request #9510 from renhwztetecs/renhw-wip-common
common: unknown hash type of judgment modification

Reviewed-by: Sage Weil <sage@redhat.com>
2016-07-14 15:15:18 -07:00
Yuri Weinstein
5d1339bbc5 Merge pull request #9729 from stiopaa1/common_workqueue_addMove
common/WorkQueue: add std move

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2016-07-14 15:14:39 -07:00
Yuri Weinstein
125c42a452 Merge pull request #9770 from tanghaodong25/submit-transaction-perfcounter
kv/RocksDB: add perfcounter for submit_transaction_sync operation

Reviewed-by: Sage Weil <sage@redhat.com>
2016-07-14 15:13:31 -07:00
Casey Bodley
49001d99ed Merge pull request #10274 from yehudasa/wip-16665
rgw: collect skips a specific coroutine stack

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2016-07-14 17:06:11 -04:00
Yehuda Sadeh
f01f2190b5 Merge pull request #10187 from dreamhost/wip-16618
rgw: fix multi-delete query param parsing.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
2016-07-14 11:42:37 -07:00
Samuel Just
6785393c04 Merge pull request #10289 from jdurgin/wip-googletest-centos
fix the rpm build for centos

Reviewed-by: Samuel Just <sjust@redhat.com>
2016-07-14 07:48:35 -07:00
Orit Wasserman
a01e32d128 Merge pull request #10271 from linuxbox2/rgw-rename
rgw_file: fix rename cases and unify unlink
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2016-07-14 15:42:08 +02:00
Orit Wasserman
6a86bfe8b8 Merge pull request #8618 from linuxbox2/rgw-file-setattr
rgw: file setattr
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2016-07-14 15:16:03 +02:00
Matt Benjamin
4de1c3c260 rgw_file: implement rgw_setattr
Introduce a new RGWSetattrs RGWOp descendant, to create or replace
sets of attrs on buckets or objects.

This version of the change uses the standard RGWRADOS::set_attrs op
(we want attribute changes to (e.g.) sync with other changes).

Previous versions of this changed incorrectly masked the values
of st->st_ino in RGWFileHandle::stat(), now fixed.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-07-14 08:33:11 -04:00
Orit Wasserman
a0ffd6b1d3 Merge pull request #10284 from linuxbox2/rgwfile_write_guard
rgw_file: refuse partial, out-of-order writes
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2016-07-14 14:31:21 +02:00
Kefu Chai
e5cf8c1a8f Merge pull request #10258 from ukernel/wip-16655
ceph-fuse: link to libtcmalloc or jemalloc

Reviewed-by: Kefu Chai <kchai@redhat.com>
2016-07-14 20:26:32 +08:00
John Spray
f8ecf532fb Merge pull request #10168 from batrick/client-inode-des
client: move Inode specific cleanup to destructor

Reviewed-by: John Spray <john.spray@redhat.com>
2016-07-14 12:38:48 +01:00
John Spray
f8394a2a5c Merge pull request #10169 from batrick/client-inode-cons
client: note order of member init in cons

Reviewed-by: John Spray <john.spray@redhat.com>
2016-07-14 12:33:20 +01:00
John Spray
ad86588abe Merge pull request #10265 from tchaikov/wip-ceph-fs-warnings
include/ceph_fs.h: guard `#define CEPH_SETATTR_*` with #ifndef

Reviewed-by: John Spray <john.spray@redhat.com>
2016-07-14 12:20:59 +01:00
John Spray
4b7a5cb2da Merge pull request #10142 from batrick/i16042
mds: move Finisher to unlocked shutdown

Reviewed-by: John Spray <john.spray@redhat.com>
2016-07-14 12:19:16 +01:00
John Spray
5195c94eb1 Merge pull request #10176 from renhwztetecs/renhw-wip-mon-mds
mon/mds: add err info when load_metadata is abnormal

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
2016-07-14 12:18:14 +01:00
John Spray
453b69f4ac Merge pull request #10192 from batrick/mon-minor
mon: use clearer code structure

Reviewed-by: songbaisen <song.baisen@zte.com.cn>
2016-07-14 11:56:42 +01:00
John Spray
b8c5647e56 Merge pull request #10200 from renhwztetecs/renhw-wip-mds-journal
osdc/mds: fixup pos parameter in the journaler

Reviewed-by: John Spray <john.spray@redhat.com>
2016-07-14 11:55:22 +01:00
Brad Hubbard
73084b3371 doc: peering.rst, fix typo
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
2016-07-14 18:34:06 +10:00
Brad Hubbard
10de0b0d0c doc: perf_counters.rst fix trivial typo
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
2016-07-14 18:33:16 +10:00