Commit Graph

49167 Commits

Author SHA1 Message Date
Matt Benjamin
ce62d80595 librgw: fix rgw_read, add unit test
rgw_read had been trivially broken by the relative path change,
fixed.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:54 -05:00
Matt Benjamin
1b797a72b5 librgw: add rgw_getattr tests
Re-implement rgw_getattr to just dispatch into RGWFileHandle::stat,
we expect all objects to stat correctly for their type.

The size of file objects is currently reporting as 0, where we expect
a small, positive size (reviewing).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:53 -05:00
Matt Benjamin
117208c7d4 librgw: re-implement rgw_create
rgw_create hadn't been re-implemented for NFS namespace behavior,
where we expect it to create empty file objects.

It needs to do a full name conflict check with prefix-matching,
to catch conflicts within object names.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:52 -05:00
Matt Benjamin
ec924b6656 librgw: update rgw_* api calls so that all take a flags arg
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:51 -05:00
Matt Benjamin
4f2dd4670f librgw: normalize obj_rec::sync calls, fix signedness of refcnt asserts
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:50 -05:00
Matt Benjamin
a50416bcd6 librgw: partial getattrs/gc checks
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:49 -05:00
Matt Benjamin
402a76f209 librgw: fix handle type in rgw_mkdir, finish dirs1 create tests
Fixes handle type after rgw_mkdir.

Add handle type checks to nfsns unit tests.  Adds tests to create
files and directories with non-bucket parent.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:47 -05:00
Matt Benjamin
d18fe86b54 librgw: set directory ctime and mtime
Adds method to RGWStatBucketRequest to access the creation_time of
the returned bucket.

(For now, that's the only timestamp we have.)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:46 -05:00
Matt Benjamin
d8175918d3 librgw: refactor nfsns dirs1 and fix a "bad unlink" case
Fix the error returns from rgw_unlink.  This is sufficient to
catch the errors that arise from operation failure.

We won't notice attempts to remove interior directories--and for
various reasons, that may be problematic (skip for now).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:45 -05:00
Matt Benjamin
558c211571 librgw: incremental bucket-object 2-level tests
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:44 -05:00
Matt Benjamin
12fd3faaec librgw: enforce S3 object name restrictions
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:43 -05:00
Matt Benjamin
66b6b1fd3e librgw: enforce S3 bucket name restrictions
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:42 -05:00
Matt Benjamin
54149b06c1 librgw: FLAG_CREATE cleanup
Split the transiently-creating concept (CREATING) from the request
to create objects used in lookup_fh.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:41 -05:00
Matt Benjamin
c1d1e15dcb librgw: nfsns dirs1 delete, make hier1 conditional
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:40 -05:00
Matt Benjamin
41b2357a46 librgw: incr. fix various nfs ops (REBASEME)
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:39 -05:00
Matt Benjamin
b008fe5204 librgw: reorg SETUP_ENUMERATE test
Stat nfsroot bucket, create it only if !exist && do_create.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:38 -05:00
Matt Benjamin
b7730c0a6b librgw: conditionally send '.' and '..' (rgw_readdir)
Ganesha traditionally does not expect them, but they may be needed
when bypassing the MDCACHE (2.4).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:37 -05:00
Matt Benjamin
5e75c1834a librgw: stop abusing parents
Also make sure that root file handle has directory as its variant
type.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:36 -05:00
Matt Benjamin
7cc6a6dbeb librgw: likely correct marker handling
New logic propagates is_truncated traveral-finished indicator up
from bucket and object list operations in all cases, and correctly
expands relative paths.

Unit test output looks promising.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:34 -05:00
Matt Benjamin
446d92c4c8 librgw: is_truncated experiments
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:33 -05:00
Matt Benjamin
bd07c2ad4f librgw nfsns: create/delete updates
Add ability to delete marker objs, and create (but not yet delete)
nfs_hier objects.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:32 -05:00
Matt Benjamin
adfb96a5c2 librgw: don't early terminate readdirs (in progress)
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:31 -05:00
Matt Benjamin
9a6944c747 librgw: remove unused st
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:30 -05:00
Matt Benjamin
603f245ea3 librgw: split nfsns bucket and object create blocks
Helps to see the respective timings.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:29 -05:00
Matt Benjamin
82aababc46 librgw_file_nfsns: bugfix MARKER1_SETUP
Successfully creates 5*1024 objects at /nfsroot/nfs_marker/, verifies
externally.

Follow-up rgw_readdir shows 999 results, apparently the initial readdir
segment works correctly, but restart from offset fails.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:28 -05:00
Matt Benjamin
e6b3be3cbf librgw: consolidate dup'd librgw, rgwlib
There can be only one.  Fixes crash on shutdown.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:27 -05:00
Matt Benjamin
103b9dde9e librgw: remove search_prefix()
Since it turns out this is a synonym for relative_object_name().

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:26 -05:00
Matt Benjamin
5b6af8a1af incremental marker
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:25 -05:00
Matt Benjamin
80d5201ae2 librgw: omit always-false depth test.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:24 -05:00
Matt Benjamin
bd4b984472 librgw: rework full_object_name() and friends
Flex RGWFileHandle::full_object_name() to build either a search
prefix (omits leading "/" and bucket name) or else the full path
to the root (e.g., for pretty-printing), depending on the value
of a boolean omit_bucket argument.

New convenience methods search_prefix() and relative_object_name()
select these, respectively.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:23 -05:00
Matt Benjamin
4158721036 librgw: add continued readdir test
Verify that files created in a single, large directory are correctly
enumerated through a sequence of rgw_readdir() calls, at successive
offsets checking cookie/marker and bounds invariants.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:22 -05:00
Matt Benjamin
1097ed4cf3 marker
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:20 -05:00
Matt Benjamin
a557c898c0 Fix copyright statements on new test driver files (Red Hat, Inc.)
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:19 -05:00
Matt Benjamin
597f69d953 librgw: dir traversal state GC added
To minimize overhead, traversal events are modeled as a linear
sequence of fh_key, timestamp tuples.  GC operates out-of-line in
LIFO order, and includes a ref cycle.  If the matching directory
object can't be found by handle, it's already gone, and nothing
happens.

To avoid lock contention, scanning of the event sequence is done
in groups of 24 by default, but in groups of sequence/4 if the
event sequence grows larger than 500 events.

Push READDIR events on successful component readdir (bucket or
object listing partial result, as appropriate).

Revise locking, timestamp and update link count in all cases.  Link
counts are approximate (metadata to represent it is not stored).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:18 -05:00
Matt Benjamin
b970fd005e librgw: restore readdir prints (move to dout)
They are useful for checking the exact strings being produced,
and in which sequence (object or common prefix).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:17 -05:00
Matt Benjamin
7acb20d877 librgw: move readdir operations into RGWFileHandle::readdir
The motivation here is to localize state updates to the class,
e.g., to hide consistency logic.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:16 -05:00
Matt Benjamin
453219cb12 librgw: implement small-directory name cache
Augment the existing marker cache with cache of name->type
mappings for up to 128 dirents (appx. 4x the common upper-bound
of 32).

The motivation here is to use the name and type information from
readdir, rather than discarding it.  Boosts perf. at least 60%
on unit test corpus.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:15 -05:00
Matt Benjamin
e8fc2b757e librgw: remote redundnat prints in RGWReaddirRequest
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:14 -05:00
Matt Benjamin
49a034afea librgw: don't increment offset at rcb (it is no longer a counter)
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:13 -05:00
Matt Benjamin
bd06808907 librgw: fill in dirent cache
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:12 -05:00
Matt Benjamin
33856c191b librgw: implement framework for fs periodic work
This can be used for, e.g., gc of stale handles and directory
traversals.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:11 -05:00
Matt Benjamin
3237f0b96a librgw: add intrusive refcnt to RGWLibFS
Ensures that file system instances cannot be disposed while (e.g.)
background processing is ongoing.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:10 -05:00
Matt Benjamin
baffb146bf librgw: header and namespace reorganization
Move RGWLibProcess and RGWLibFrontend into new rgw_lib_frontend.h
header, which can include rgw_lib.h and rgw_file.h.

This allows the already-defined RGWLibProcess to call methods on
RGWLibFS objects (e.g., to perform GC, which follows in a separate
commit).

In addition, move all of the library and file interface into
namespace rgw.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:09 -05:00
Matt Benjamin
e448c4eb7f librgw: use string_ref remove_prefix
Save a few cycles over re-assigning a string_ref suffix substring
to itself.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:07 -05:00
Matt Benjamin
8db2c6a969 librgw: fix initialization of string_ref from temporary
Found by valgrind.  The good news:  amounts to proof that the
sref notation works.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:06 -05:00
Matt Benjamin
b19912c9b2 cohort_lru: use correct new/delete operator pair
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:05 -05:00
Matt Benjamin
eaf156397a librgw: move ObjUnref trace print ahead of...unref
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:04 -05:00
Matt Benjamin
1f2ec092f7 librgw: cleanup add_marker and handlers
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:03 -05:00
Matt Benjamin
e6eee9fa53 librgw: rename RGWGetObjRequest, RGWListBucketRequest
The behavior of these requests is basically specific to the
needs of NFS READDIR and READ.  (RGWWriteRequest already updated.)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:02 -05:00
Matt Benjamin
ef6367fc35 common: add (tweaked) sstring.h
Add sstring.h from Seastar, slightly adapted for c++2011 and
boost::string_ref.

An sstring resembles std::string, but the size of its small-string
optimization (static allocation) is controlled by a template
argument (cf. small_vector).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2016-02-12 12:07:01 -05:00