facilitates the full usage of the Nginx cache endpoint
with s3 tools that support AWSv4
like s3cmd,aws-cli, benchmarking tools like hsbench
and also hadoop/s3a.
Co-authored-by: Or Friedmann <ofriedma@redhat.com>
Signed-off-by: Mark Kogan <mkogan@redhat.com>
make topic and subscription read commands an official feature
Fixes: https://tracker.ceph.com/issues/43536
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
open id connect provider related REST APIs and removing
references to token introspection for validating incoming
web token.
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
rgw: Adding data cache and CDN capabilities
Reviewed-by: Mark Kogan <mkogan@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Warn users about the implications of enabling this option when there is
no trusted proxy in front of radosgw.
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
This feature is meant to add data cache feature to the RGW.
It is using Nginx as a cache server.
This feature adds 2 new apis, Auth api and Cache api.
Some Performance tests using hsbench:
16K objs:
RGW direct access:
Mode: GET, Ops: 3001, MB/s: 46.89, Lat(ms): [ min: 30.4, avg: 33.2, 99%: 34.7, max: 35.2 ]
Nginx access (objs have not been cached)
Mode: GET, Ops: 1363, MB/s: 21.30, Lat(ms): [ min: 63.8, avg: 73.8, 99%: 78.1, max: 86.6 ]
Nginx access (objs have been cached)
Mode: GET, Ops: 2446, MB/s: 38.22, Lat(ms): [ min: 36.9, avg: 41.0, 99%: 43.9, max: 45.9 ]
512K objs:
RGW direct access:
Mode: GET, Ops: 1492, MB/s: 746.00 Lat(ms): [ min: 60.4, avg: 66.7, 99%: 73.5, max: 75.9 ]
Nginx access (objs have not been cached)
Mode: GET, Ops: 1382, MB/s: 691.00, Lat(ms): [ min: 64.5, avg: 72.1, 99%: 77.9, max: 82.8 ]
Nginx access (objs have been cached)
Mode: GET, Ops: 2947, MB/s: 1473.50, Lat(ms): [ min: 3.3, avg: 32.7, 99%: 62.2, max: 72.1 ]
2M objs:
RGW direct access:
Mode: GET, Ops: 613, MB/s: 1226.00, Lat(ms): [ min: 143.6, avg: 162.0, 99%: 180.2, max: 190.1 ]
Nginx access (objs have not been cached)
Mode: GET, Ops: 462, MB/s: 924.00, Lat(ms): [ min: 180.2, avg: 215.0, 99%: 243.2, max: 248.3 ]
Nginx access (objs have been cached)
Mode: GET, Ops: 1392, MB/s: 2784.00, Lat(ms): [ min: 3.0, avg: 5.3, 99%: 18.8, max: 30.2 ]
10M objs:
RGW direct access:
Mode: GET, Ops: 135, MB/s: 1350.00, Lat(ms): [ min: 191.1, avg: 265.8, 99%: 373.1, max: 382.8 ]
Nginx access (objs have not been cached)
Mode: GET, Ops: 120, MB/s: 1200.00, Lat(ms): [ min: 302.1, avg: 428.8, 99%: 561.2, max: 583.7 ]
Nginx access (objs have been cached)
Mode: GET, Ops: 281, MB/s: 2810.00, Lat(ms): [ min: 3.2, avg: 8.3, 99%: 16.9, max: 25.6 ]
gdal_translate 4GiB image gdal_translate -co NUM_THREADS=ALL_CPUS /vsis3/hello/sat.tif
Nginx (have not cached):
real 0m24.714s
user 0m8.692s
sys 0m10.360s
Nginx (have been cached):
real 0m21.070s
user 0m9.140s
sys 0m10.316s
RGW:
real 0m21.859s
user 0m8.850s
sys 0m10.386s
The results are showing that for objects larger than 512K the cache will increase the performance by twice or more.
For small objs, the overhead of sending the auth request will make the cache less efficient
The result for cached objects in the 10MB test can be explained by net limit of 25 Gb/s(it could reach more)
In Gdal (image decoder/encoder over s3 using range requests) the results were not that different because of Gdal single cpu encoding/decoding.
Gdal have been chosen because of the ability to check the smart cache of the nginx.
https://www.nginx.com/blog/smart-efficient-byte-range-caching-nginx/
Signed-off-by: Or Friedmann <ofriedma@redhat.com>
RTD does not support installing system packages, the only ways to install
dependencies are setuptools and pip. while ditaa is a tool written in
Java. so we need to find a native python tool allowing us to render ditaa
images. plantweb is able to the web service for rendering the ditaa
diagram. so let's use it as a fallback if "ditaa" is not around.
also start a new line after the directive, otherwise planweb server will
return 500 at seeing the diagram.
Signed-off-by: Kefu Chai <kchai@redhat.com>
When trying to use AWS S3 SDKs with a non-default storage class, clients should call their storage class as one of the generic storage class names provided by Amazon (such as STANDARD_IA, ONEZONE_IA, GLACIER etc), or else the SDK will drop the request complaining the storage class name is not allowed
jenkins render docs
Signed-off-by: Shon Paz <spaz@redhat.com>
We already have Kafka supported as a push endpoint (has been merged lately), in addition it is important to know that although we have extra filters AWS S3 notifications doesn't have, it will cost in extending the AWS SDKs with the new capabilities.
Signed-off-by: Shon Paz <spaz@redhat.com>
We already have Kafka supported as a push endpoint (has been merged lately), in addition it is important to know that although we have extra filters AWS S3 notifications doesn't have, it will cost in extending the AWS SDKs with the new capabilities.
Signed-off-by: Shon Paz <spaz@redhat.com>
Correcting STS documentation to remove s3curl.pl command for getsessiontoken and replacing it with user policy
Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>
use APIs instead of apis to be consistent throughout.
fixes: https://tracker.ceph.com/issues/44374
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
Beast frontend currently accepts a hardcoded number of connections
that is defined by boost::asio::socket_base::max_connections. This
commit makes it configurable via a 'max_connections' config option
on rgw frontend.
Fixes: https://tracker.ceph.com/issues/43952
Signed-off-by: Tiago Pasqualini <tiago.pasqualini@canonical.com>
opaque data may be set in topic configuration and later on sent inside
nottifications triggered by that topic.
Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
Swift accounts are not like normal accounts, they are more akin to a
bank account that multile people could share. Or in the case of a cloud
it is usually mapped to the tenant.
Radosgw deals with this with a user and subuser, which is great, but a
little confusing. So this patch adds a note to those used to the Swift
API to make it more clear.
Signed-off-by: Matthew Oliver <moliver@suse.com>
* Minor improvements to Vault documentation
* Add teuthology tests for Transit secrets engine
* Add unit tests for KV secrets engine, minor improvements to Transit
secrets engine
* use string_view::npos instead of string::npos
Signed-off-by: Andrea Baglioni <andrea.baglioni@workday.com>
Signed-off-by: Sergio de Carvalho <sergio.carvalho@workday.com>
* Drop polymorfism for KMS class
* Fix issue in kms-key selection
* Update documentation for Vault section
Signed-off-by: Andrea Baglioni <andrea.baglioni@workday.com>
Signed-off-by: Sergio de Carvalho <sergio.carvalho@workday.com>
* refactor rgw_kms.cc to support extension to multiple secret engines.
* introduced support to Vault Namesapces
* added support for Vault Agent
Signed-off-by: Andrea Baglioni <andrea.baglioni@workday.com>
Signed-off-by: Sergio de Carvalho <sergio.carvalho@workday.com>
* add 'rgw crypt vault prefix' config setting to allow restricting
secret space in Vault where RGW can retrieve keys from
* refuse Vault token file if permissions are too open
* improve concatenation of URL paths to avoid constructing an invalid
URL (missing or double '/')
* doc: clarify SSE-KMS keys must be 256-bit long and base64 encoded,
document Vault policies and tokens, plus other minor doc improvements
* qa: check SHA256 signature of Vault zip download
* qa: fix teuthology tests broken by previous PR which made SSE-KMS
backend default to Barbican
Signed-off-by: Andrea Baglioni <andrea.baglioni@workday.com>
Signed-off-by: Sergio de Carvalho <sergio.carvalho@workday.com>
This commit adds a configurable option rgw_max_dynamic_shards that
provides a maximum bucket index shard count that dynamic resharding
can take a bucket to; the default is 1999. Note: this does not limit
the number of bucket index shards when set manually.
This commit also only allows prime shard counts when
rgw_max_dynamix_shards is no larger than 1999. Once it is larger, then
it allows any shard count, including non-prime values.
Finally, this commit adds unit tests to make sure the bucket index
shard count calculations work as expected.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
Clarify supported secret engine in the Vault documentation.
Signed-off-by: Andrea Baglioni <andrea.baglioni@workday.com>
Signed-off-by: Sergio de Carvalho <sergio.carvalho@workday.com>
Minor fix to config documentation.
Signed-off-by: Andrea Baglioni <andrea.baglioni@workday.com>
Signed-off-by: Sergio de Carvalho <sergio.carvalho@workday.com>
Extend server-side encryption functionality in Rados Gateway to support
HashiCorp Vault as a Key Management System in addition to existing
support for OpenStack Barbican.
This is the first part of this change, supporting Vault's token-based
authentication only. Agent-based authentication as well as other
features such as Vault namespaces will be added in subsequent commits.
Note that Barbican remains the default backend for SSE-KMS
(rgw crypt s3 kms backend) to avoid breaking existing deployments.
Feature: https://tracker.ceph.com/issues/41062
Notes: https://pad.ceph.com/p/rgw_sse-kms
Implemented so far:
* Move existing SSE-KMS functions from rgw_crypt.cc to rgw_kms.cc
* Vault authentication with a token read from file
* Add new ceph.conf settings for Vault
* Document new ceph.conf settings
* Update main encryption documentation page
* Add documentation page for SSE-KMS using Vault
Signed-off-by: Andrea Baglioni <andrea.baglioni@workday.com>
Signed-off-by: Sergio de Carvalho <sergio.carvalho@workday.com>
The value for rgw_ldap_secrect has to be a path to the file containing
the secret not the secret itself.
Signed-off-by: Robin Müller <github@mail.coder-hugo.de>
As per amazon s3 spec -
https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html
* The s3 bucket names should not contain upper case letters or underscore.
* Name cannot end with dash or have consecutive periods, or dashes adjacent
to periods.
* Each label in the bucket name must start and end with a lowercase
letter or a number.
* Name cannot exceed 63 characters.
This change is to enforce these rules if rgw_relaxed_s3_bucket_names is set to
'false' which is by default.
Fixes: https://tracker.ceph.com/issues/36293
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
Improve and add to documentation for "bucket move" functionality;
including use moving to multi-tenancy and further deprecating bucket-id
which is no longer necessary.
Fixes: http://tracker.ceph.com/issues/35885
Signed-off-by: Marcus Watts <mwatts@redhat.com>
In jewel, "rgw keystone implicit tenants" only applied to swift. As of
luminous), this option applies to s3 also.
Sites that used this feature with jewel now have outstanding data that
depends on the old behavior.
The fix here is to expand "rgw keystone implicit tenants" so that it
can be set to any of "none", "all", "s3" or "swift" (also 0=false=none,
1=true=all). When set to "s3" or "swift", the actual id lookup
is also partitioned.
Formerly "rgw keystone implicit tenants" was a legacy opt.
This change converts it to the new style of option,
including support for dynamically changing it.
Fixes: http://tracker.ceph.com/issues/24348
Signed-off-by: Marcus Watts <mwatts@redhat.com>
The S3 action is case-sensitive and was fixed in #21916, but there were
two more occurrences left.
Signed-off-by: Thomas Kriechbaumer <thomas@kriechbaumer.name>
rgw: add S3 object lock feature to support object worm
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
rgw: update the "radosgw-admin reshard status"
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
rgw: normalize v6 endpoint behaviour for the beast frontend
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
'radosgw-admin period pull' fetches a period configuration, but does not
update the realm's current_period to use it. the 'realm pull' command
does both, and the difference is especially important in the failover
case
Fixes: http://tracker.ceph.com/issues/39655
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Some minor updates to the resharding documentation with the aim of
clarifying the language and using terms consistently.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
config-ref: add a note on current scheduler settings.
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Adding a note on configuirables for max concurrent requests and the rest of
experimental options for tuning dmclock scheduler
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
Removed Kilo references in Keystone docs. Updated documentation
to align with Ocata & later releases.
Fixes: https://tracker.ceph.com/issues/38721
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>
beast frontend option to set the TCP_NODELAY socket option to match the tcp_nodelay option in civetweb.
Fixes: https://tracker.ceph.com/issues/34308
Signed-off-by: Or Friedmann <ofriedma@redhat.com>
all of these civetweb options have to be on the same line as rgw_frontends
Fixes: https://tracker.ceph.com/issues/37770
Signed-off-by: Casey Bodley <cbodley@redhat.com>
This patch add the documentation about QAT acceleration for encryption
and compression and how to configure.
Signed-off-by: Qiaowei Ren <qiaowei.ren@intel.com>
Since the ACL documentation is a bit sparse, let people know exactly
what operations they can expect to enable with a given ACL grant.
Fixes: https://tracker.ceph.com/issues/38523
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
The documentation for the S3 PHP client usage is about an old client. This update the examples to the current S3 PHP client.
Signed-off-by: Laurent VOULLEMIER <laurent.voullemier@gmail.com>
