open id connect provider related REST APIs and removing
references to token introspection for validating incoming
web token.
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
This feature is meant to add data cache feature to the RGW.
It is using Nginx as a cache server.
This feature adds 2 new apis, Auth api and Cache api.
Some Performance tests using hsbench:
16K objs:
RGW direct access:
Mode: GET, Ops: 3001, MB/s: 46.89, Lat(ms): [ min: 30.4, avg: 33.2, 99%: 34.7, max: 35.2 ]
Nginx access (objs have not been cached)
Mode: GET, Ops: 1363, MB/s: 21.30, Lat(ms): [ min: 63.8, avg: 73.8, 99%: 78.1, max: 86.6 ]
Nginx access (objs have been cached)
Mode: GET, Ops: 2446, MB/s: 38.22, Lat(ms): [ min: 36.9, avg: 41.0, 99%: 43.9, max: 45.9 ]
512K objs:
RGW direct access:
Mode: GET, Ops: 1492, MB/s: 746.00 Lat(ms): [ min: 60.4, avg: 66.7, 99%: 73.5, max: 75.9 ]
Nginx access (objs have not been cached)
Mode: GET, Ops: 1382, MB/s: 691.00, Lat(ms): [ min: 64.5, avg: 72.1, 99%: 77.9, max: 82.8 ]
Nginx access (objs have been cached)
Mode: GET, Ops: 2947, MB/s: 1473.50, Lat(ms): [ min: 3.3, avg: 32.7, 99%: 62.2, max: 72.1 ]
2M objs:
RGW direct access:
Mode: GET, Ops: 613, MB/s: 1226.00, Lat(ms): [ min: 143.6, avg: 162.0, 99%: 180.2, max: 190.1 ]
Nginx access (objs have not been cached)
Mode: GET, Ops: 462, MB/s: 924.00, Lat(ms): [ min: 180.2, avg: 215.0, 99%: 243.2, max: 248.3 ]
Nginx access (objs have been cached)
Mode: GET, Ops: 1392, MB/s: 2784.00, Lat(ms): [ min: 3.0, avg: 5.3, 99%: 18.8, max: 30.2 ]
10M objs:
RGW direct access:
Mode: GET, Ops: 135, MB/s: 1350.00, Lat(ms): [ min: 191.1, avg: 265.8, 99%: 373.1, max: 382.8 ]
Nginx access (objs have not been cached)
Mode: GET, Ops: 120, MB/s: 1200.00, Lat(ms): [ min: 302.1, avg: 428.8, 99%: 561.2, max: 583.7 ]
Nginx access (objs have been cached)
Mode: GET, Ops: 281, MB/s: 2810.00, Lat(ms): [ min: 3.2, avg: 8.3, 99%: 16.9, max: 25.6 ]
gdal_translate 4GiB image gdal_translate -co NUM_THREADS=ALL_CPUS /vsis3/hello/sat.tif
Nginx (have not cached):
real 0m24.714s
user 0m8.692s
sys 0m10.360s
Nginx (have been cached):
real 0m21.070s
user 0m9.140s
sys 0m10.316s
RGW:
real 0m21.859s
user 0m8.850s
sys 0m10.386s
The results are showing that for objects larger than 512K the cache will increase the performance by twice or more.
For small objs, the overhead of sending the auth request will make the cache less efficient
The result for cached objects in the 10MB test can be explained by net limit of 25 Gb/s(it could reach more)
In Gdal (image decoder/encoder over s3 using range requests) the results were not that different because of Gdal single cpu encoding/decoding.
Gdal have been chosen because of the ability to check the smart cache of the nginx.
https://www.nginx.com/blog/smart-efficient-byte-range-caching-nginx/
Signed-off-by: Or Friedmann <ofriedma@redhat.com>
RTD does not support installing system packages, the only ways to install
dependencies are setuptools and pip. while ditaa is a tool written in
Java. so we need to find a native python tool allowing us to render ditaa
images. plantweb is able to the web service for rendering the ditaa
diagram. so let's use it as a fallback if "ditaa" is not around.
also start a new line after the directive, otherwise planweb server will
return 500 at seeing the diagram.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Extend server-side encryption functionality in Rados Gateway to support
HashiCorp Vault as a Key Management System in addition to existing
support for OpenStack Barbican.
This is the first part of this change, supporting Vault's token-based
authentication only. Agent-based authentication as well as other
features such as Vault namespaces will be added in subsequent commits.
Note that Barbican remains the default backend for SSE-KMS
(rgw crypt s3 kms backend) to avoid breaking existing deployments.
Feature: https://tracker.ceph.com/issues/41062
Notes: https://pad.ceph.com/p/rgw_sse-kms
Implemented so far:
* Move existing SSE-KMS functions from rgw_crypt.cc to rgw_kms.cc
* Vault authentication with a token read from file
* Add new ceph.conf settings for Vault
* Document new ceph.conf settings
* Update main encryption documentation page
* Add documentation page for SSE-KMS using Vault
Signed-off-by: Andrea Baglioni <andrea.baglioni@workday.com>
Signed-off-by: Sergio de Carvalho <sergio.carvalho@workday.com>
This patch add the documentation about QAT acceleration for encryption
and compression and how to configure.
Signed-off-by: Qiaowei Ren <qiaowei.ren@intel.com>
* doc/release-notes.rst: escape asterisks not used for inline emphasis
fix bad hyper links
* doc/rados/troubleshooting/troubleshooting-osd.rst: escape asterisks
not used for inline emphasis
* doc/radosgw/index.rst: add orphan docs to toc
* doc/dev/perf_histograms.rst: indent block quote
* doc/install/manual-freebsd-deployment.rst: fix block quotes
* doc/mgr/administrator.rst: escape asterisks not used for inline emphasis
* doc/start/quick-rbd.rst: add missing hyper link target
Signed-off-by: Kefu Chai <kchai@redhat.com>
* indent :maxdepth: directive properly, otherwise sphinx complains
index.rst:45: WARNING: toctree contains reference to nonexisting
document u'rados/configuration/ :maxdepth: 1'
* also reindent the entries in toc, so we can add paths without
documenting their titles. because otherwise the paths inclues an extra
space, without the proper indent.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* upgrade_to_jewel.rst: silence following warning
upgrade_to_jewel.rst:1: WARNING: Title overline too short.
* index.rst: add links to not-included docs
add api.rst, layout.rst and upgrade_to_jewel.rst to toc
silences warning like
api.rst:: WARNING: document isn't included in any toctree
Signed-off-by: Kefu Chai <kchai@redhat.com>
This file documents how to configure RGW to use Apache/FastCGI, so rename
the file and modify the title and intro to make that clear.
Also, add a note that CGI can pose a security risk - e.g. http://httpoxy.org
Signed-off-by: Nathan Cutler <ncutler@suse.com>
