RepoMirrors/ceph
1
0
Fork 0
mirror of https://github.com/ceph/ceph synced 2025-02-24 19:47:44 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'wip-wsgi' into next

Browse Source 
* wip-wsgi:
  ceph-rest-api: separate into module and front-end for WSGI deploy
  ceph-rest-api: make main program be "shell" around WSGI guts

  Reviewed-by: Sage Weil <sage@inktank.com>
This commit is contained in:
Dan Mick 2013-07-12 16:38:10 -07:00
commit b5659b5e7f
6 changed files with 527 additions and 433 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
ceph.spec.in
View File

@ -564,6 +564,7 @@ fi
%{python_sitelib}/rbd.py*
%{python_sitelib}/cephfs.py*
%{python_sitelib}/ceph_argparse.py*
%{python_sitelib}/ceph_rest_api.py*
#################################################################################
%files -n rest-bench

47
doc/man/8/ceph-rest-api.rst
View File

@ -23,14 +23,24 @@ Options
.. option:: -c/--conf *conffile*
names the ceph.conf file to use for configuration. If -c
is not specified, the configuration file is searched for in
this order:
names the ceph.conf file to use for configuration. If -c is not
specified, the default depends on the state of the --cluster option
(default 'ceph'; see below). The configuration file is searched
for in this order:
* $CEPH_CONF
* /etc/ceph/ceph.conf
* ~/.ceph/ceph.conf
* ceph.conf (in the current directory)
* /etc/ceph/${cluster}.conf
* ~/.ceph/${cluster}.conf
* ${cluster}.conf (in the current directory)
so you can also pass this option in the environment as CEPH_CONF.
.. option:: --cluster *clustername*
set *clustername* for use in the $cluster metavariable, for
locating the ceph.conf file. The default is 'ceph'.
You can also pass this option in the environment as
CEPH_CLUSTER_NAME.
.. option:: -n/--name *name*
@ -38,14 +48,16 @@ Options
client-specific configuration options in the config file, and
also is the name used for authentication when connecting
to the cluster (the entity name appearing in ceph auth list output,
for example). The default is 'client.restapi'.
for example). The default is 'client.restapi'. You can also
pass this option in the environment as CEPH_NAME.
Configuration parameters
========================
Supported configuration parameters include:
* **restapi client name** the 'clientname' used for auth and ceph.conf
* **restapi keyring** the keyring file holding the key for 'clientname'
* **restapi public addr** ip:port to listen on (default 0.0.0.0:5000)
* **restapi base url** the base URL to answer requests on (default /api/v0.1)
@ -82,6 +94,25 @@ the value of **restapi base url**, and that path will give a full list
of all known commands. The command set is very similar to the commands
supported by the **ceph** tool.
Deployment as WSGI application
==============================
When deploying as WSGI application (say, with Apache/mod_wsgi,
or nginx/uwsgi, or gunicorn, etc.), use the ``ceph_rest_api.py`` module
(``ceph-rest-api`` is a thin layer around this module). The standalone web
server is of course not used, so address/port configuration is done in
the WSGI server. Also, configuration switches are not passed; rather,
environment variables are used:
* CEPH_CONF holds -c/--conf
* CEPH_CLUSTER_NAME holds --cluster
* CEPH_NAME holds -n/--name
Any errors reading configuration or connecting to the cluster cause
ImportError to be raised with a descriptive message on import; see
your WSGI server documentation for how to see those messages in case
of problem.
Availability
============

51
man/ceph-rest-api.8
View File

@ -1,4 +1,4 @@
.TH "CEPH-REST-API" "8" "July 10, 2013" "dev" "Ceph"
.TH "CEPH-REST-API" "8" "July 12, 2013" "dev" "Ceph"
.SH NAME
ceph-rest-api \- ceph RESTlike administration server
.
@ -45,19 +45,30 @@ command\-line tool through an HTTP\-accessible interface.
.INDENT 0.0
.TP
.B \-c/\-\-conf *conffile*
names the ceph.conf file to use for configuration. If \-c
is not specified, the configuration file is searched for in
this order:
names the ceph.conf file to use for configuration. If \-c is not
specified, the default depends on the state of the \-\-cluster option
(default \(aqceph\(aq; see below). The configuration file is searched
for in this order:
.INDENT 7.0
.IP \(bu 2
$CEPH_CONF
.IP \(bu 2
/etc/ceph/ceph.conf
/etc/ceph/${cluster}.conf
.IP \(bu 2
~/.ceph/ceph.conf
~/.ceph/${cluster}.conf
.IP \(bu 2
ceph.conf (in the current directory)
${cluster}.conf (in the current directory)
.UNINDENT
.sp
so you can also pass this option in the environment as CEPH_CONF.
.UNINDENT
.INDENT 0.0
.TP
.B \-\-cluster *clustername*
set \fIclustername\fP for use in the $cluster metavariable, for
locating the ceph.conf file. The default is \(aqceph\(aq.
You can also pass this option in the environment as
CEPH_CLUSTER_NAME.
.UNINDENT
.INDENT 0.0
.TP
@ -66,13 +77,16 @@ specifies the client \(aqname\(aq, which is used to find the
client\-specific configuration options in the config file, and
also is the name used for authentication when connecting
to the cluster (the entity name appearing in ceph auth list output,
for example). The default is \(aqclient.restapi\(aq.
for example). The default is \(aqclient.restapi\(aq. You can also
pass this option in the environment as CEPH_NAME.
.UNINDENT
.SH CONFIGURATION PARAMETERS
.sp
Supported configuration parameters include:
.INDENT 0.0
.IP \(bu 2
\fBrestapi client name\fP the \(aqclientname\(aq used for auth and ceph.conf
.IP \(bu 2
\fBrestapi keyring\fP the keyring file holding the key for \(aqclientname\(aq
.IP \(bu 2
\fBrestapi public addr\fP ip:port to listen on (default 0.0.0.0:5000)
@ -109,6 +123,27 @@ path is incomplete/partially matching. Requesting / will redirect to
the value of \fBrestapi base url\fP, and that path will give a full list
of all known commands. The command set is very similar to the commands
supported by the \fBceph\fP tool.
.SH DEPLOYMENT AS WSGI APPLICATION
.sp
When deploying as WSGI application (say, with Apache/mod_wsgi,
or nginx/uwsgi, or gunicorn, etc.), use the \fBceph_rest_api.py\fP module
(\fBceph\-rest\-api\fP is a thin layer around this module). The standalone web
server is of course not used, so address/port configuration is done in
the WSGI server. Also, configuration switches are not passed; rather,
environment variables are used:
.INDENT 0.0
.IP \(bu 2
CEPH_CONF holds \-c/\-\-conf
.IP \(bu 2
CEPH_CLUSTER_NAME holds \-\-cluster
.IP \(bu 2
CEPH_NAME holds \-n/\-\-name
.UNINDENT
.sp
Any errors reading configuration or connecting to the cluster cause
ImportError to be raised with a descriptive message on import; see
your WSGI server documentation for how to see those messages in case
of problem.
.SH AVAILABILITY
.sp
\fBceph\-rest\-api\fP is part of the Ceph distributed file system. Please refer to the Ceph documentation at

3
src/Makefile.am
View File

@ -1697,7 +1697,8 @@ dist-hook:
python_PYTHON = pybind/rados.py \
pybind/rbd.py \
pybind/cephfs.py \
pybind/ceph_argparse.py
pybind/ceph_argparse.py \
pybind/ceph_rest_api.py
# headers... and everything else we want to include in a 'make dist'
# that autotools doesn't magically identify.

444
src/ceph-rest-api
View File

@ -1,6 +1,8 @@
#!/usr/bin/python
# vim: ts=4 sw=4 smarttab expandtab
import argparse
import inspect
import os
import sys
@ -25,430 +27,40 @@ if MYDIR.endswith('src') and \
os.execvp('python', ['python'] + sys.argv)
sys.path.insert(0, os.path.join(MYDIR, 'pybind'))
import argparse
import collections
import ConfigParser
import errno
import json
import logging
import logging.handlers
import rados
import textwrap
import xml.etree.ElementTree
import xml.sax.saxutils
import flask
from ceph_argparse import *
#
# Globals
#
APPNAME = '__main__'
DEFAULT_BASEURL = '/api/v0.1'
DEFAULT_ADDR = '0.0.0.0:5000'
DEFAULT_LOG_LEVEL = 'warning'
DEFAULT_CLIENTNAME = 'client.restapi'
DEFAULT_LOG_FILE = '/var/log/ceph/' + DEFAULT_CLIENTNAME + '.log'
app = flask.Flask(APPNAME)
LOGLEVELS = {
'critical':logging.CRITICAL,
'error':logging.ERROR,
'warning':logging.WARNING,
'info':logging.INFO,
'debug':logging.DEBUG,
}
# my globals, in a named tuple for usage clarity
glob = collections.namedtuple('gvars',
'args cluster urls sigdict baseurl clientname')
glob.args = None
glob.cluster = None
glob.urls = {}
glob.sigdict = {}
glob.baseurl = ''
glob.clientname = ''
def parse_args():
parser = argparse.ArgumentParser(description="Ceph REST API webapp")
parser.add_argument('-c', '--conf', help='Ceph configuration file')
parser.add_argument('-n', '--name', help='Ceph client config/key name')
parser.add_argument('--cluster', help='Ceph cluster name')
parser.add_argument('-n', '--name', help='Ceph client name')
return parser.parse_args()
def load_conf(conffile=None):
import contextlib
class _TrimIndentFile(object):
def __init__(self, fp):
self.fp = fp
def readline(self):
line = self.fp.readline()
return line.lstrip(' \t')
def _optionxform(s):
s = s.replace('_', ' ')
s = '_'.join(s.split())
return s
def parse(fp):
cfg = ConfigParser.RawConfigParser()
cfg.optionxform = _optionxform
ifp = _TrimIndentFile(fp)
cfg.readfp(ifp)
return cfg
def load(path):
f = file(path)
with contextlib.closing(f):
return parse(f)
# XXX this should probably use cluster name
if conffile:
return load(conffile)
elif 'CEPH_CONF' in os.environ:
conffile = os.environ['CEPH_CONF']
elif os.path.exists('/etc/ceph/ceph.conf'):
conffile = '/etc/ceph/ceph.conf'
elif os.path.exists(os.path.expanduser('~/.ceph/ceph.conf')):
conffile = os.path.expanduser('~/.ceph/ceph.conf')
elif os.path.exists('ceph.conf'):
conffile = 'ceph.conf'
else:
return None
return load(conffile)
def get_conf(cfg, key):
try:
return cfg.get(glob.clientname, 'restapi_' + key)
except ConfigParser.NoOptionError:
return None
# XXX this is done globally, and cluster connection kept open; there
# are facilities to pass around global info to requests and to
# tear down connections between requests if it becomes important
def api_setup():
"""
Initialize the running instance. Open the cluster, get the command
signatures, module,, perms, and help; stuff them away in the glob.urls
dict.
"""
glob.args = parse_args()
conffile = glob.args.conf or ''
if glob.args.name:
glob.clientname = glob.args.name
glob.logfile = '/var/log/ceph' + glob.clientname + '.log'
glob.clientname = glob.args.name or DEFAULT_CLIENTNAME
glob.cluster = rados.Rados(name=glob.clientname, conffile=conffile)
glob.cluster.connect()
cfg = load_conf(conffile)
glob.baseurl = get_conf(cfg, 'base_url') or DEFAULT_BASEURL
if glob.baseurl.endswith('/'):
glob.baseurl
addr = get_conf(cfg, 'public_addr') or DEFAULT_ADDR
addrport = addr.rsplit(':', 1)
addr = addrport[0]
if len(addrport) > 1:
port = addrport[1]
else:
port = DEFAULT_ADDR.rsplit(':', 1)
port = int(port)
loglevel = get_conf(cfg, 'log_level') or 'warning'
logfile = get_conf(cfg, 'log_file') or glob.logfile
app.logger.addHandler(logging.handlers.WatchedFileHandler(logfile))
app.logger.setLevel(LOGLEVELS[loglevel.lower()])
for h in app.logger.handlers:
h.setFormatter(logging.Formatter(
'%(asctime)s %(name)s %(levelname)s: %(message)s'))
ret, outbuf, outs = json_command(glob.cluster,
prefix='get_command_descriptions')
if ret:
app.logger.error('Can\'t contact cluster for command descriptions: %s',
outs)
sys.exit(1)
try:
glob.sigdict = parse_json_funcsigs(outbuf, 'rest')
except Exception as e:
app.logger.error('Can\'t parse command descriptions: %s', e)
sys.exit(1)
# glob.sigdict maps "cmdNNN" to a dict containing:
# 'sig', an array of argdescs
# 'help', the helptext
# 'module', the Ceph module this command relates to
# 'perm', a 'rwx*' string representing required permissions, and also
# a hint as to whether this is a GET or POST/PUT operation
# 'avail', a comma-separated list of strings of consumers that should
# display this command (filtered by parse_json_funcsigs() above)
glob.urls = {}
for cmdnum, cmddict in glob.sigdict.iteritems():
cmdsig = cmddict['sig']
url, params = generate_url_and_params(cmdsig)
if url in glob.urls:
continue
else:
perm = cmddict['perm']
urldict = {'paramsig':params,
'help':cmddict['help'],
'module':cmddict['module'],
'perm':perm,
}
method_dict = {'r':['GET'],
'w':['PUT', 'DELETE']}
for k in method_dict.iterkeys():
if k in perm:
methods = method_dict[k]
app.add_url_rule(url, url, handler, methods=methods)
glob.urls[url] = urldict
url += '.<fmt>'
app.add_url_rule(url, url, handler, methods=methods)
glob.urls[url] = urldict
app.logger.debug("urls added: %d", len(glob.urls))
app.add_url_rule('/<path:catchall_path>', '/<path:catchall_path>',
handler, methods=['GET', 'PUT'])
return addr, port
def generate_url_and_params(sig):
"""
Digest command signature from cluster; generate an absolute
(including glob.baseurl) endpoint from all the prefix words,
and a dictionary of non-prefix parameters
"""
url = ''
params = []
for desc in sig:
if desc.t == CephPrefix:
url += '/' + desc.instance.prefix
elif desc.t == CephChoices and \
len(desc.instance.strings) == 1 and \
desc.req and \
not str(desc.instance).startswith('--'):
url += '/' + str(desc.instance)
else:
params.append(desc)
return glob.baseurl + url, params
def concise_sig_for_uri(sig):
"""
Return a generic description of how one would send a REST request for sig
"""
prefix = []
args = []
for d in sig:
if d.t == CephPrefix:
prefix.append(d.instance.prefix)
else:
args.append(d.name + '=' + str(d))
sig = '/'.join(prefix)
if args:
sig += '?' + '&'.join(args)
return sig
def show_human_help(prefix):
"""
Dump table showing commands matching prefix
"""
# XXX this really needs to be a template
#s = '<html><body><style>.colhalf { width: 50%;} body{word-wrap:break-word;}</style>'
#s += '<table border=1><col class=colhalf /><col class=colhalf />'
#s += '<th>Possible commands:</th>'
# XXX the above mucking with css doesn't cause sensible columns.
s = '<html><body><table border=1><th>Possible commands:</th><th>Method</th><th>Description</th>'
possible = []
permmap = {'r':'GET', 'rw':'PUT'}
line = ''
for cmdsig in sorted(glob.sigdict.itervalues(), cmp=descsort):
concise = concise_sig(cmdsig['sig'])
if concise.startswith(prefix):
line = ['<tr><td>']
wrapped_sig = textwrap.wrap(concise_sig_for_uri(cmdsig['sig']), 40)
for sigline in wrapped_sig:
line.append(flask.escape(sigline) + '\n')
line.append('</td><td>')
line.append(permmap[cmdsig['perm']])
line.append('</td><td>')
line.append(flask.escape(cmdsig['help']))
line.append('</td></tr>\n')
s += ''.join(line)
s += '</table></body></html>'
if line:
return s
else:
return ''
@app.before_request
def log_request():
"""
For every request, log it. XXX Probably overkill for production
"""
app.logger.info(flask.request.url + " from " + flask.request.remote_addr + " " + flask.request.user_agent.string)
app.logger.debug("Accept: %s", flask.request.accept_mimetypes.values())
@app.route('/')
def root_redir():
return flask.redirect(glob.baseurl)
def make_response(fmt, output, statusmsg, errorcode):
"""
If formatted output, cobble up a response object that contains the
output and status wrapped in enclosing objects; if nonformatted, just
use output. Return HTTP status errorcode in any event.
"""
response = output
if fmt:
if 'json' in fmt:
try:
native_output = json.loads(output or '[]')
response = json.dumps({"output":native_output,
"status":statusmsg})
except:
return flask.make_response("Error decoding JSON from " +
output, 500)
elif 'xml' in fmt:
# one is tempted to do this with xml.etree, but figuring out how
# to 'un-XML' the XML-dumped output so it can be reassembled into
# a piece of the tree here is beyond me right now.
#ET = xml.etree.ElementTree
#resp_elem = ET.Element('response')
#o = ET.SubElement(resp_elem, 'output')
#o.text = output
#s = ET.SubElement(resp_elem, 'status')
#s.text = statusmsg
#response = ET.tostring(resp_elem)
response = '''
<response>
<output>
{0}
</output>
<status>
{1}
</status>
</response>'''.format(response, xml.sax.saxutils.escape(statusmsg))
return flask.make_response(response, errorcode)
def handler(catchall_path=None, fmt=None):
"""
Main endpoint handler; generic for every endpoint
"""
if (catchall_path):
ep = catchall_path.replace('.<fmt>', '')
else:
ep = flask.request.endpoint.replace('.<fmt>', '')
if ep[0] != '/':
ep = '/' + ep
# Extensions override Accept: headers override defaults
if not fmt:
if 'application/json' in flask.request.accept_mimetypes.values():
fmt = 'json'
elif 'application/xml' in flask.request.accept_mimetypes.values():
fmt = 'xml'
# demand that endpoint begin with glob.baseurl
if not ep.startswith(glob.baseurl):
return make_response(fmt, '', 'Page not found', 404)
relative_endpoint = ep[len(glob.baseurl)+1:]
prefix = ' '.join(relative_endpoint.split('/')).strip()
# show "match as much as you gave me" help for unknown endpoints
if not ep in glob.urls:
helptext = show_human_help(prefix)
if helptext:
resp = flask.make_response(helptext, 400)
resp.headers['Content-Type'] = 'text/html'
return resp
else:
return make_response(fmt, '', 'Invalid endpoint ' + ep, 400)
urldict = glob.urls[ep]
paramsig = urldict['paramsig']
# allow '?help' for any specifically-known endpoint
if 'help' in flask.request.args:
response = flask.make_response('{0}: {1}'.\
format(prefix + concise_sig(paramsig), urldict['help']))
response.headers['Content-Type'] = 'text/plain'
return response
# if there are parameters for this endpoint, process them
if paramsig:
args = {}
for k, l in flask.request.args.iterlists():
if len(l) == 1:
args[k] = l[0]
else:
args[k] = l
# is this a valid set of params?
try:
argdict = validate(args, paramsig)
except Exception as e:
return make_response(fmt, '', str(e) + '\n', 400)
else:
# no parameters for this endpoint; complain if args are supplied
if flask.request.args:
return make_response(fmt, '', ep + 'takes no params', 400)
argdict = {}
argdict['format'] = fmt or 'plain'
argdict['module'] = urldict['module']
argdict['perm'] = urldict['perm']
app.logger.debug('sending command prefix %s argdict %s', prefix, argdict)
ret, outbuf, outs = json_command(glob.cluster, prefix=prefix,
inbuf=flask.request.data, argdict=argdict)
if ret:
return make_response(fmt, '', 'Error: {0} ({1})'.format(outs, ret), 400)
response = make_response(fmt, outbuf, outs or 'OK', 200)
if fmt:
contenttype = 'application/' + fmt.replace('-pretty','')
else:
contenttype = 'text/plain'
response.headers['Content-Type'] = contenttype
return response
#
# main
#
addr, port = api_setup()
parsed_args = parse_args()
if parsed_args.conf:
os.environ['CEPH_CONF'] = parsed_args.conf
if parsed_args.cluster:
os.environ['CEPH_CLUSTER_NAME'] = parsed_args.cluster
if parsed_args.name:
os.environ['CEPH_NAME'] = parsed_args.name
if __name__ == '__main__':
import inspect
files = [os.path.split(fr[1])[-1] for fr in inspect.stack()]
if 'pdb.py' in files:
app.run(host=addr, port=port, debug=True, use_reloader=False, use_debugger=False)
else:
app.run(host=addr, port=port, debug=True)
# import now that env vars are available to imported module
try:
import ceph_rest_api
except Exception as e:
print >> sys.stderr, "Error importing ceph_rest_api: ", str(e)
sys.exit(1)
# importing ceph_rest_api has set module globals 'app', 'addr', and 'port'
files = [os.path.split(fr[1])[-1] for fr in inspect.stack()]
if 'pdb.py' in files:
ceph_rest_api.app.run(host=ceph_rest_api.addr, port=ceph_rest_api.port,
debug=True, use_reloader=False, use_debugger=False)
else:
ceph_rest_api.app.run(host=ceph_rest_api.addr, port=ceph_rest_api.port,
debug=True)

414
src/pybind/ceph_rest_api.py Executable file
View File

@ -0,0 +1,414 @@
#!/usr/bin/python
# vim: ts=4 sw=4 smarttab expandtab
import os
import sys
import argparse
import collections
import ConfigParser
import errno
import json
import logging
import logging.handlers
import rados
import textwrap
import xml.etree.ElementTree
import xml.sax.saxutils
import flask
from ceph_argparse import *
#
# Globals
#
APPNAME = '__main__'
DEFAULT_BASEURL = '/api/v0.1'
DEFAULT_ADDR = '0.0.0.0:5000'
DEFAULT_LOG_LEVEL = 'warning'
DEFAULT_CLIENTNAME = 'client.restapi'
DEFAULT_LOG_FILE = '/var/log/ceph/' + DEFAULT_CLIENTNAME + '.log'
app = flask.Flask(APPNAME)
LOGLEVELS = {
'critical':logging.CRITICAL,
'error':logging.ERROR,
'warning':logging.WARNING,
'info':logging.INFO,
'debug':logging.DEBUG,
}
# my globals, in a named tuple for usage clarity
glob = collections.namedtuple('gvars', 'cluster urls sigdict baseurl')
glob.cluster = None
glob.urls = {}
glob.sigdict = {}
glob.baseurl = ''
def load_conf(clustername='ceph', conffile=None):
import contextlib
class _TrimIndentFile(object):
def __init__(self, fp):
self.fp = fp
def readline(self):
line = self.fp.readline()
return line.lstrip(' \t')
def _optionxform(s):
s = s.replace('_', ' ')
s = '_'.join(s.split())
return s
def parse(fp):
cfg = ConfigParser.RawConfigParser()
cfg.optionxform = _optionxform
ifp = _TrimIndentFile(fp)
cfg.readfp(ifp)
return cfg
def load(path):
f = file(path)
with contextlib.closing(f):
return parse(f)
if conffile:
# from CEPH_CONF
return load(conffile)
else:
for path in [
'/etc/ceph/{0}.conf'.format(clustername),
os.path.expanduser('~/.ceph/{0}.conf'.format(clustername)),
'{0}.conf'.format(clustername),
]:
if os.path.exists(path):
return load(path)
raise EnvironmentError('No conf file found for "{0}"'.format(clustername))
def get_conf(cfg, clientname, key):
try:
return cfg.get(clientname, 'restapi_' + key)
except ConfigParser.NoOptionError:
return None
# XXX this is done globally, and cluster connection kept open; there
# are facilities to pass around global info to requests and to
# tear down connections between requests if it becomes important
def api_setup():
"""
Initialize the running instance. Open the cluster, get the command
signatures, module, perms, and help; stuff them away in the glob.urls
dict.
"""
conffile = os.environ.get('CEPH_CONF', '')
clustername = os.environ.get('CEPH_CLUSTER_NAME', 'ceph')
clientname = os.environ.get('CEPH_NAME', DEFAULT_CLIENTNAME)
try:
err = ''
cfg = load_conf(clustername, conffile)
except Exception as e:
err = "Can't load Ceph conf file: " + str(e)
app.logger.critical(err)
app.logger.critical("CEPH_CONF: %s", conffile)
app.logger.critical("CEPH_CLUSTER_NAME: %s", clustername)
raise EnvironmentError(err)
client_logfile = '/var/log/ceph' + clientname + '.log'
glob.cluster = rados.Rados(name=clientname, conffile=conffile)
glob.cluster.connect()
glob.baseurl = get_conf(cfg, clientname, 'base_url') or DEFAULT_BASEURL
if glob.baseurl.endswith('/'):
glob.baseurl
addr = get_conf(cfg, clientname, 'public_addr') or DEFAULT_ADDR
addrport = addr.rsplit(':', 1)
addr = addrport[0]
if len(addrport) > 1:
port = addrport[1]
else:
port = DEFAULT_ADDR.rsplit(':', 1)
port = int(port)
loglevel = get_conf(cfg, clientname, 'log_level') or DEFAULT_LOG_LEVEL
logfile = get_conf(cfg, clientname, 'log_file') or client_logfile
app.logger.addHandler(logging.handlers.WatchedFileHandler(logfile))
app.logger.setLevel(LOGLEVELS[loglevel.lower()])
for h in app.logger.handlers:
h.setFormatter(logging.Formatter(
'%(asctime)s %(name)s %(levelname)s: %(message)s'))
ret, outbuf, outs = json_command(glob.cluster,
prefix='get_command_descriptions')
if ret:
err = "Can't contact cluster for command descriptions: {0}".format(outs)
app.logger.error(err)
raise EnvironmentError(ret, err)
try:
glob.sigdict = parse_json_funcsigs(outbuf, 'rest')
except Exception as e:
err = "Can't parse command descriptions: {}".format(e)
app.logger.error(err)
raise EnvironmentError(err)
# glob.sigdict maps "cmdNNN" to a dict containing:
# 'sig', an array of argdescs
# 'help', the helptext
# 'module', the Ceph module this command relates to
# 'perm', a 'rwx*' string representing required permissions, and also
# a hint as to whether this is a GET or POST/PUT operation
# 'avail', a comma-separated list of strings of consumers that should
# display this command (filtered by parse_json_funcsigs() above)
glob.urls = {}
for cmdnum, cmddict in glob.sigdict.iteritems():
cmdsig = cmddict['sig']
url, params = generate_url_and_params(cmdsig)
if url in glob.urls:
continue
else:
perm = cmddict['perm']
urldict = {'paramsig':params,
'help':cmddict['help'],
'module':cmddict['module'],
'perm':perm,
}
method_dict = {'r':['GET'],
'w':['PUT', 'DELETE']}
for k in method_dict.iterkeys():
if k in perm:
methods = method_dict[k]
app.add_url_rule(url, url, handler, methods=methods)
glob.urls[url] = urldict
url += '.<fmt>'
app.add_url_rule(url, url, handler, methods=methods)
glob.urls[url] = urldict
app.logger.debug("urls added: %d", len(glob.urls))
app.add_url_rule('/<path:catchall_path>', '/<path:catchall_path>',
handler, methods=['GET', 'PUT'])
return addr, port
def generate_url_and_params(sig):
"""
Digest command signature from cluster; generate an absolute
(including glob.baseurl) endpoint from all the prefix words,
and a dictionary of non-prefix parameters
"""
url = ''
params = []
for desc in sig:
if desc.t == CephPrefix:
url += '/' + desc.instance.prefix
elif desc.t == CephChoices and \
len(desc.instance.strings) == 1 and \
desc.req and \
not str(desc.instance).startswith('--'):
url += '/' + str(desc.instance)
else:
params.append(desc)
return glob.baseurl + url, params
def concise_sig_for_uri(sig):
"""
Return a generic description of how one would send a REST request for sig
"""
prefix = []
args = []
for d in sig:
if d.t == CephPrefix:
prefix.append(d.instance.prefix)
else:
args.append(d.name + '=' + str(d))
sig = '/'.join(prefix)
if args:
sig += '?' + '&'.join(args)
return sig
def show_human_help(prefix):
"""
Dump table showing commands matching prefix
"""
# XXX this really needs to be a template
#s = '<html><body><style>.colhalf { width: 50%;} body{word-wrap:break-word;}</style>'
#s += '<table border=1><col class=colhalf /><col class=colhalf />'
#s += '<th>Possible commands:</th>'
# XXX the above mucking with css doesn't cause sensible columns.
s = '<html><body><table border=1><th>Possible commands:</th><th>Method</th><th>Description</th>'
possible = []
permmap = {'r':'GET', 'rw':'PUT'}
line = ''
for cmdsig in sorted(glob.sigdict.itervalues(), cmp=descsort):
concise = concise_sig(cmdsig['sig'])
if concise.startswith(prefix):
line = ['<tr><td>']
wrapped_sig = textwrap.wrap(concise_sig_for_uri(cmdsig['sig']), 40)
for sigline in wrapped_sig:
line.append(flask.escape(sigline) + '\n')
line.append('</td><td>')
line.append(permmap[cmdsig['perm']])
line.append('</td><td>')
line.append(flask.escape(cmdsig['help']))
line.append('</td></tr>\n')
s += ''.join(line)
s += '</table></body></html>'
if line:
return s
else:
return ''
@app.before_request
def log_request():
"""
For every request, log it. XXX Probably overkill for production
"""
app.logger.info(flask.request.url + " from " + flask.request.remote_addr + " " + flask.request.user_agent.string)
app.logger.debug("Accept: %s", flask.request.accept_mimetypes.values())
@app.route('/')
def root_redir():
return flask.redirect(glob.baseurl)
def make_response(fmt, output, statusmsg, errorcode):
"""
If formatted output, cobble up a response object that contains the
output and status wrapped in enclosing objects; if nonformatted, just
use output. Return HTTP status errorcode in any event.
"""
response = output
if fmt:
if 'json' in fmt:
try:
native_output = json.loads(output or '[]')
response = json.dumps({"output":native_output,
"status":statusmsg})
except:
return flask.make_response("Error decoding JSON from " +
output, 500)
elif 'xml' in fmt:
# one is tempted to do this with xml.etree, but figuring out how
# to 'un-XML' the XML-dumped output so it can be reassembled into
# a piece of the tree here is beyond me right now.
#ET = xml.etree.ElementTree
#resp_elem = ET.Element('response')
#o = ET.SubElement(resp_elem, 'output')
#o.text = output
#s = ET.SubElement(resp_elem, 'status')
#s.text = statusmsg
#response = ET.tostring(resp_elem)
response = '''
<response>
<output>
{0}
</output>
<status>
{1}
</status>
</response>'''.format(response, xml.sax.saxutils.escape(statusmsg))
return flask.make_response(response, errorcode)
def handler(catchall_path=None, fmt=None):
"""
Main endpoint handler; generic for every endpoint
"""
if (catchall_path):
ep = catchall_path.replace('.<fmt>', '')
else:
ep = flask.request.endpoint.replace('.<fmt>', '')
if ep[0] != '/':
ep = '/' + ep
# Extensions override Accept: headers override defaults
if not fmt:
if 'application/json' in flask.request.accept_mimetypes.values():
fmt = 'json'
elif 'application/xml' in flask.request.accept_mimetypes.values():
fmt = 'xml'
# demand that endpoint begin with glob.baseurl
if not ep.startswith(glob.baseurl):
return make_response(fmt, '', 'Page not found', 404)
relative_endpoint = ep[len(glob.baseurl)+1:]
prefix = ' '.join(relative_endpoint.split('/')).strip()
# show "match as much as you gave me" help for unknown endpoints
if not ep in glob.urls:
helptext = show_human_help(prefix)
if helptext:
resp = flask.make_response(helptext, 400)
resp.headers['Content-Type'] = 'text/html'
return resp
else:
return make_response(fmt, '', 'Invalid endpoint ' + ep, 400)
urldict = glob.urls[ep]
paramsig = urldict['paramsig']
# allow '?help' for any specifically-known endpoint
if 'help' in flask.request.args:
response = flask.make_response('{0}: {1}'.\
format(prefix + concise_sig(paramsig), urldict['help']))
response.headers['Content-Type'] = 'text/plain'
return response
# if there are parameters for this endpoint, process them
if paramsig:
args = {}
for k, l in flask.request.args.iterlists():
if len(l) == 1:
args[k] = l[0]
else:
args[k] = l
# is this a valid set of params?
try:
argdict = validate(args, paramsig)
except Exception as e:
return make_response(fmt, '', str(e) + '\n', 400)
else:
# no parameters for this endpoint; complain if args are supplied
if flask.request.args:
return make_response(fmt, '', ep + 'takes no params', 400)
argdict = {}
argdict['format'] = fmt or 'plain'
argdict['module'] = urldict['module']
argdict['perm'] = urldict['perm']
app.logger.debug('sending command prefix %s argdict %s', prefix, argdict)
ret, outbuf, outs = json_command(glob.cluster, prefix=prefix,
inbuf=flask.request.data, argdict=argdict)
if ret:
return make_response(fmt, '', 'Error: {0} ({1})'.format(outs, ret), 400)
response = make_response(fmt, outbuf, outs or 'OK', 200)
if fmt:
contenttype = 'application/' + fmt.replace('-pretty','')
else:
contenttype = 'text/plain'
response.headers['Content-Type'] = contenttype
return response
addr, port = api_setup()