Misc

Exceptions

exception swift.common.exceptions.AuditException
Bases: exceptions.Exception
exception swift.common.exceptions.AuthException
Bases: exceptions.Exception
exception swift.common.exceptions.ChunkReadTimeout(seconds=None, exception=None)
Bases: eventlet.timeout.Timeout
exception swift.common.exceptions.ChunkWriteTimeout(seconds=None, exception=None)
Bases: eventlet.timeout.Timeout
exception swift.common.exceptions.ConnectionTimeout(seconds=None, exception=None)
Bases: eventlet.timeout.Timeout
exception swift.common.exceptions.DriveNotMounted
Bases: exceptions.Exception
exception swift.common.exceptions.LockTimeout(seconds=None, msg=None)
Bases: swift.common.exceptions.MessageTimeout
exception swift.common.exceptions.MessageTimeout(seconds=None, msg=None)
Bases: eventlet.timeout.Timeout

Constraints

swift.common.constraints.MAX_FILE_SIZE
Max file size allowed for objects
swift.common.constraints.MAX_META_COUNT
Max number of metadata items
swift.common.constraints.MAX_META_NAME_LENGTH
Max length of the name of a key for metadata
swift.common.constraints.MAX_META_OVERALL_SIZE
Max overall size of metadata
swift.common.constraints.MAX_META_VALUE_LENGTH
Max length of the value of a key for metadata
swift.common.constraints.MAX_OBJECT_NAME_LENGTH
Max object name length
swift.common.constraints.check_float(string)

Helper function for checking if a string can be converted to a float.

Parameter:string – string to be verified as a float
Returns:True if the string can be converted to a float, False otherwise
swift.common.constraints.check_metadata(req)

Check metadata sent for objects in the request headers.

Parameter:req – request object
Raises HTTPBadRequest:
 bad metadata
swift.common.constraints.check_mount(root, drive)

Verify that the path to the device is a mount point and mounted. This allows us to fast fail on drives that have been unmounted because of issues, and also prevents us for accidently filling up the root partition.

Parameters:
  • root – base path where the devices are mounted
  • drive – drive name to be checked
Returns:

True if it is a valid mounted device, False otherwise

swift.common.constraints.check_object_creation(req, object_name)

Check to ensure that everything is alright about an object to be created.

Parameters:
  • req – HTTP request object
  • object_name – name of object to be created
Raises HTTPRequestEntityTooLarge:
 

the object is too large

Raises HTTPLengthRequered:
 

missing content-length header and not a chunked request

Raises HTTPBadRequest:
 

missing or bad content-type header, or bad metadata

swift.common.constraints.check_xml_encodable(string)

Validate if a string can be encoded in xml.

Parameter:string – string to be validated
Returns:True if the string can be encoded in xml, False otherwise

Utils

Miscellaneous utility functions for use with Swift.

class swift.common.utils.NamedLogger(logger, server)

Bases: object

Cheesy version of the LoggerAdapter available in Python 3

class swift.common.utils.NullLogger
A no-op logger for eventlet wsgi.
swift.common.utils.compute_eta(start_time, current_value, final_value)

Compute an ETA. Now only if we could also have a progress bar...

Parameters:
  • start_time – Unix timestamp when the operation began
  • current_value – Current value
  • final_value – Final value
Returns:

ETA as a tuple of (length of time, unit of time) where unit of time is one of (‘h’, ‘m’, ‘s’)

swift.common.utils.drop_buffer_cache(fd, offset, length)

Drop ‘buffer’ cache for the given range of the given file.

Parameters:
  • fd – file descriptor
  • offset – start offset
  • length – length
swift.common.utils.drop_privileges(user)

Sets the userid of the current process

Parameter:user – User id to change privileges to
swift.common.utils.fallocate(fd, size)

Pre-allocate disk space for a file file.

Parameters:
  • fd – file descriptor
  • size – size to allocate (in bytes)
swift.common.utils.get_logger(conf, name)

Get the current system logger using config settings.

Log config and defaults:

log_facility = LOG_LOCAL0
log_level = INFO
Parameters:
  • conf – Configuration dict to read settings from
  • name – Name of the logger
swift.common.utils.get_param(req, name, default=None)

Get parameters from an HTTP request ensuring proper handling UTF-8 encoding.

Parameters:
  • req – Webob request object
  • name – parameter name
  • default – result to return if the parameter is not found
Returns:

HTTP request parameter value

swift.common.utils.get_time_units(time_amount)

Get a nomralized length of time in the largest unit of time (hours, minutes, or seconds.)

Parameter:time_amount – length of time in seconds
Returns:A touple of (length of time, unit of time) where unit of time is one of (‘h’, ‘m’, ‘s’)
swift.common.utils.hash_path(account, container=None, object=None, raw_digest=False)

Get the connonical hash for an account/container/object

Parameters:
  • account – Account
  • container – Container
  • object – Object
  • raw_digest – If True, return the raw version rather than a hex digest
Returns:

hash string

swift.common.utils.iter_devices_partitions(devices_dir, item_type)

Iterate over partitions accross all devices.

Parameters:
  • devices_dir – Path to devices
  • item_type – One of ‘accounts’, ‘containers’, or ‘objects’
Returns:

Each iteration returns a tuple of (device, partition)

swift.common.utils.lock_parent_directory(filename, timeout=10)

Context manager that acquires a lock on the parent directory of the given file path. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).

Parameters:
  • filename – file path of the parent directory to be locked
  • timeout – timeout (in seconds)
swift.common.utils.lock_path(*args, **kwds)

Context manager that acquires a lock on a directory. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).

Parameters:
  • directory – directory to be locked
  • timeout – timeout (in seconds)
swift.common.utils.mkdirs(path)

Ensures the path is a directory or makes it if not. Errors if the path exists but is a file or on permissions failure.

Parameter:path – path to create
swift.common.utils.normalize_timestamp(timestamp)

Format a timestamp (string or numeric) into a standardized xxxxxxxxxx.xxxxx format.

Parameter:timestamp – unix timestamp
Returns:normalized timestamp as a string
swift.common.utils.renamer(old, new)

Attempt to fix^H^H^Hhide race conditions like empty object directories being removed by backend processes during uploads, by retrying.

Parameters:
  • old – old path to be renamed
  • new – new path to be renamed to
swift.common.utils.split_path(path, minsegs=1, maxsegs=None, rest_with_last=False)

Validate and split the given HTTP request path.

Examples:

['a'] = split_path('/a')
['a', None] = split_path('/a', 1, 2)
['a', 'c'] = split_path('/a/c', 1, 2)
['a', 'c', 'o/r'] = split_path('/a/c/o/r', 1, 3, True)
Parameters:
  • path – HTTP Request path to be split
  • minsegs – Minimum number of segments to be extracted
  • maxsegs – Maximum number of segments to be extracted
  • rest_with_last – If True, trailing data will be returned as part of last segment. If False, and there is trailing data, raises ValueError.
Returns:

list of segments with a length of maxsegs (non-existant segments will return as None)

swift.common.utils.storage_directory(datadir, partition, hash)

Get the storage directory

Parameters:
  • datadir – Base data directory
  • partition – Partition
  • hash – Account, container or object hash
Returns:

Storage directory

Remove any file in a given path that that was last modified before mtime.

Parameter:path – Path to remove file from
Mtime:Timestamp of oldest file to keep
swift.common.utils.whataremyips()

Get the machine’s ip addresses using ifconfig

Returns:list of Strings of IPv4 ip addresses

Auth

WSGI

WSGI tools for use with swift.

swift.common.wsgi.monkey_patch_mimetools()
mimetools.Message defaults content-type to “text/plain” This changes it to default to None, so we can detect missing headers.
swift.common.wsgi.run_wsgi(app, conf, *args, **kwargs)

Loads common settings from conf, then instantiates app and runs the server using the specified number of workers.

Parameters:
  • app – WSGI callable
  • conf – Configuration dictionary

Client

Cloud Files client library used internally

exception swift.common.client.ClientException(msg, http_scheme='', http_host='', http_port='', http_path='', http_query='', http_status=0, http_reason='', http_device='')
Bases: exceptions.Exception
class swift.common.client.Connection(authurl, user, key, retries=5, preauthurl=None, preauthtoken=None, snet=False)

Bases: object

Convenience class to make requests that will also retry the request

delete_container(container)
Wrapper for delete_container
delete_object(container, obj)
Wrapper for delete_object
get_account(marker=None, limit=None, prefix=None, full_listing=False)
Wrapper for get_account
get_container(container, marker=None, limit=None, prefix=None, delimiter=None, full_listing=False)
Wrapper for get_container
get_object(container, obj, resp_chunk_size=None)
Wrapper for get_object
head_account()
Wrapper for head_account
head_container(container)
Wrapper for head_container
head_object(container, obj)
Wrapper for head_object
post_object(container, obj, metadata)
Wrapper for post_object
put_container(container)
Wrapper for put_container
put_object(container, obj, contents, metadata={}, content_length=None, etag=None, chunk_size=65536, content_type=None)
Wrapper for put_object
swift.common.client.delete_container(url, token, container, http_conn=None)

Delete a container

Parameters:
  • url – storage URL
  • token – auth token
  • container – container name to delete
  • http_conn – HTTP connection object (If None, it will create the conn object)
Raises ClientException:
 

HTTP DELETE request failed

swift.common.client.delete_object(url, token, container, name, http_conn=None)

Delete object

Parameters:
  • url – storage URL
  • token – auth token
  • container – container name that the object is in
  • name – object name to delete
  • http_conn – HTTP connection object (If None, it will create the conn object)
Raises ClientException:
 

HTTP DELETE request failed

swift.common.client.get_account(url, token, marker=None, limit=None, prefix=None, http_conn=None, full_listing=False)

Get a listing of containers for the account.

Parameters:
  • url – storage URL
  • token – auth token
  • marker – marker query
  • limit – limit query
  • prefix – prefix query
  • http_conn – HTTP connection object (If None, it will create the conn object)
  • full_listing – if True, return a full listing, else returns a max of 10000 listings
Returns:

a list of accounts

Raises ClientException:
 

HTTP GET request failed

swift.common.client.get_auth(url, user, key, snet=False)

Get authentication credentials

Parameters:
  • url – authentication URL
  • user – user to auth as
  • key – key or passowrd for auth
  • snet – use SERVICENET internal network default is False
Returns:

tuple of (storage URL, storage token, auth token)

Raises ClientException:
 

HTTP GET request to auth URL failed

swift.common.client.get_container(url, token, container, marker=None, limit=None, prefix=None, delimiter=None, http_conn=None, full_listing=False)

Get a listing of objects for the container.

Parameters:
  • url – storage URL
  • token – auth token
  • container – container name to get a listing for
  • marker – marker query
  • limit – limit query
  • prefix – prefix query
  • delimeter – string to delimit the queries on
  • http_conn – HTTP connection object (If None, it will create the conn object)
  • full_listing – if True, return a full listing, else returns a max of 10000 listings
Returns:

a list of objects

Raises ClientException:
 

HTTP GET request failed

swift.common.client.get_object(url, token, container, name, http_conn=None, resp_chunk_size=None)

Get an object

Parameters:
  • url – storage URL
  • token – auth token
  • container – container name that the object is in
  • name – object name to get
  • http_conn – HTTP connection object (If None, it will create the conn object)
  • resp_chunk_size – if defined, chunk size of data to read
Returns:

a list of objects

Raises ClientException:
 

HTTP GET request failed

swift.common.client.head_account(url, token, http_conn=None)

Get account stats.

Parameters:
  • url – storage URL
  • token – auth token
  • http_conn – HTTP connection object (If None, it will create the conn object)
Returns:

a tuple of (container count, object count, bytes used)

Raises ClientException:
 

HTTP HEAD request failed

swift.common.client.head_container(url, token, container, http_conn=None)

Get container stats.

Parameters:
  • url – storage URL
  • token – auth token
  • container – container name to get stats for
  • http_conn – HTTP connection object (If None, it will create the conn object)
Returns:

a tuple of (object count, bytes used)

Raises ClientException:
 

HTTP HEAD request failed

swift.common.client.head_object(url, token, container, name, http_conn=None)

Get object info

Parameters:
  • url – storage URL
  • token – auth token
  • container – container name that the object is in
  • name – object name to get info for
  • http_conn – HTTP connection object (If None, it will create the conn object)
Returns:

a tuple of (content type, content length, last modfied, etag, dictionary of metadata)

Raises ClientException:
 

HTTP HEAD request failed

swift.common.client.http_connection(url)

Make an HTTPConnection or HTTPSConnection

Parameter:url – url to connect to
Returns:tuple of (parsed url, connection object)
Raises ClientException:
 Unable to handle protocol scheme
swift.common.client.post_object(url, token, container, name, metadata, http_conn=None)

Change object metadata

Parameters:
  • url – storage URL
  • token – auth token
  • container – container name that the object is in
  • name – object name to change
  • metadata – dictionary of object metadata
  • http_conn – HTTP connection object (If None, it will create the conn object)
Raises ClientException:
 

HTTP POST request failed

swift.common.client.put_container(url, token, container, http_conn=None)

Create a container

Parameters:
  • url – storage URL
  • token – auth token
  • container – container name to create
  • http_conn – HTTP connection object (If None, it will create the conn object)
Raises ClientException:
 

HTTP PUT request failed

swift.common.client.put_object(url, token, container, name, contents, metadata={}, content_length=None, etag=None, chunk_size=65536, content_type=None, http_conn=None)

Put an object

Parameters:
  • url – storage URL
  • token – auth token
  • container – container name that the object is in
  • name – object name to put
  • contents – file like object to read object data from
  • metadata – dictionary of object metadata
  • content_length – value to send as content-length header
  • etag – etag of contents
  • chunk_size – chunk size of data to write
  • content_type – value to send as content-type header
  • http_conn – HTTP connection object (If None, it will create the conn object)
Returns:

etag from server response

Raises ClientException:
 

HTTP PUT request failed

swift.common.client.quote(value, safe='/')
Patched version of urllib.quote that encodes utf8 strings before quoting

Direct Client

Internal client library for making calls directly to the servers rather than through the proxy.

swift.common.direct_client.direct_delete_container(node, part, account, container, conn_timeout=5, response_timeout=15, headers={})
swift.common.direct_client.direct_delete_object(node, part, account, container, obj, conn_timeout=5, response_timeout=15, headers={})

Delete object directly from the object server.

Parameters:
  • node – node dictionary from the ring
  • part – partition the container is on
  • account – account name
  • container – container name
  • obj – object name
  • conn_timeout – timeout in seconds for establishing the connection
  • response_timeout – timeout in seconds for getting the response
Returns:

response from server

swift.common.direct_client.direct_get_container(node, part, account, container, marker=None, limit=None, prefix=None, delimiter=None, conn_timeout=5, response_timeout=15)

Get container listings directly from the container server.

Parameters:
  • node – node dictionary from the ring
  • part – partition the container is on
  • account – account name
  • container – container name
  • marker – marker query
  • limit – query limit
  • prefix – prefix query
  • delimeter – delimeter for the query
  • conn_timeout – timeout in seconds for establishing the connection
  • response_timeout – timeout in seconds for getting the response
Returns:

list of objects

swift.common.direct_client.direct_get_object(node, part, account, container, obj, conn_timeout=5, response_timeout=15)

Get object directly from the object server.

Parameters:
  • node – node dictionary from the ring
  • part – partition the container is on
  • account – account name
  • container – container name
  • obj – object name
  • conn_timeout – timeout in seconds for establishing the connection
  • response_timeout – timeout in seconds for getting the response
Returns:

object

swift.common.direct_client.direct_head_container(node, part, account, container, conn_timeout=5, response_timeout=15)

Request container information directly from the container server.

Parameters:
  • node – node dictionary from the ring
  • part – partition the container is on
  • account – account name
  • container – container name
  • conn_timeout – timeout in seconds for establishing the connection
  • response_timeout – timeout in seconds for getting the response
Returns:

tuple of (object count, bytes used)

swift.common.direct_client.direct_head_object(node, part, account, container, obj, conn_timeout=5, response_timeout=15)

Request object information directly from the object server.

Parameters:
  • node – node dictionary from the ring
  • part – partition the container is on
  • account – account name
  • container – container name
  • obj – object name
  • conn_timeout – timeout in seconds for establishing the connection
  • response_timeout – timeout in seconds for getting the response
Returns:

tuple of (content-type, object size, last modified timestamp, etag, metadata dictionary)

swift.common.direct_client.quote(value, safe='/')
swift.common.direct_client.retry(func, *args, **kwargs)

Helper function to retry a given function a number of times.

Parameters:
  • func – callable to be called
  • retries – number of retries
  • error_log – logger for errors
  • args – arguments to send to func
  • kwargs – keyward arguments to send to func (if retries or error_log are sent, they will be deleted from kwargs before sending on to func)
Returns:

restult of func

Buffered HTTP

Monkey Patch httplib.HTTPResponse to buffer reads of headers. This can improve performance when making large numbers of small HTTP requests. This module also provides helper functions to make HTTP connections using BufferedHTTPResponse.

Warning

If you use this, be sure that the libraries you are using do not access the socket directly (xmlrpclib, I’m looking at you :/), and instead make all calls through httplib.

class swift.common.bufferedhttp.BufferedHTTPConnection(host, port=None, strict=None, timeout=<object object at 0xf77c46e0>)

Bases: httplib.HTTPConnection

HTTPConnection class that uses BufferedHTTPResponse

response_class
alias of BufferedHTTPResponse
class swift.common.bufferedhttp.BufferedHTTPResponse(sock, debuglevel=0, strict=0, method=None)

Bases: httplib.HTTPResponse

HTTPResponse class that buffers reading of headers

swift.common.bufferedhttp.http_connect(ipaddr, port, device, partition, method, path, headers=None, query_string=None)

Helper function to create a HTTPConnection object that is buffered for backend Swift services.

Parameters:
  • ipaddr – IPv4 address to connect to
  • port – port to connect to
  • device – device of the node to query
  • partition – partition on the device
  • method – HTTP method to request (‘GET’, ‘PUT’, ‘POST’, etc.)
  • path – request path
  • headers – dictionary of headers
  • query_string – request query string
Returns:

HTTPConnection object

swift.common.bufferedhttp.http_connect_raw(ipaddr, port, method, path, headers=None, query_string=None)

Helper function to create a HTTPConnection object that is buffered.

Parameters:
  • ipaddr – IPv4 address to connect to
  • port – port to connect to
  • method – HTTP method to request (‘GET’, ‘PUT’, ‘POST’, etc.)
  • path – request path
  • headers – dictionary of headers
  • query_string – request query string
Returns:

HTTPConnection object

Healthcheck

class swift.common.healthcheck.HealthCheckController(*args, **kwargs)

Bases: object

Basic controller used for monitoring.

MemCacheD

Lucid comes with memcached: v1.4.2. Protocol documentation for that version is at:

http://github.com/memcached/memcached/blob/1.4.2/doc/protocol.txt

class swift.common.memcached.MemcacheRing(servers, connect_timeout=0.29999999999999999, io_timeout=2.0, tries=3)

Bases: object

Simple, consistent-hashed memcache client.

delete(key)

Deletes a key/value pair from memcache.

Parameter:key – key to be deleted
get(key)

Gets the object specified by key. It will also unpickle the object before returning if it is pickled in memcache.

Parameter:key – key
Returns:value of the key in memcache
get_multi(keys, server_key)

Gets multiple values from memcache for the given keys.

Parameters:
  • keys – keys for values to be retrieved from memcache
  • servery_key – key to use in determining which server in the ring is used
Returns:

list of values

incr(key, delta=1, timeout=0)

Increments a key which has a numeric value by delta. If the key can’t be found, it’s added as delta.

Parameters:
  • key – key
  • delta – amount to add to the value of key (or set as the value if the key is not found)
  • timeout – ttl in memcache
set(key, value, serialize=True, timeout=0)

Set a key/value pair in memcache

Parameters:
  • key – key
  • value – value
  • serialize – if True, value is pickled before sending to memcache
  • timeout – ttl in memcache
set_multi(mapping, server_key, serialize=True, timeout=0)

Sets multiple key/value pairs in memcache.

Parameters:
  • mapping – dictonary of keys and values to be set in memcache
  • servery_key – key to use in determining which server in the ring is used
  • serialize – if True, value is pickled before sending to memcache
  • timeout – ttl for memcache

Table Of Contents

Previous topic

Developer’s Authorization

This Page