Object

Object Server

Object Server for Swift

class swift.obj.server.DiskFile(path, device, partition, account, container, obj, keep_data_fp=False, disk_chunk_size=65536)

Bases: object

Manage object files on disk.

Parameters:
  • path – path to devices on the node
  • device – device name
  • partition – partition on the device the object lives in
  • account – account name for the object
  • container – container name for the object
  • obj – object name for the object
  • keep_data_fp – if True, don’t close the fp, otherwise close it
  • disk_chunk_Size – size of chunks on file reads
app_iter_range(start, stop)
Returns an iterator over the data file for range (start, stop)
close()
Close the file.
is_deleted()

Check if the file is deleted.

Returns:True if the file doesn’t exist or has been flagged as deleted.
mkstemp(*args, **kwds)
Contextmanager to make a temporary file.
put(fd, tmppath, metadata, extension='.data')

Finalize writing the file on disk, and renames it from the temp file to the real location. This should be called after the data has been written to the temp file.

Params fd:

file descriptor of the temp file

Parameters:
  • tmppath – path to the temporary file being used
  • metadata – dictionary of metada to be written
  • extention – extension to be used when making the file
unlinkold(timestamp)

Remove any older versions of the object file. Any file that has an older timestamp than timestamp will be deleted.

Parameter:timestamp – timestamp to compare with each file
class swift.obj.server.ObjectController(conf)

Bases: object

Implements the WSGI application for the Swift Object Server.

DELETE(request)
Handle HTTP DELETE requests for the Swift Object Server.
GET(request)
Handle HTTP GET requests for the Swift Object Server.
HEAD(request)
Handle HTTP HEAD requests for the Swift Object Server.
POST(request)
Handle HTTP POST requests for the Swift Object Server.
PUT(request)
Handle HTTP PUT requests for the Swift Object Server.
REPLICATE(request)
Handle REPLICATE requests for the Swift Object Server. This is used by the object replicator to get hashes for directories.
container_update(op, account, container, obj, headers_in, headers_out, objdevice)

Update the container when objects are updated.

Parameters:
  • op – operation performed (ex: ‘PUT’, or ‘DELETE’)
  • account – account name for the object
  • container – container name for the object
  • obj – object name
  • headers_in – dictionary of headers from the original request
  • headers_out – dictionary of headers to send in the container request
  • objdevice – device name that the object is in
swift.obj.server.read_metadata(fd)

Helper function to read the pickled metadata from an object file.

Parameter:fd – file descriptor to load the metadata from
Returns:dictionary of metadata

Object Replicator

class swift.obj.replicator.ObjectReplicator(conf, logger)

Bases: object

Replicate objects.

Encapsulates most logic and data needed by the object replication process. Each call to .run() performs one replication pass. It’s up to the caller to do this in a loop.

check_ring()

Check to see if the ring has been updated

Returns:boolean indicating whether or not the ring has changed
heartbeat()
Loop that runs in the background during replication. It periodically logs progress and attempts to detect lockups, killing any running coroutines if the replicator hasn’t made progress since last hearbeat.
kill_coros()
Utility function that kills all coroutines currently running.
rsync(node, job, suffixes)

Synchronize local suffix directories from a partition with a remote node.

Parameters:
  • node – the “dev” entry for the remote node to sync with
  • job – information about the partition being synced
  • suffixes – a list of suffixes which need to be pushed
Returns:

boolean indicating success or failure

run()
Run a replication pass
stats_line()
Logs various stats for the currently running replication pass.
update(job)

High-level method that replicates a single partition.

Parameter:job – a dict containing info about the partition to be replicated
update_deleted(job)

High-level method that replicates a single partition that doesn’t belong on this node.

Parameter:job – a dict containing info about the partition to be replicated
swift.obj.replicator.get_hashes(partition_dir, do_listdir=True, reclaim_age=604800)

Get a list of hashes for the suffix dir. do_listdir causes it to mistrust the hash cache for suffix existence at the (unexpectedly high) cost of a listdir. reclaim_age is just passed on to hash_suffix.

Parameters:
  • partition_dir – absolute path of partition to get hashes for
  • do_listdir – force existence check for all hashes in the partition
  • reclaim_age – age at which to remove tombstones
Returns:

tuple of (number of suffix dirs hashed, dictionary of hashes)

swift.obj.replicator.hash_suffix(path, reclaim_age)

Performs reclamation and returns an md5 of all (remaining) files.

Parameter:reclaim_age – age in seconds at which to remove tombstones
swift.obj.replicator.invalidate_hash(suffix_dir)

Invalidates the hash for a suffix_dir in the partition’s hashes file.

Parameter:suffix_dir – absolute path to suffix dir whose hash needs invalidating
swift.obj.replicator.recalculate_hashes(partition_dir, suffixes, reclaim_age=604800)

Recalculates hashes for the given suffixes in the partition and updates them in the partition’s hashes file.

Parameters:
  • partition_dir – directory of the partition in which to recalculate
  • suffixes – list of suffixes to recalculate
  • reclaim_age – age in seconds at which tombstones should be removed

Object Updater

class swift.obj.updater.ObjectUpdater(server_conf, updater_conf)

Bases: object

Update object information in container listings.

get_container_ring()
Get the container ring. Load it, if it hasn’t been yet.
object_sweep(device)

If there are async pendings on the device, walk each one and update.

Parameter:device – path to device
object_update(node, part, op, obj, headers)

Perform the object update to the container

Parameters:
  • node – node dictionary from the container ring
  • part – partition that holds the container
  • op – operation performed (ex: ‘POST’ or ‘DELETE’)
  • obj – object name being updated
  • headers – headers to send with the update
process_object_update(update_path, device)

Process the object information to be updated and update.

Parameters:
  • update_path – path to pickled object update file
  • device – path to device
update_forever()
Run the updater continuously.
update_once_single_threaded()
Run the updater once
swift.obj.updater.random()
random() -> x in the interval [0, 1).

Object Auditor

class swift.obj.auditor.ObjectAuditor(server_conf, auditor_conf)

Bases: object

Audit objects.

audit_forever()
Run the object audit until stopped.
audit_once()
Run the object audit once.
get_container_ring()

Get the container ring, loading it if neccesary.

Returns:container ring
object_audit(device)
Walk the device, and audit any objects found.
swift.obj.auditor.random()
random() -> x in the interval [0, 1).

Table Of Contents

Previous topic

Account DB and Container DB

Next topic

Developer’s Authorization

This Page