Account DB and Container DB

DB

Database code for Swift

class swift.common.db.AccountBroker(db_file, timeout=25, logger=None, account=None, container=None, pending_timeout=10, stale_reads_ok=False)

Bases: swift.common.db.DatabaseBroker

Encapsulates working with a account database.

can_delete_db(cutoff)

Check if the accont DB can be deleted.

Returns:True if the account can be deleted, False otherwise
create_account_stat_table(conn, put_timestamp)

Create account_stat table which is specific to the account DB.

Parameters:
  • conn – DB connection object
  • put_timestamp – put timestamp
create_container_table(conn)

Create container table which is specific to the account DB.

Parameter:conn – DB connection object
empty()

Check if the account DB is empty.

Returns:True if the database has no active containers.
get_container_timestamp(container_name)

Get the put_timestamp of a container.

Parameter:container_name – container name
Returns:put_timestamp of the container
get_info()

Get global data for the account.

Returns:a tuple of (account, created_at, put_timestamp, delete_timestamp, container_count, object_count, bytes_used, hash, id)
get_random_containers(max_count=100)

Get random containers from the DB. This is used by the account_auditor when testing random containerss for existence.

Parameter:max_count – maximum number of containers to get
Returns:list of container names
is_deleted()

Check if the account DB is considered to be deleted.

Returns:True if the account DB is considered to be deleted, False otherwise
is_status_deleted()
Only returns true if the status field is set to DELETED.
list_containers_iter(limit, marker, prefix, delimiter)

Get a list of containerss sorted by name starting at marker onward, up to limit entries. Entries will begin with the prefix and will not have the delimiter after the prefix.

Parameters:
  • limit – maximum number of entries to get
  • marker – marker query
  • prefix – prefix query
  • delimeter – delimeter for query
Returns:

list of tuples of (name, object_count, bytes_used, 0)

merge_items(item_list, source=None)

Merge items into the container table.

Parameters:
  • item_list – list of dictionaries of {‘name’, ‘put_timestamp’, ‘delete_timestamp’, ‘object_count’, ‘bytes_used’, ‘deleted’}
  • source – if defined, update incoming_sync with the source
put_container(name, put_timestamp, delete_timestamp, object_count, bytes_used)

Create a container with the given attributes.

Parameters:
  • name – name of the container to create
  • put_timestamp – put_timestamp of the container to create
  • delete_timestamp – delete_timestamp of the container to create
  • object_count – number of objects in the container
  • bytes_used – number of bytes used by the container
reclaim(container_timestamp, sync_timestamp)

Delete rows from the container table that are marked deleted and whose created_at timestamp is < object_timestamp. Also deletes rows from incoming_sync and outgoing_sync where the updated_at timestamp is < sync_timestamp.

Parameters:
  • object_timestamp – max created_at timestamp of container rows to delete
  • sync_timestamp – max update_at timestamp of sync rows to delete
update_put_timestamp(timestamp)

Update the put_timestamp. Only modifies it if it is greater than the current timestamp.

Parameter:timestamp – put timestamp
swift.common.db.BROKER_TIMEOUT
Timeout for trying to connect to a DB
class swift.common.db.ContainerBroker(db_file, timeout=25, logger=None, account=None, container=None, pending_timeout=10, stale_reads_ok=False)

Bases: swift.common.db.DatabaseBroker

Encapsulates working with a container database.

create_container_stat_table(conn, put_timestamp=None)

Create the container_stat table which is specifc to the container DB.

Parameters:
  • conn – DB connection object
  • put_timestamp – put timestamp
create_object_table(conn)

Create the object table which is specifc to the container DB.

Parameter:conn – DB connection object
delete_object(name, timestamp)

Mark an object deleted.

Parameters:
  • name – object name to be deleted
  • timestamp – timestamp when the object was marked as deleted
empty()

Check if the DB is empty.

Returns:True if the database has no active objects, False otherwise
get_info()

Get global data for the container.

Returns:a tuple of (account, container, created_at, put_timestamp, delete_timestamp, object_count, bytes_used, reported_put_timestamp, reported_delete_timestamp, reported_object_count, reported_bytes_used, hash, id)
get_random_objects(max_count=100)

Get random objects from the DB. This is used by the container_auditor when testing random objects for existence.

Parameter:max_count – maximum number of objects to get
Returns:list of object names
is_deleted(timestamp=None)

Check if the DB is considered to be deleted.

Returns:True if the DB is considered to be deleted, False otherwise
list_objects_iter(limit, marker, prefix, delimiter, path=None, format=None)

Get a list of objects sorted by name starting at marker onward, up to limit entries. Entries will begin with the prefix and will not have the delimiter after the prefix.

Parameters:
  • limit – maximum number of entries to get
  • marker – marker query
  • prefix – prefix query
  • delimeter – delimeter for query
  • path – if defined, will set the prefix and delimter based on the path
  • format – TOOD: remove as it is no longer used
Returns:

list of tuples of (name, created_at, size, content_type, etag)

merge_items(item_list, source=None)

Merge items into the object table.

Parameters:
  • item_list – list of dictionaries of {‘name’, ‘created_at’, ‘size’, ‘content_type’, ‘etag’, ‘deleted’}
  • source – if defined, update incoming_sync with the source
put_object(name, timestamp, size, content_type, etag, deleted=0)

Creates an object in the DB with its metadata.

Parameters:
  • name – object name to be created
  • timestamp – timestamp of when the object was created
  • size – object size
  • content_type – object content-type
  • etag – object etag
  • deleted – if True, marks the object as deleted and sets the deteleted_at timestamp to timestamp
reclaim(object_timestamp, sync_timestamp)

Delete rows from the object table that are marked deleted and whose created_at timestamp is < object_timestamp. Also deletes rows from incoming_sync and outgoing_sync where the updated_at timestamp is < sync_timestamp.

Parameters:
  • object_timestamp – max created_at timestamp of object rows to delete
  • sync_timestamp – max update_at timestamp of sync rows to delete
reported(put_timestamp, delete_timestamp, object_count, bytes_used)

Update reported stats.

Parameters:
  • put_timestamp – put_timestamp to update
  • delete_timestamp – delete_timestamp to update
  • object_count – object_count to update
  • bytes_used – bytes_used to update
update_put_timestamp(timestamp)

Update the put_timestamp. Only modifies it if it is greater than the current timestamp.

Parameter:timestamp – put timestamp
class swift.common.db.DatabaseBroker(db_file, timeout=25, logger=None, account=None, container=None, pending_timeout=10, stale_reads_ok=False)

Bases: object

Encapsulates working with a database.

delete_db(timestamp)

Mark the DB as deleted

Parameter:timestamp – delete timestamp
get(*args, **kwds)
Use with the “with” statement; returns a database connection.
get_items_since(start, count)

Get a list of objects in the database between start and end.

Parameters:
  • start – start ROWID
  • count – number to get
Returns:

list of objects between start and end

get_replication_info()

Get information about the DB required for replication.

Returns:tuple of (hash, id, created_at, put_timestamp, delete_timestamp) from the DB
get_sync(id, incoming=True)

Gets the most recent sync point for a server from the sync table.

Parameters:
  • id – remote ID to get the sync_point for
  • incoming – if True, get the last incoming sync, otherwise get the last outgoing sync
Returns:

the sync point, or -1 if the id doesn’t exist.

get_syncs(incoming=True)

Get a serialized copy of the sync table.

Parameter:incoming – if True, get the last incoming sync, otherwise get the last outgoing sync
Returns:list of {‘remote_id’, ‘sync_point’}
initialize(put_timestamp=None)

Create the DB

Parameter:put_timestamp – timestamp of initial PUT request
lock(*args, **kwds)
Use with the “with” statement; locks a database.
merge_syncs(sync_points, incoming=True)

Merge a list of sync points with the incoming sync table.

Parameters:
  • sync_points – list of sync points where a sync point is a dict of {‘sync_point’, ‘remote_id’}
  • incoming – if True, get the last incoming sync, otherwise get the last outgoing sync
merge_timestamps(created_at, put_timestamp, delete_timestamp)

Used in replication to handle updating timestamps.

Parameters:
  • created_at – create timestamp
  • put_timestamp – put timestamp
  • delete_timestamp – delete timestamp
newid(remote_id)

Re-id the database. This should be called after an rsync.

Parameter:remote_id – the ID of the remote database being rsynced in
exception swift.common.db.DatabaseConnectionError(path, msg, timeout=0)

Bases: sqlite3.DatabaseError

More friendly error messages for DB Errors.

class swift.common.db.GreenDBConnection(*args, **kwargs)

Bases: sqlite3.Connection

SQLite DB Connection handler that plays well with eventlet.

commit()
execute(*args, **kwargs)
swift.common.db.PENDING_CAP
Max number of pending entries
swift.common.db.PICKLE_PROTOCOL
Pickle protocol to use
swift.common.db.chexor(old, name, timestamp)

Each entry in the account and container databases is XORed by the 128-bit hash on insert or delete. This serves as a rolling, order-independent hash of the contents. (check + XOR)

Parameters:
  • old – hex representation of the current DB hash
  • name – name of the object or container being inserted
  • timestamp – timestamp of the new record
Returns:

a hex representation of the new hash value

swift.common.db.dict_factory(crs, row)
This should only be used when you need a real dict, i.e. when you’re going to serialize the results.
swift.common.db.get_db_connection(path, timeout=30, okay_to_create=False)

Returns a properly configured SQLite database connection.

Parameters:
  • path – path to DB
  • timeout – timeout for connection
  • okay_to_create – if True, create the DB if it doesn’t exist
Returns:

DB connection object

DB replicator

class swift.common.db_replicator.ReplConnection(node, partition, hash_, logger)

Bases: swift.common.bufferedhttp.BufferedHTTPConnection

Helper to simplify POSTing to a remote server.

post(*args)

Make an HTTP POST request

Parameter:args – list of json-encodable objects
Returns:httplib response object
class swift.common.db_replicator.Replicator(server_conf, replicator_conf)

Bases: object

Implements the logic for directing db replication.

replicate_forever()
Replicate dbs under the given root in an infinite loop.
replicate_once()
Run a replication pass once.
roundrobin_datadirs(datadirs)

Generator to walk the data dirs in a round robin manner, evenly hitting each device on the system.

Parameter:datadirs – a list of paths to walk
class swift.common.db_replicator.ReplicatorRpc(root, datadir, broker_class, mount_check=True)

Bases: object

Handle Replication RPC calls. TODO: redbo document please :)

complete_rsync(drive, db_file, args)
dispatch(post_args, args)
merge_items(broker, args)
merge_syncs(broker, args)
rsync_then_merge(drive, db_file, args)
sync(broker, args)
swift.common.db_replicator.quarantine_db(object_file, server_type)

In the case that a corrupt file is found, move it to a quarantined area to allow replication to fix it.

Parameters:
  • object_file – path to corrupt file
  • server_type – type of file that is corrupt (‘container’ or ‘account’)

Table Of Contents

Previous topic

Container

Next topic

Object

This Page