Package flumotion :: Package component :: Package misc :: Package httpserver :: Module cachedprovider :: Class FileProviderLocalCachedPlug
[hide private]

Class FileProviderLocalCachedPlug

source code

extern.log.log.Loggable --+            
                          |            
            plugs.base.Plug --+        
                              |        
       plugs.base.ComponentPlug --+    
                                  |    
      fileprovider.FileProviderPlug --+
                                      |
            extern.log.log.Loggable --+
                                      |
                                     FileProviderLocalCachedPlug

WARNING: Currently does not work properly in combination with rate-control.

I'm caching the files taken from a mounted network file system to a shared local directory. Multiple instances can share the same cache directory, but it's recommended to use slightly different values for the property cleanup-high-watermark. I'm using the directory access time to know when the cache usage changed and keep an estimation of the cache usage for statistics.

I'm creating a unique thread to do the file copying block by block, for all files to be copied to the cache. Using a thread instead of a reactor.callLater 'loop' allow for higher copy throughput and do not slow down the mail loop when lots of files are copied at the same time. Simulations with real request logs show that using a thread gives better results than the equivalent asynchronous implementation.

Instance Methods [hide private]
 
__init__(self, args) source code
 
start(self, component) source code
 
stop(self, component) source code
 
startStatsUpdates(self, updater)
Start updating statistics.
source code
 
stopStatsUpdates(self)
Stop updating statistics.
source code
FilePath
getRootPath(self)
Returns: the root of the file repository
source code
 
getLogName(self, path, id=None)
Returns a log name for a path, shortened to a maximum size specified by the global variable MAX_LOGNAME_SIZE.
source code
 
getIdentifier(self, path)
Returns an identifier for a path.
source code
 
getCachePath(self, path) source code
 
getTempPath(self, path) source code
 
updateCacheUsageStatistics(self) source code
 
updateCacheUsage(self)
Returns: the cache usage, in bytes
source code
tuple
allocateCacheSpace(self, size)
Try to reserve cache space.
source code
 
releaseCacheSpace(self, tag) source code
 
getCopySession(self, path) source code
 
createCopySession(self, path, file, info) source code
 
outdateCopySession(self, path) source code
 
removeCopySession(self, session) source code
 
activateSession(self, session) source code
 
disableSession(self, session) source code
 
_activateCopyLoop(self) source code
 
_disableCopyLoop(self) source code

Inherited from plugs.base.ComponentPlug: restart

Inherited from extern.log.log.Loggable: __providedBy__, debug, doLog, error, info, log, logFunction, logObjectName, warning, warningFailure, writeMarker

Class Variables [hide private]
  logCategory = 'fileprovider-localcached'
Implementors can provide a category to log their messages under.

Inherited from extern.log.log.Loggable: __implemented__, __provides__

Method Details [hide private]

__init__(self, args)
(Constructor)

source code 
Parameters:
  • args - The plug args
Overrides: plugs.base.Plug.__init__
(inherited documentation)

start(self, component)

source code 
Overrides: plugs.base.ComponentPlug.start

stop(self, component)

source code 
Overrides: plugs.base.ComponentPlug.stop

startStatsUpdates(self, updater)

source code 

Start updating statistics.

Overrides: fileprovider.FileProviderPlug.startStatsUpdates
(inherited documentation)

stopStatsUpdates(self)

source code 

Stop updating statistics.

Overrides: fileprovider.FileProviderPlug.stopStatsUpdates
(inherited documentation)

getRootPath(self)

source code 
Returns: FilePath
the root of the file repository
Overrides: fileprovider.FileProviderPlug.getRootPath
(inherited documentation)

getLogName(self, path, id=None)

source code 

Returns a log name for a path, shortened to a maximum size specified by the global variable MAX_LOGNAME_SIZE. The log name will be the filename part of the path postfixed by the id in brackets if id is not None.

getIdentifier(self, path)

source code 

Returns an identifier for a path. The identifier is a digest of the path encoded in hex string. The hash function used is SHA1. It caches the identifiers in a dictionary indexed by path and with a maximum number of entry specified by the constant ID_CACHE_MAX_SIZE.

updateCacheUsage(self)

source code 
Returns:
the cache usage, in bytes

allocateCacheSpace(self, size)

source code 

Try to reserve cache space.

If there is not enough space and the cache cleanup is enabled, it will delete files from the cache starting with the ones with oldest access time until the cache usage drops below the fraction specified by the property cleanup-low-threshold.

Returns a 'tag' that should be used to 'free' the cache space using releaseCacheSpace. This tag is needed to better estimate the cache usage, if the cache usage has been updated since cache space has been allocated, freeing up the space should not change the cache usage estimation.

Parameters:
  • size (int) - size to reserve, in bytes
Returns: tuple
an allocation tag or None if the allocation failed.