a.t.OrderedItemProducer(PoolUser) : class documentation

Part of asynqueue.threads View In Hierarchy Source for threads.py Main doc for AsynQueue

Implements interfaces: twisted.internet.interfaces.IPushProducer

Produces blocking iterations in the order they are requested via an asynchronous function call.

I am an implementor of Twisted's IPushProducer interface that produces an iteration to a blocking call fb for every time you call a non-blocking item-generating function fb via my produceItem method. Significantly, the items are buffered as needed so that the iterations appear in the order of the calls to produceItem that generated them, not necessarily in the order in which they are actually generated.

Start things off by constructing an instance of me, with an existing task queue if you have one you want me to use, and then running start with your blocking f-args-kw combination. Then call produceItem repeatedly with whatever f-args-kw combination results (eventually) in new items to iterate. These calls may return deferred results and should not block.

When you are done having me produce iterations, call stop. Whatever loop the blocking-iterator call is in will then terminate and function fb should end.

Instance Variable i My Consumerator instance, which acts like an iterator for whatever function you supply to start.
Instance Variable q The TaskQueue instance I use, either supplied by you during construction or instantiated by me. Either way, you will have to call TaskQueue.shutdown on this eventually when you're done with the queue.
Method __init__ Undocumented
Method start No summary
Method produceItem Runs fp(*args, **kw) to generate an item that I produce as an iteration to whatever blocking call was (or will be) set running via start.
Method stop No summary
Method stopProducing Undocumented
Method pauseProducing Undocumented
Method resumeProducing Undocumented
Method _writeItem Undocumented
Method _flushBuffer Undocumented

Inherited from PoolUser:

Class Method setup No summary
Class Method shutdown Undocumented
Class Method deferToThreadInPool Runs the f-args-kw call combo in one of my threads, returning a Deferred that fires with the eventual result. Can be run from the class or any instance of me with the exact same result.
Method pool Returns a reference to the class-wide threadpool, starting it if this is the first time it's been used.
i =
My Consumerator instance, which acts like an iterator for whatever function you supply to start.
q =
The TaskQueue instance I use, either supplied by you during construction or instantiated by me. Either way, you will have to call TaskQueue.shutdown on this eventually when you're done with the queue.
def __init__(self, maxThreads=None):
Undocumented
def start(self, fb, *args, **kw):

Starts the blocking function call fb(i, *args, **kw) that relies on my Consumerator instance i for iterations, in traditional blocking fashion. The function must accept i as its first argument, and can also accept further arguments *args and keywords **kw, which you can specify in your call to start.

Returns A Deferred that fires when the blocking call has started in its own thread. Shouldn't take long at all, unless the pool is fully occupied and we need to wait for a thread to get freed up.
def produceItem(self, fp, *args, **kw):

Runs fp(*args, **kw) to generate an item that I produce as an iteration to whatever blocking call was (or will be) set running via start.

While I am running, the returned Deferred fires after the call to fp with the item value produced by the call to f. You don't need to do anything with these deferreds if you don't want to use them for concurrency limiting; they are accounted for in stop.

If an exception is raised during the call, the dFinished callback is called with a corresponding Failure object and iterations will be stopped. This makes more sense than firing an errback of the Deferred returning from this produceItem method, because it's the end result of calling stop to indicate that iterations are done. *That* is when the user should expect a status of the overall item-producing operation.

If my stopProducing method has been called, I no longer produce iterations and calls to this method do not run fp. The returned Deferred fires immediately with None.

@defer.inlineCallbacks
def stop(self, failureObj=None):

Call this to indicate that iterations are done. After any pending calls from produceItem are done, my Consumerator will raise StopIteration for the blocking iteration-caller in fb and that function should exit. Whatever value it returns will fire the Deferred that is returned here.

This method's Deferred may have fired already, if fb exited early for some reason, or with a Failure object that may have been generated either by the iteration caller or by a call to the fp function of produceItem.

Repeated calls to this method make no sense and will be rewarded with deferreds immediately firing with None.

def _writeItem(self, item):
Undocumented
def _flushBuffer(self):
Undocumented
def stopProducing(self):
Undocumented
def pauseProducing(self):
Undocumented
def resumeProducing(self):
Undocumented
API Documentation for AsynQueue, generated by pydoctor at 2016-11-16 14:52:11.