Implements interfaces: twisted.internet.interfaces.IPushProducer

Produces blocking iterations in the order they are requested via an asynchronous function call.

I am an implementor of Twisted's IPushProducer interface that produces an iteration to a blocking call fb for every time you call a non-blocking item-generating function fb via my produceItem method. Significantly, the items are buffered as needed so that the iterations appear in the order of the calls to produceItem that generated them, not necessarily in the order in which they are actually generated.

Start things off by constructing an instance of me, with an existing task queue if you have one you want me to use, and then running start with your blocking f-args-kw combination. Then call produceItem repeatedly with whatever f-args-kw combination results (eventually) in new items to iterate. These calls may return deferred results and should not block.

When you are done having me produce iterations, call stop. Whatever loop the blocking-iterator call is in will then terminate and function fb should end.

Instance Variable	i	My `Consumerator` instance, which acts like an iterator for whatever function you supply to `start`.
Instance Variable	q	The `TaskQueue` instance I use, either supplied by you during construction or instantiated by me. Either way, you will have to call `TaskQueue.shutdown` on this eventually when you're done with the queue.
Method	__init__	Undocumented
Method	start	No summary
Method	produceItem	Runs `fp(args, *kw)` to generate an item that I produce as an iteration to whatever blocking call was (or will be) set running via `start`.
Method	stop	No summary
Method	stopProducing	Undocumented
Method	pauseProducing	Undocumented
Method	resumeProducing	Undocumented
Method	_writeItem	Undocumented
Method	_flushBuffer	Undocumented

Inherited from PoolUser:

Class Method	setup	Sets up stuff class-wide, with all the potential pitfalls that entails.
Class Method	shutdown	Shuts down all threads, returning a `Deferred` that fires when everything's done, class-wide.
Class Method	deferToThreadInPool	Runs the `f-args-kw` call combo in one of my threads, returning a `Deferred` that fires with the eventual result. Can be run from the class or any instance of me with the exact same result.
Method	pool	Returns a reference to the class-wide threadpool, starting it if this is the first time it's been used.

i =

My Consumerator instance, which acts like an iterator for whatever function you supply to start.

q =

The TaskQueue instance I use, either supplied by you during construction or instantiated by me. Either way, you will have to call TaskQueue.shutdown on this eventually when you're done with the queue.

def __init__(self, maxThreads=None):

Undocumented

def start(self, fb, *args, **kw):

Starts the blocking function call fb(i, *args, **kw) that relies on my Consumerator instance i for iterations, in traditional blocking fashion. The function must accept i as its first argument, and can also accept further arguments *args and keywords **kw, which you can specify in your call to start.

Returns A Deferred that fires when the blocking call has started in its own thread. Shouldn't take long at all, unless the pool is fully occupied and we need to wait for a thread to get freed up.

def produceItem(self, fp, *args, **kw):

Runs fp(*args, **kw) to generate an item that I produce as an iteration to whatever blocking call was (or will be) set running via start.

While I am running, the returned Deferred fires after the call to fp with the item value produced by the call to f. You don't need to do anything with these deferreds if you don't want to use them for concurrency limiting; they are accounted for in stop.

If an exception is raised during the call, the dFinished callback is called with a corresponding Failure object and iterations will be stopped. This makes more sense than firing an errback of the Deferred returning from this produceItem method, because it's the end result of calling stop to indicate that iterations are done. *That* is when the user should expect a status of the overall item-producing operation.

If my stopProducing method has been called, I no longer produce iterations and calls to this method do not run fp. The returned Deferred fires immediately with None.

@defer.inlineCallbacks
def stop(self, failureObj=None):

Call this to indicate that iterations are done. After any pending calls from produceItem are done, my Consumerator will raise StopIteration for the blocking iteration-caller in fb and that function should exit. Whatever value it returns will fire the Deferred that is returned here.

This method's Deferred may have fired already, if fb exited early for some reason, or with a Failure object that may have been generated either by the iteration caller or by a call to the fp function of produceItem.

Repeated calls to this method make no sense and will be rewarded with deferreds immediately firing with None.

def _writeItem(self, item):

Undocumented

def _flushBuffer(self):

Undocumented

def stopProducing(self):

Undocumented

def pauseProducing(self):

Undocumented

def resumeProducing(self):

Undocumented

API Documentation for AsynQueue, generated by pydoctor at 2022-11-17 13:13:24.

asynqueue.threads.OrderedItemProducer(PoolUser) class documentation

`asynqueue.threads.OrderedItemProducer(PoolUser)` class documentation