I maintain a roster of the parameter values and SSEs of Individual objects that a Population has had and possibly replaced.

ParametersN_maxThe most records I can have in my roster. When the roster is full, adding a non-duplicative Individual will bump the highest-SSE one currently in the roster to make room. The default of 1500 seems like a sensible compromise between reasonably compact .dat file size and informative plots.
Instance Variable names A sequence of my individuals' parameter names, supplied as the sole constructor argument.
Instance Variable X A 2-D Numpy array of SSEs (first column) and parameter values (remaining columns) of one individual.
Instance Variable K A list of indices to rows of X, each entry in the list corresponding to a row of X.
Instance Variable Kp A set of the values (not indices) of K that are for individuals currently in the population.
Instance Variable Kn A set of the values (not indices) of K that are for individuals who never were in the population.
Instance Variable kr A dict containing row indices, keyed by the hashes of Individual instances.
Method __init__ History(names, N_max=None)
Method __getstate__ For storage-efficient pickling.
Method __setstate__ For unpickling.
Method shutdown Undocumented
Method __len__ My length is the number of records in my roster.
Method __getitem__ Access the SSE and parameter values corresponding to index k of my K list.
Method __iter__ I iterate over 1-D Numpy arrays of parameter values in ascending order of the SSEs they resulted in.
Method clear Call to have me return to a virginal state with no SSE+values combinations recorded or considered for removal, an empty population, and an N_total of zero.
Method value_vs_SSE Obtains a 1-D Numpy array of the SSEs of my individuals and matching 1-D Numpy arrays for each of the parameter values in names.
Method kkr Returns (1) the index k of my K list where the row index of the new record should appear in my X array, and (2) that row index kr.
Method add Adds the SSE and parameter values of the supplied individual i to my roster, unless it has an SSE of inf, in which case it is ignored.
Method purge Purges my history of the record at row index kr.
Method notInPop Call this with an integer row index or an Individual instance that was added via add to remove its row of my X array from being considered part of the current population.
Method purgePop Purges the history of all members of the current population. (Presumably, they will get added back again after re-evaluation.)
Method _initialize Undocumented
names =
A sequence of my individuals' parameter names, supplied as the sole constructor argument.
X =
A 2-D Numpy array of SSEs (first column) and parameter values (remaining columns) of one individual.
K =
A list of indices to rows of X, each entry in the list corresponding to a row of X.
Kp =
A set of the values (not indices) of K that are for individuals currently in the population.
Kn =
A set of the values (not indices) of K that are for individuals who never were in the population.
kr =
A dict containing row indices, keyed by the hashes of Individual instances.
def __init__(self, names, N_max=None):

History(names, N_max=None)

def __getstate__(self):

For storage-efficient pickling.

def __setstate__(self, state):

For unpickling.

def _initialize(self):
Undocumented
def shutdown(self):
Undocumented
def __len__(self):

My length is the number of records in my roster.

Note: Immediate result, not locked! Mostly for unit testing.

def __getitem__(self, k):

Access the SSE and parameter values corresponding to index k of my K list.

Note: Immediate result, not locked! Mostly for unit testing.

def __iter__(self):

I iterate over 1-D Numpy arrays of parameter values in ascending order of the SSEs they resulted in.

Note: Immediate result, not locked! Mostly for unit testing.

def clear(self):

Call to have me return to a virginal state with no SSE+values combinations recorded or considered for removal, an empty population, and an N_total of zero.

Returns a Deferred that fires when the lock has been acquired and everything is cleared.

def value_vs_SSE(self, *args, **kw):

Obtains a 1-D Numpy array of the SSEs of my individuals and matching 1-D Numpy arrays for each of the parameter values in names.

Waits to acquire the lock and then calls Analysis.value_vs_SSE on my instance a, returning a Deferred that fires with the eventual result.

def kkr(self, SSE, N):

Returns (1) the index k of my K list where the row index of the new record should appear in my X array, and (2) that row index kr.

First, index k is obtained, by seeing where the K list points to a record with an SSE closest but above the new one. Then each row index in the K list is examined to see if the previous row of my X array is unallocated. If so, that is the row index for the new record. Otherwise, is the next row of my X array is unallocated, that is used instead. If both adjacent rows of X are already allocated, the next row index in the K list is examined.

If there are no row indices in K that point to a row of X with an unallocated adjacent row, the row index is determined to be the current length of k.

Note: With the original for-loop Python, the search for an unallocated row was very CPU intensive when you get a big history accumulated::

   # Pick a row index for the new record
   for kr in self.K:
       if kr > 0 and kr-1 not in self.K:
           return k, kr-1
       if kr < N-1 and kr+1 not in self.K:
           return k, kr+1
   return k, N

The reason is that the list was being searched for an item with every iteration, twice!

The optimized version does this same thing with much more efficiently, by creating a local (array) copy of K and sorting it in place. Then only adjacent elements needs to be inspected with each iteration. (It may be just as fast with a local sorted list instead of a Numpy array.)

@defer.inlineCallbacks
def add(self, i, neverInPop=False):

Adds the SSE and parameter values of the supplied individual i to my roster, unless it has an SSE of inf, in which case it is ignored.

If the roster is already full, bumps the record deemed most expendable before adding a record for i. That determination is made by a call to my ClosestPairFinder instance cpf.

Returns a Deferred that fires with the row index of the new record when it has been written, or None if no record was written.

ParametersneverInPopSet True to have the individual added without ever having been part of the population.
def purge(self, kr):

Purges my history of the record at row index kr.

Removes the row index from my K list and has my cpf instance of ClosestPairFinder disregard the row, because it's now gone.

Note: Does not remove the index from the values of my kr dict, as that is a time-consuming process and the caller can likely just clear the whole thing anyhow.

def notInPop(self, x):

Call this with an integer row index or an Individual instance that was added via add to remove its row of my X array from being considered part of the current population.

def purgePop(self):

Purges the history of all members of the current population. (Presumably, they will get added back again after re-evaluation.)

API Documentation for ade, generated by pydoctor at 2021-09-18 08:41:05.