This PR adds some functionality to aid in processing segmentation results for larger files.

• vectorize, unvectorize arrays to efficiently convert to/from [n-pixels, Time]
• functions to calculate number of neurons / patch given a neuron density [seems to way over-calculate]
• calculate number of patches
• --summary cli flag to give quick metrics for number of accepted / rejected / total, searches a path recursively to gather all pickled results files and summarize each
• --summary_plots boolean flag to generate a semi-performant picture of the spatial components
• --run_slurm python cli, same functionality as this mbo utilities cnmf_grid_search.sh
• HPC grid search
• Better defaults for stride/overlap/rf
• Summary --cnmf / --mcorr flags to control which summary you get
• In house mesmerize-core (requires new install steps
• params_from_metadata() very helpful
• add default gnb = 3
• validation / sanity checks for default ops and metadata
• rename everything ops -> params

You could theoretically run --summary ~/data/ will search through every directory inside that path, up to 3 subdirectories deep, and get every successful cnmf run to summarize.

5 batch runs spread across multiple batch runs (uuid column omitted):

This is very fast because we don't need to move our currently [n_pixels x time] formatted data into a format we can plot. (this is what the newly added vectorizer class/methods do).

--summary_plots now needs to reshape every one of the neurons.
This was measured at ~200 components/second. Don't use this flag for a "quick" summary, it's more useful to be used after runs on the cluster.

Nicely print params for all cnmf items recursively searched through the given directory:

Final summary.csv report:

HPC Grid Search
Each grid search outputs a single batch.pickle file. This is transferred locally in a directory with $SLURM_JOB_ID:

Summary of Accepted/Rejected Components

This is just a reshaped version of the spatial footprints for accepted/rejected neurons

Using max of spatial footprint

Taking the maximum value of each component before any reshaping has been the most computationally performant way to show users contours. We can use idx_components to modify the colors.

Displaying the maximum value is instant, locating the center coordinate can be parallelized. With a 32 core CPU this takes a minute per ~40,000 contours:

from joblib import Parallel, delayed
import numpy as np
from tqdm import tqdm

def _calculate_centers(A, dims):
    def calculate_center(i):
        ixs = np.where(A[:, i].toarray() > 0.07)[0]
        return np.array(np.unravel_index(ixs, dims)).mean(axis=1)[::-1]

    # Use joblib to parallelize the center calculation for each column in A
    centers = Parallel(n_jobs=-1)(delayed(calculate_center)(i) for i in tqdm(range(A.shape[1]), desc="Calculating neuron center coordinates"))

    return np.array(centers)

Calculating neuron center coordinates: 100%|██████████| 38100/38100 [00:50<00:00, 749.90it/s]