LeaveGroupsOut#

class mvpy.crossvalidation.LeaveGroupsOut(n_splits: int = 5, n_repeats: int = 1, random_state: int | Generator | Generator | None = None)[source]#

Implements a leave-groups-out cross-validator.

Functionally, this class is analogous to using RepeatedKFold over unique values of the supplied groups. For example, for inputs \(X\) of shape (100, ...) and groups \(G\) of shape (100, 1) where \(G\in {1...5}\) with \(k = 5\), each group in \(G\) corresponds to one fold \(k\).

However, LeaveGroupsOut also supports multi-label situations. For example, consider a gram matrix \(A(X, X^T)\in\mathcal{R}^{n\times n}\) where \(A_{i,j}\) is the neural similarity between participants \(i\) and \(j\) that we wish to model as a function of a gram matrix \(B(y, y^T)\in\mathcal{R}^{n\times n}\) where \(B_{i,j}\) describes behavioural similarity of participants \(i\) and \(j\). Typically, we would like to model \(u(A) = \beta u(B) + \varepsilon\) where \(u\) simply defines the upper triangle of the matrix. In cross-validation, however, this would lead to leakage because of the dyadic relationship present in each sample \(A_{i,j}\). Consequently, we must treat participants as grouping variables such that train and test sets are constructed over participants rather than samples. In this case, we would construct \(G\in\mathcal{R}^{n\times n\times 2}\) where \(G_{i,j} = (i, j)\) to ensure that we train on a subset of participants and test on a separate subset of participants.

Warning

If multiple labels per sample are present in LeaveGroupsOut’s group parameter such that (n_samples, ..., n_groups) where n_groups > 1, make sure that data are roughly balanced. Otherwise, fold sizes may vary greatly.

Parameters:

n_splitsint, default=5: Number of splits to use.
n_repeatsint, default=1: Number of repeats to perform.
random_stateOptional[Union[int, np.random._generator.Generator, torch._C.Generator]], default=None: Random state to use for shuffling (either integer seed or numpy/torch generator), if any.

Attributes:

n_splitsint, default=5: Number of splits to use.
n_repeatsint, default=1: Number of repeats to perform.
random_stateOptional[Union[int, np.random._generator.Generator, torch._C.Generator]], default=None: Random state to use for shuffling (either integer seed or numpy/torch generator), if any.
rkf_RepeatedKFold: Repeated k-fold cross-validation object used under the hood.

Examples

>>> import torch
>>> from mvpy.crossvalidation import LeaveGroupsOut
>>> X = torch.arange(6)
>>> g = torch.arange(2).repeat(3)
>>> kf = LeaveGroupsOut(n_splits = 2, n_repeats = 1)
>>> for f_i, (train, test) in enumerate(kf.split(X, groups = g)):
>>>     print(f'Fold{f_i}: train={train}|{g[train]}    test={test}|{g[test]}')
Fold0: train=tensor([1, 3, 5])|tensor([1, 1, 1])    test=tensor([0, 2, 4])|tensor([0, 0, 0])
Fold1: train=tensor([0, 2, 4])|tensor([0, 0, 0])    test=tensor([1, 3, 5])|tensor([1, 1, 1])

Split the dataset into iterable (train, test).

Parameters:

XUnion[np.ndarray, torch.Tensor]: Input data of shape (n_samples, …)
ynp.ndarray | torch.Tensor | None, default=None: Target data of shape (n_samples, …). Unused, but parameter available for consistency.
groupsnp.ndarray | torch.Tensor | None, default=None: Group labels of data. Labels are of shape (n_samples, …, n_labels). One sample may have multiple labels.

Returns:

kfUnion[collections.abc.Generator[tuple[np.ndarray, np.ndarray], None, None], collections.abc.Generator[tuple[torch.Tensor, torch.Tensor], None, None]]: Iterable generator of (train, test) pairs.