cross_val_score

mvpy.crossvalidation.cross_val_score(model: Pipeline | BaseEstimator, X: ndarray | Tensor, y: ndarray | Tensor | None = None, cv: int | Any | None = 5, metric: Metric | Tuple[Metric] | None = None, return_validator: bool = True, n_jobs: int | None = None, verbose: int | bool = False) → ndarray | Tensor | Dict | Tuple[Validator, ndarray | Tensor | Dict]

Implements a shorthand for automated cross-validation scoring over estimators or pipelines.

This function acts as a shorthand for Validator where it will automatically create and fit the validator, returning either only its output scores or, if return_validator is True, both the fitted validator object and the scores in a tuple.

For more information, please see Validator.

Parameters:

modelsklearn.pipeline.Pipeline | sklearn.base.BaseEstimator

The model to fit and score. Can be either a pipeline or estimator object.

Xnp.ndarray | torch.Tensor

The input data of arbitrary shape.

yOptional[np.ndarray | torch.Tensor], default=None

The outcome data of arbitrary shape.

cvOptional[int | Any], default=5

The cross-validation procedure to follow. Either an object exposing a split() method, such as KFold or an integer specifying the number of folds to use in KFold.

metricOptional[mvpy.metrics.Metric, Tuple[mvpy.metrics.Metric]], default=None

The metric to use for scoring. If None, this will default to the score() method exposed by model.

return_validatorbool, default=True

Should the underlying validator object be returned?

n_jobsOptional[int], default=None

How many jobs should be used to parallelise the cross-validation procedure?

verboseint | bool, default=False

Should progress be reported verbosely?

Returns:

validatorOptional[mvpy.crossvalidation.Validator]

If return_validator is True, the underlying validator object.

scorenp.ndarray | torch.Tensor | Dict[str, np.ndarray] | Dict[str, torch.Tensor]

The scores from cross-validation of arbitrary output shape.

Warning

If multiple values are supplied for metric, this function will output a dictionary of {Metric.name: score, ...} rather than a stacked array. This is to provide consistency across cases where metrics may or may not differ in their output shapes.

Warning

When specifying n_jobs here, be careful not to specify any number of jobs in the model. Otherwise, this will lead to a situation where individual jobs each try to initialise more low-level jobs, severely hurting performance.

See also

mvpy.crossvalidation.Validator

The underlying Validator class.

Examples

>>> import torch
>>> from mvpy.estimators import ReceptiveField
>>> from mvpy.crossvalidation import cross_val_score
>>> ß = torch.tensor([1., 2., 3., 2., 1.])
>>> X = torch.normal(0, 1, (100, 1, 50))
>>> y = torch.nn.functional.conv1d(X, ß[None,None,:], padding = 'same')
>>> y = y + torch.normal(0, 1, y.shape)
>>> trf = ReceptiveField(-2, 2, 1, alpha = 1e-5)
>>> validator, scores = cross_val_score(trf, X, y)
>>> scores.mean()
tensor(0.9432)