Directional Metrics

Directional metrics are used to evaluate the performance of forecasting models, especially in time-series and financial contexts where the trend direction is often more important than the exact magnitude of the change.

Directional Accuracy Score

ds_utils.metrics.time_series.directional_accuracy_score(y_true: ndarray, y_pred: ndarray, baseline: ndarray | None = None, sample_weight: ndarray | None = None, handle_equal: str = 'exclude') → float[source]

Calculate the directional accuracy score.

Directional accuracy (DA) measures the proportion of time steps for which the predicted direction of change matches the true direction of change, relative to a baseline.

The formula is: DA = (1/n) * Σ I(sign(y_true - baseline) == sign(y_pred - baseline))

Parameters:

  • y_true – Ground truth (correct) target values.

  • y_pred – Estimated target values.

  • baseline – Baseline values to compare against. If None, uses the previous value of y_true (time-series mode). If provided, must have the same shape as y_true.

  • sample_weight – Sample weights.

  • handle_equal – How to treat samples where y_true == baseline. - ‘exclude’: Filter out these samples (default). - ‘correct’: Count as correct if y_pred == baseline, else incorrect. - ‘incorrect’: Always count as incorrect.

Returns:

Directional accuracy score as a float.

Raises:

ValueError – If handle_equal is invalid, if shapes mismatch, or if insufficient samples are provided for time-series mode.

Time series example: >>> import numpy as np >>> y_true = np.array([100, 102, 98, 101, 99]) >>> y_pred = np.array([100.5, 103, 97, 102, 98]) >>> directional_accuracy_score(y_true, y_pred) 1.0

Custom baseline example: >>> y_true = np.array([102, 98, 101, 99, 102]) >>> baseline = np.array([100, 100, 100, 100, 100]) >>> y_pred = np.array([101, 99, 99, 101, 99]) >>> directional_accuracy_score(y_true, y_pred, baseline=baseline) 0.4

Code Example

from ds_utils.metrics.time_series import directional_accuracy_score
import numpy as np

# Directional accuracy — time series mode (uses previous value as baseline)
# true changes from baseline: +2, -4, +3, -2
# pred changes from baseline: +3, -5, +4, -3  -> all directions match
y_true = np.array([100, 102, 98, 101, 99])
y_pred = np.array([101, 103, 97, 102, 98])
da = directional_accuracy_score(y_true, y_pred)
print(f"Directional Accuracy: {da:.2%}")

Output:
Directional Accuracy: 100.00%

Directional Bias Score

ds_utils.metrics.time_series.directional_bias_score(y_true: ndarray, y_pred: ndarray, sample_weight: ndarray | None = None, handle_equal: str = 'exclude') → float[source]

Calculate the directional bias score.

Directional bias (DB) measures the systematic tendency of a model to over-predict or under-predict the target values.

The formula is: DB = (n_over - n_under) / n

Parameters:

  • y_true – Ground truth (correct) target values.

  • y_pred – Estimated target values.

  • sample_weight – Sample weights.

  • handle_equal – How to treat samples where y_pred == y_true. - ‘exclude’: Filter out these samples (default). - ‘neutral’: Keep samples; they contribute 0 to the bias.

Returns:

Directional bias score as a float in [-1, 1]. 1.0 = complete over-prediction, -1.0 = complete under-prediction, 0.0 = balanced predictions.

Raises:

ValueError – If handle_equal is invalid, if shapes mismatch, or if no samples remain after filtering.

Complete over-prediction example: >>> import numpy as np >>> y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0]) >>> y_pred = np.array([1.1, 2.1, 3.1, 4.1, 5.1]) >>> directional_bias_score(y_true, y_pred) 1.0

Complete under-prediction example: >>> y_pred = np.array([0.9, 1.9, 2.9, 3.9, 4.9]) >>> directional_bias_score(y_true, y_pred) -1.0

Balanced example: >>> y_pred = np.array([1.1, 1.9, 3.1, 3.9, 5.0]) >>> directional_bias_score(y_true, y_pred) 0.0

Code Example

from ds_utils.metrics.time_series import directional_bias_score
import numpy as np

# Directional bias — detect systematic over/under-prediction
# All predictions are 0.1 above the true value
y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
y_pred = np.array([1.1, 2.1, 3.1, 4.1, 5.1])
bias = directional_bias_score(y_true, y_pred)
print(f"Directional Bias: {bias:.2f}")

Output:
Directional Bias: 1.00