PyTorch Binary Classification: Fixing the Loss vs. Metric Gap

Accuracy is a lie when you're dealing with imbalanced datasets in binary classification. I hit a wall recently where my model showed 92% accuracy during training, but the precision-recall curve looked like a disaster. The core issue is the fundamental difference between the objective function (what the optimizer minimizes) and the evaluation metrics (what we actually care about).

In PyTorch, we typically use BCEWithLogitsLoss for binary tasks. This is a smooth, differentiable function necessary for backpropagation. However, you can't use accuracy as a loss function because it's non-differentiable.

To properly diagnose model performance, I've moved to a setup that tracks these specific metrics per epoch:

• Log Loss (Cross-Entropy): The actual value the optimizer is attacking. If this drops but accuracy doesn't move, you're likely dealing with a severe class imbalance.
• Precision & Recall: Essential for understanding the trade-off between false positives and false negatives.
• F1-Score: The harmonic mean of the two, which is the only number I trust for "overall" performance on skewed data.
• AUC-ROC: Measures the model's ability to distinguish between classes regardless of the decision threshold.

For anyone building a custom training loop, don't rely on the raw loss. I use torchmetrics to handle the accumulation across batches to avoid the "average of averages" mistake.

Here is the basic logic for calculating the F1-score correctly within a validation loop:

import torch
from torchmetrics.classification import BinaryF1Score

# Initialize metric
f1_metric = BinaryF1Score().to(device)

model.eval()
with torch.no_grad():
    for inputs, targets in val_loader:
        inputs, targets = inputs.to(device), targets.to(device)
        logits = model(inputs)
        # Convert logits to binary predictions (0 or 1)
        preds = (torch.sigmoid(logits) > 0.5).float()
        f1_metric.update(preds, targets)

final_f1 = f1_metric.compute()
print(f"Validation F1 Score: {final_f1}")

If you see your loss decreasing while your F1-score plateaus or drops, check your class weights in the loss function. Adding pos_weight to BCEWithLogitsLoss usually fixes the bias toward the majority class.