EfficientNetB0 hit 89% accuracy for Pakistan Sign Language

An accuracy score of 95% is meaningless if your model is effectively cheating. I hit this wall while building an alphabet-recognition model for a PSL Flutter app; my initial high scores were a lie because near-duplicate images from the same recording sessions were leaking between my training and test sets. If a model sees a slightly blurred version of an image it has already memorized, you aren't testing generalization—you're testing memory.

To fix this, I rebuilt the entire pipeline to ensure the model was actually learning sign patterns rather than specific image artifacts. Here is the technical breakdown of how I moved from "fake" accuracy to a production-ready deployment.

Solving the Data Leakage Problem

The dataset consisted of 40 Urdu alphabet classes across roughly 22.8K images, sourced from photos, augmentations, and video frames. The danger here is that frames from the same 2-second clip are nearly identical. To stop the leakage, I implemented a group-based split using a group_id (a combination of label and source sample). This ensures that every image tracing back to the same physical sample stays together in one set.

def split_groups_within_label(group_df: pd.DataFrame, val_ratio: float = 0.15, test_ratio: float = 0.15):
 groups = sorted(group_df['group_id'].drop_duplicates().tolist())
 random.Random(SEED).shuffle(groups)
 total = len(groups)
 n_test = max(1, round(total * test_ratio)) if total >= 3 else 0
 n_val = max(1, round(total * val_ratio)) if total >= 3 else 0
 # ...assign whole groups to train/val/test, never split within a group

I followed three strict rules to maintain integrity:

• Video frames were extracted only after the split decision was made.
• Augmented images were only kept in the training set if their original source was also in training.
• No single group was ever split across train, validation, and test sets.

The Model Architecture and Training

For a mobile deployment, I needed a balance between speed and precision. I chose EfficientNetB0 due to its efficiency on edge devices. I used a two-stage transfer learning approach: first using the model as a frozen feature extractor, then unfrozing the top layers for fine-tuning.

base_model = tf.keras.applications.EfficientNetB0(
 include_top=False, weights='imagenet', input_shape=IMG_SIZE + (3,)
)
base_model.trainable = False

Performance Benchmarks

After the rigorous split, the "optimistic" numbers dropped, but the real-world reliability increased. Here are the final metrics on a genuinely unseen test set:

• Test Accuracy: 89.2%
• Balanced Accuracy: 93.0%
• Model Format: .tflite (optimized for Flutter)
• Latency: Low enough for real-time mobile inference

By focusing on per-class evaluation, I can now identify exactly which signs are underperforming and target those specifically with more data. This is a much more sustainable AI workflow than simply chasing a higher global accuracy percentage. The final model was exported to TFLite with accompanying label maps, making it ready for immediate integration into the mobile app.