Resolved

AdamW import error and dataset loading failures in PyTorch training script

Python
PyTorch
AI/ML
Hugging Face
Transformers
Dataset
Debugging
Migration

Fixed AdamW optimizer import from transformers to torch.optim and resolved Python 3.14 dataset compatibility issues | Implemented robust fallback mechanisms for financial news datasets in Hugging Face ecosystem

Resolved critical errors in a PyTorch training script for financial sentiment analysis, including an import error for the AdamW optimizer and dataset compatibility issues with Python 3.14.

Issue

The training script failed due to:

  • ImportError: AdamW optimizer could not be imported from transformers due to API changes.
  • DatasetNotFoundError: Financial news datasets failed to load due to deprecated parameters, Python 3.14 incompatibility, and missing datasets on Hugging Face Hub.

Culprit

  • Deprecated API: The AdamW optimizer was moved from transformers to torch.optim in newer library versions.
  • Dataset Issues: Problems included deprecated trust_remote_code parameters, Python 3.14 serialization errors, and missing datasets.
  • Legacy Scripts: Many datasets relied on outdated loading methods incompatible with modern library versions.

Solution

  • Fixed AdamW Import: Updated the import statement to use PyTorch’s native torch.optim.AdamW.
  • Multi-Dataset Fallback: Implemented a priority-based system to load multiple dataset alternatives (e.g., financial_phrasebank, twitter-financial-news-sentiment).
  • Removed Deprecated Parameters: Eliminated trust_remote_code and adopted config-aware loading for better compatibility.
  • Synthetic Data Fallback: Added automatic generation of synthetic data when all loading attempts fail.
  • Python 3.14 Compatibility: Prioritized Parquet/Arrow formats and added explicit error handling for serialization issues.

The solution ensures backward compatibility and robust dataset loading while maintaining support for modern Python versions.