Python dependency conflicts and outdated package versions causing build instability

Migrated a Python-based trading system from pip/requirements.txt to Poetry to resolve dependency conflicts and ensure reproducible builds. The update addressed version inconsistencies, missing transitive dependencies, and Python compatibility issues.

Issue

The project experienced unpredictable builds due to:

• Version conflicts between PyTorch and dependent packages when running pip install --upgrade.
• Non-reproducible environments where pip install -r requirements.txt pulled inconsistent minor/patch versions across installations.
• Missing transitive dependencies causing runtime ModuleNotFoundError exceptions.
• Python 3.14 compatibility issues with older package versions (e.g., scikit-learn and numpy).

Key error patterns included:

• Conflicting dependencies between PyTorch 2.9.0 and older versions.
• Runtime errors when datasets==4.2.0 was installed instead of 4.3.0.

Culprit

• Unpinned Requirements: requirements.txt used loose version specifiers (e.g., transformers>=4.0.0), allowing pip to install incompatible minor/patch versions.
• Missing Dependency Tree: Pip’s flat dependency resolution failed to track transitive dependencies or enforce consistent versions.
• Virtualenv Management: Manual virtualenv handling led to inconsistent Python versions and accidental system-wide installations.

Solution

• Migrated to Poetry: Replaced requirements.txt with pyproject.toml and introduced poetry.lock for deterministic dependency resolution.
• Pinned All Dependencies: All packages were pinned to stable versions to ensure compatibility (e.g., PyTorch 2.9.0, Transformers 4.57.1, Datasets 4.3.0).
• Automated Virtualenv Management: Poetry created and managed an in-project virtualenv (.venv/) with Python 3.14.
• Backward Compatibility: Retained an auto-generated requirements.txt for legacy systems and documented migration steps.

Verification confirmed no version conflicts and reproducible builds across environments.