Resolved

Stock metadata system duplicates camera sensor data across JSON files

TypeScript
Astro
JSON
Metadata
EXIF
Image Processing
Data Architecture
Frontend

Media metadata architecture with duplicated sensor specs in each JSON file caused maintenance issues | Refactored to use runtime sensor lookup with a centralized database

Issue

The stock metadata system faced three critical challenges that hindered maintenance and scalability:

  1. Redundant Sensor Data: Every JSON file contained identical camera sensor specifications (model, size, type, megapixels) for the same camera model, creating thousands of duplicate entries across files.

  2. Inconsistent Bit Depth Reporting: DJI cameras incorrectly reported 16-bit bit depth in EXIF data despite capturing at 14-bit (or 12-bit for telephoto lenses), leading to misleading metadata.

  3. Inconsistent Watermarking: Image previews included tiled watermarks, while video previews lacked watermarking entirely.

These issues made updates to sensor specifications time-consuming and error-prone.

Culprit

The architecture’s inefficiencies stemmed from:

  • Embedded Sensor Redundancy: Each JSON file contained full sensor specifications, resulting in ~30% file size bloat across thousands of files.

  • EXIF Inaccuracy: Cameras wrote conflicting metadata (e.g., 16-bit JPEG inside DNG containers) without proper corrections.

  • Pipeline Disparity: Image and video processing pipelines lacked alignment, leading to inconsistent watermarking behavior.

Solution

The refactor implemented a centralized data architecture with these key improvements:

Centralized Sensor Database

A single-source-of-truth sensor database (src/data/sensor-database.json) replaced duplicated sensor data, storing all camera specifications in one place. This enabled runtime enrichment of metadata without modifying individual files.

Bit Depth Correction

A custom bit depth override mechanism allowed explicit correction of incorrect EXIF values, ensuring accurate reporting in the final metadata.

Unified Watermarking Pipeline

A consolidated ImageMagick + ffmpeg pipeline was implemented to apply watermarks to both images and videos, ensuring consistency across all previews.

Structural Optimization

The directory structure was streamlined to separate original assets, previews, and metadata, reducing file size and improving maintainability:

  • src/data/sensor-database.json: Centralized sensor specifications
  • public/stock-previews/*.json: Minimal metadata with runtime sensor enrichment
  • /source-folder/: Original files + generated previews

Result: Reduced JSON file sizes by ~30% and eliminated the need to regenerate all metadata for sensor updates, while ensuring consistent watermarking across image and video assets.