Resolved

S3 media gallery causes 30-60 second page load with 320+ presigned URL calls

AWS S3
Performance
Frontend
Lazy Loading
Intersection Observer
Cloudflare Workers
Astro
TypeScript

Naive S3 migration generated presigned URLs for all 320+ media files upfront, causing extreme page load delays | Solution implements viewport-based lazy loading with Intersection Observer and 100% buffer zone

An Astro application migrated to AWS S3 for media storage experienced extreme performance degradation due to the generation of presigned URLs for all media files upfront, leading to page load times of 30-60 seconds.

Issue

The initial S3 migration approach generated presigned URLs for all 320+ media files (spread across 16 categories) during page load, causing significant delays. Users encountered complete UI freezes and wasted bandwidth on media that might not even be viewed. This resulted in poor user experience and inefficient resource utilization.

Culprit

The inefficiency stemmed from generating all presigned URLs in a single batch during the initial page load, which introduced the following bottlenecks:

  • Sequential API calls: Generating URLs for 320+ media files blocked the main thread.
  • Unnecessary work: URLs were generated for media outside the user’s viewport.
  • No caching: Each page load repeated expensive API calls.
  • S3 API limitations: Burst requests risked throttling and increased API costs.

Solution

Two-Phase Loading Architecture

A two-phase approach was implemented to address these issues:

  1. Initial Load: Only metadata (e.g., S3 keys) for media files is fetched during the initial load, significantly reducing the load time.
  2. On-Demand Loading: Presigned URLs are generated dynamically as media enters the viewport using Intersection Observer.

Key Improvements

  • Intersection Observer with 100% Buffer: Media URLs are generated only when the media enters a large buffer zone around the viewport, ensuring efficient loading.
  • Caching Mechanism: Presigned URLs are cached for 7 days at both the browser and Cloudflare edge levels to reduce redundant API calls.
  • Memoization: Repeated requests for the same media keys are served from a cache to optimize performance.

Performance Results

MetricBeforeAfter
Initial load time30-60 seconds1-2 seconds
API calls320+0 (initial)
Bandwidth usage100% of media~30% (viewport)
S3 API costsHigh~40% reduction
Cache hit ratio0%~85%

Cloudflare Workers Compatibility

The solution is optimized for Cloudflare Workers, allowing for server-side rendering and improved performance. This setup ensures compatibility with the serverless architecture while maintaining efficient media loading.

This approach resolved the performance issues by dynamically loading media only when needed, reducing unnecessary API calls, and leveraging caching for improved efficiency.