Data Engineering & Backend

Intelligent Game Discovery Engine

A crowdsourced recommendation platform featuring an automated ETL data pipeline, algorithmic suggestions, and high-performance infinite scrolling.

JavaSpring BootMySQLReactIGDB APIUnix Timestamp SyncFull-Text Search

The Problem: Analysis Paralysis

I wanted a way to discover new games based on what my friends were actually playing, without the friction of constantly asking them "What's in your top 5?"

Existing databases like IGN or Steam are overwhelming—they contain millions of shovelware titles and cluttered interfaces. I needed a curated, clean ecosystem that only surfaced quality titles and automated the recommendation process.

My Goals

  • Automate Discovery: Stop manually searching through huge databases.
  • Clean UX: Remove the noise. Only show relevant, popular games.
  • Algorithmic Curation: Use data to predict what I might like next.

Automated Data Pipeline (ETL)

Cron Job
1
Delta Sync
Get last_timestamp
Loop 500 items until empty
2
Transform
Parse JSON to Objects
Clean & Validate Data
3
Upsert
Save new Games to DB
Update Timestamp
User
Name Search
Genre Filter
Platform Filter
Curated DB

Timestamp-Based Delta Sync

To avoid fetching the entire database daily, I implemented an efficient state-tracking system using Unix Timestamps.

  • Backend stores the last_updated timestamp of the previous successful sync.
  • Scheduler requests only games created after this time in batches of 250.
  • The loop continues until the API returns an empty JSON array, ensuring total data consistency.

Prefetching & Infinite Scroll

To handle thousands of high-res covers without lag, I implemented a seamless pagination strategy.

IF (userScroll > 50% of pageHeight)
THEN fetch(nextPage) in background

This creates an illusion of an infinite list by loading the next set of 20 games into memory before the user ever reaches the bottom.

Engineered Features

MySQL Full-Text Index

I moved beyond basic SQL "LIKE" queries. By implementing a Full-Text Index, the search engine handles natural language queries, allowing users to find games even if they type words out of order.

Recommendation Algo

A custom weighting algorithm that analyzes selected genres and platforms to generate a "Compatibility Score," surfacing hidden gems that statistically match the user's taste.

Self-Healing Data

The system is designed to run autonomously. The scheduled cron job ensures the catalog is never stale, automatically ingesting the latest releases without manual admin input.

Return to Portfolio