Tuesday, August 27, 2019

Changelog 2019-08-27 - Two Fixes

Gamestop is back online

Gamestop changed their website two weeks ago and the scraper has been broken for as long. I finally figured out how to fix everything so the scraper is back online.

Moving over to the new version was not straight forward. First, none of the filters you apply on the website are reflected in the URL. I had to dig through the network requests to find the params that set various filters and manually add them to the URL. This enables us to filter only for Gamestop Exclusives and put a large number of items on the page at once so we don't need to scrape multiple times. Second, we needed to get the HTML without using basic get requests. Their server is configured to only respond to legit client requests, so browser spoofing was required to achieve success. However, this means we can drop the Incapsula breaker library as it appears their site no longer uses Incapsula to prevent access. Finally, we needed to filter sold out times. An item being sold out is not obvious when looking at the search results page. Fortunately, the availability is specified deep in the HTML of the page so we're able to parse that out to determine if something is in stock and reverse engineer a URL to post to various sites.

Changed to work with new Funko Shop

Previously, all Funko shop exclusives were listed at funko-shop.com. However, Funko has removed that site and merged it into shop.funko.com. Their weekly exclusives will now be sold on shop.funko.com. I have updated the scraper to pull the new items from there. This also has the benefit of pulling any restocks of exclusives that might happen. We are still scraping the store exclusives page as well on shop.funko.com.

1 comment:

Hot Cash Give Away Jan 2021

  Hey everyone! Time for another round of Hot Cash codes! Hot Cash ends soon so be sure to use these ASAP! Comment below with the codes you ...