Tuesday, August 27, 2019

Changelog 2019-08-27 - Two Fixes

Gamestop is back online

Gamestop changed their website two weeks ago and the scraper has been broken for as long. I finally figured out how to fix everything so the scraper is back online.

Moving over to the new version was not straight forward. First, none of the filters you apply on the website are reflected in the URL. I had to dig through the network requests to find the params that set various filters and manually add them to the URL. This enables us to filter only for Gamestop Exclusives and put a large number of items on the page at once so we don't need to scrape multiple times. Second, we needed to get the HTML without using basic get requests. Their server is configured to only respond to legit client requests, so browser spoofing was required to achieve success. However, this means we can drop the Incapsula breaker library as it appears their site no longer uses Incapsula to prevent access. Finally, we needed to filter sold out times. An item being sold out is not obvious when looking at the search results page. Fortunately, the availability is specified deep in the HTML of the page so we're able to parse that out to determine if something is in stock and reverse engineer a URL to post to various sites.

Changed to work with new Funko Shop

Previously, all Funko shop exclusives were listed at funko-shop.com. However, Funko has removed that site and merged it into shop.funko.com. Their weekly exclusives will now be sold on shop.funko.com. I have updated the scraper to pull the new items from there. This also has the benefit of pulling any restocks of exclusives that might happen. We are still scraping the store exclusives page as well on shop.funko.com.

Thursday, August 15, 2019

Changelog 2019-08-13 - One Fix

Preorder posts now group by store name

Preorder posts made on Reddit will now sort every time they are updated. For example, if store A lists an item, then store B, then A again, instead of reading:

* Store A - Item 1
* Store B - Item 1
* Store A - Item 2

It will read:

* Store A - Item 1
* Store A - Item 2
* Store B - Item 1

This will make browsing easier. Stores are grouped by priority, then alphabetically with national retailers having a higher priority than small run businesses as they tend to be the more popular choices. 

Tuesday, August 13, 2019

Changelog 2019-08-13 - Two Fixes

Walgreens is now more reliable

Walgreens bot now checks each specific item page before posting about it. Sometimes, an item will have the "Add to cart" option on the search page, but the individual page will say sold out. This extra step accounts for the individual pages and reduces false positives.

Fixed Walgreens URL formatting

Walgreens URLs are case sensitive. Made a small fix to ensure that the parts that need to be capitalized stay capitalized.

Tuesday, August 6, 2019

Changelog 2019-08-06 - Two Fixes

Hot Topic now checks exact stock counts

Thanks to u/ on reddit, it was brought to my attention that Hot Topic is once again letting exact inventory numbers come through in the HTML source of their web page. With this, we can accurately detect if something is truly in stock or not, and exactly how many are in stock, before posting.

Re-Added Hot Topic to the InvCounter Chrome Extensions

Due to the above change, I was able to make modifications to the InvCounter Chrome Extension, enabling it to work again on Hot Topic's website. Work is still in progress for Box Lunch.

Hot Cash Give Away Jan 2021

  Hey everyone! Time for another round of Hot Cash codes! Hot Cash ends soon so be sure to use these ASAP! Comment below with the codes you ...