Friday, June 28, 2019

Changelog 2019-06-28 - One Fix

shop.funko.com is now tracked under @FunkoShopAlert

Funko started releasing store exclusives on their mainwebsite. These will now be tracked on the @FunkoShopAlert twitter account. 

Friday, June 21, 2019

Changelog 2019-06-22 - Two Fixes

Both fixes relate to the Reddit SwapBot not storing swaps that are archived.

For context, the bot maintains a list of "archived" comments aka comments that are unconfirmed and older than 3 days. It moves these comments to the archive list where it checks them once a day for the next 27 days before finally dumping them. However, there was an issue where the archive run was taking so long to run that a normal run would happen while the archive run was still happening but finish after the archive run, thus overwriting and changed to the database that the archive run made. This was fixed with two tricks...

Archive runs block out ten minutes of time.

Archive runs only less than 5 minutes to run, but if we try to do a normal run any time less than 10 minutes after the start of an archive run, the program will return right away. This prevents us from having conflicts while running.

Tags that are replies to tags will be ignored.

The real reason this was an issue though is because some of the folks on r/pkmntcgtrades like to tag the bot while replying to a tag. For example, user A tags user B and the Bot. User B then replies to user A by tagging user A and the Bot. This looks like a brand new attempt to confirm a trade in the eyes of the Bot, so it adds it to its list. Of course, this never gets "confirmed" because, as far as A and B know, they already confirmed the trade. So the Bot wastes resources checking B's comment that will never be confirmed (and even if it was confirmed, it would be thrown out anyway due to deduplication efforts). So now the bot will ignore these comments entirely.

Wednesday, June 19, 2019

Changelog 2019-06-19 - One Fix

Old blog posts will no longer be alerted upon

Due to the change documented in this blog post, I had to edit the way the database stored blog posts that have already been seen. Given this format change, I was not able to easily backfill the database with all of the old blog posts I had already alerted upon. I was content to assume that old blog posts would not make it to the front page of https://funko.com/blogs/news too often and any old ones that come up could be swiftly deleted.

However, recently, about a dozen old posts came up and were alerted upon. This, of course, makes for a poor user experience. I decided to find a way to truly backfill the database. After some thinking, I was able to figure out how to backfill with the data I had and added 802 old entries to the new database.

Moving forward, we should no longer see old blog posts being alerted upon as new blog posts. It is possible that one or two slip through, but for the most part, this should never happen again.

Friday, June 14, 2019

Changelog 2019-06-14 - Two Fixes

@FunkoFYE is more reliable now

FYE's site isn't great and their search functionality isn't fantastic either. There is a button to click that should give you all of their exclusives. However, it throws in a bunch of their commons as well in the search results and returns way too many items to reliably scrape.

The way around this is to use that search page and click *another* hidden button to filter down to just exclusives. However, there is no visible URL change so a URL for scraping had to be determined by looking at the network requests.

The RegEx had to also be updated to accommodate for these new pages. However, the name of the product as well as the URL should be a lot cleaner. Previously, we were posting products that had a lot of garbage in the product name as it was derived from the URL. Now, the item name is derived from the page itself so it should look a lot cleaner going forward.

Preorder links will more reliably be appended to their respective posts

With the recent update I made allowing newly release products to have their links automatically appended to preorder threads made on reddit, I made a decision to say "if there are any preorder threads made in the past two hours, only look at those. Otherwise, look at all threads made in the past 24 hours." This worked well up until today when there were quite a few releases.

Normally, if only one announcement is made in a day, it doesn't matter if it has been 2 or 22 hours, the script will still find the correct preorder post. But, if something is announced, then something else is announced, and the first announcement's product is released more than 2 hours from the first announcement but less than 2 hours after the second announcement, we won't append to the correct preorder thread.

To fix this, I have decided to eliminate all fancy checking and guessing. Instead, all preorder threads made in the past 24 hours are candidates. The script will then find whichever it most closely matches from that pool.

Wednesday, June 12, 2019

Changelog 2019-06-12 - Two Fixes

@FunkoGS added "Limited Edition" added as a keyword

Gamestop traditionally uses the phrases "exclusive" and "Only at Gamestop" as a way to describe their exclusive items which is what @FunkoGS wants to alert on. However, the recent E3 exclusive pops used the phrase "limited edition" which my scrapers were not prepared for. A quick change to the code by adding a new URL to watch fixes this problem and ensures that all future E3 drops, as well as restocks, will be caught by the scraper and posted to @FunkoGS and r/FunkoPop

Hot Topic and Box Lunch alert on entire inventory again

Recent changes to Hot Topic's and Box Lunch's websites made it such that URL parameters could not be set by the user. Previously, I could set "sz=1000" in the url and it would return every item Hot Topic or Box Lunch has to offer. With the change, this just caused it to default to 120 items which is about 1/7 of their entire inventory, so not everything was being alerted on. The fix was to add multiple URLs to parse rather than one URL with every item on the page, so all Hot Topic and Box Lunch items should be properly alerted on going forward.

Friday, June 7, 2019

Changelog 2019-06-07 - One Fix

PreoOrder links on PreOrder posts can now contain extra messages

Preorder link posts were a place to also share discount codes as well as links to products. With this change, we can hard code in messages to be appended to a link on a store by store basis. In this case, we added a standing discount code for popnbeards.com to appear next to their links in the preorder link threads.

Thursday, June 6, 2019

Changelog 2019-06-06 - New Feature

Preorder Posts will now automatically update when new items are found

With a few limitations which will be detailed below, any new item that is found by the web scrapers will automatically be appended to the appropriate preorder post on reddit.

Preorder posts are created instantaneously when a new Funko blog post is scraped and posted to Reddit. Once the preorder post is made, the blog post and the link to the preorder post are also posted to @PopCommunity on Twitter. Thus, these preorder posts have two means of visibility. The idea of them is to have the community share links to stores they have found which are offering preorders for the items that were just announced. The intention of this project was to cut down on the clutter found on r/FunkoPop as people would make a new post for preorders for each store found, so for highly anticipated releases, the sub would become cluttered with various preorder links. Additionally, it was impossible to navigate the space and see all of the posted options for preorders as the first preorder link to go up was usually the most visible, even if it was not the best option.

So with the preorder posts that are automatically generated, this fixes the issue by consolidating preorder links to a single posts. However, there is a manual step where I have to edit the post to include the preorder links that are found.

Considering my blog scrapers already find and identify new items, it makes sense to have them automatically update the preorder posts as well. This is done by looking at all of the preorder link threads created in the last day, giving a heavier weight to those created in the past 2 hours, and finding which post is most likely to be the correct post based on the count of shared words in the title of the item and the title of the blog. Once a candidate blog post is identified, it is updated with the new item.

This will not work perfectly every time, however. If we find a new item but that item was just late going up and does not correspond to any preorder posts, we might accidentally append it to a random preorder post. As such, we need to have some sort of threshold for the number of common words when determining in which post this preorder link belongs. This helps us with false positives but it creates a new problem. If a blog post has a sufficiently small number of words in the title, we may never cross the threshold of required common words and nothing will ever be appended to the post.

There are ways around this but for now, I am going to pilot this project and see how it does and refine it from there.

Hot Cash Give Away Jan 2021

  Hey everyone! Time for another round of Hot Cash codes! Hot Cash ends soon so be sure to use these ASAP! Comment below with the codes you ...