Thursday, May 23, 2019

Changelog 2019-05-23 - One Fix

@FunkoWalgreens is back online

The twitter bot @FunkoWalgreens has been offline for some time as it was unreliable. A change to the way the HTML is scraped allowed for all products to reliably be captured on each run. The bot will once again work as intended.

Issues revolved around the bot trying to scrape the page as a spoofed browser due to limitations o f a normal get request. However, adding appropriate headers to the get request allows for the entire page the be returned. Using browser spoofing was ineffective because the site would lazy-load, causing us to only pick up about half the items on each execution.

No comments:

Post a Comment

Hot Cash Give Away Jan 2021

  Hey everyone! Time for another round of Hot Cash codes! Hot Cash ends soon so be sure to use these ASAP! Comment below with the codes you ...