how do I get to the scrapeable part?

Questions and answers about anything related to Helium Scraper
Post Reply
eden
Posts: 2
Joined: Mon Oct 22, 2012 12:05 am

how do I get to the scrapeable part?

Post by eden » Sun Jul 14, 2013 3:27 am

I have a couple websites that display the info I want, in eminently scrapeable form- but I have to ask the search function for each record, one at a time, by that record's unique identifier in order to get to the page I can scrape.
For records whose unique identifier is a number of a known format within a known range, is there a way to automate asking the search function for each possible record? Likewise, is there a way to automate entering all possible combinations for a record set whose records' unique identifiers are a pair of short text strings?
Thanks.

crookedleaf
Posts: 38
Joined: Tue Dec 11, 2012 6:44 pm

Re: how do I get to the scrapeable part?

Post by crookedleaf » Thu Sep 05, 2013 1:01 am

eden wrote:I have a couple websites that display the info I want, in eminently scrapeable form- but I have to ask the search function for each record, one at a time, by that record's unique identifier in order to get to the page I can scrape.
For records whose unique identifier is a number of a known format within a known range, is there a way to automate asking the search function for each possible record? Likewise, is there a way to automate entering all possible combinations for a record set whose records' unique identifiers are a pair of short text strings?
Thanks.
not sure if it's possible with what you are doing, but you can have one table with all the unique ID's. then create an action using NEW ACTION > EXECUTE ACTION TREE > MORE and selecting "Write From Database". what this essentially does is allow you to loop through searching the unique ID's. so you set the actions to navigate to the page that has the search. create a kind for the search box, then add the "Write From Database" action. INSIDE the action, you put your extract option. you may also need to create a kind for a submit button if there is one needed to submit the search (this will need to be the step before the extract action).

Post Reply