Turning pages without "Next" button
Posted: Wed Jun 29, 2011 3:22 am
There are a few cases on which a set of results pages won't provide a "Next" button, but instead, just a list of page numbers.
The attached project is a simple solution to this problem. To use it, just import it into the project you are working on, or use it as a starting point, and then create a kind that selects the page number that comes after the current page. Then navigate to another page and add the page number that comes after that page. So if you are, for instance, at page 1, select the link that takes you to page 2, and create a kind called "Next" with it. Then navigate to page 2, select the link that takes you to page 3 and add it to your "Next" kind.
After this, when you navigate to any other page, this kind will always select the next page number just as if it were a "Next" button and you can use it as such for every automation purpose. It will also work with the Go through all pages premade project.
Without this project, if you try to follow the steps I describe above, your kind will most likely end up selecting all page numbers, instead of just the next one. This is because this project contains a set of JavaScript gatherers that gather properties from elements that are located before the element you select when creating a kind, which on this case means that it will gather properties from the current page number when you select the next page number. Since the current page number is different that all other page numbers (i.e. is not a clickable link), Helium Scraper will be able to distinguish between the next page number and any other page number.
I've tested it on several search engines and sites and it worked OK. But if you notice that your kind starts selecting every page number instead of just the next one, let me know and I'll add whatever JavaScript gatherer is necessary to have it working properly.
The attached project is a simple solution to this problem. To use it, just import it into the project you are working on, or use it as a starting point, and then create a kind that selects the page number that comes after the current page. Then navigate to another page and add the page number that comes after that page. So if you are, for instance, at page 1, select the link that takes you to page 2, and create a kind called "Next" with it. Then navigate to page 2, select the link that takes you to page 3 and add it to your "Next" kind.
After this, when you navigate to any other page, this kind will always select the next page number just as if it were a "Next" button and you can use it as such for every automation purpose. It will also work with the Go through all pages premade project.
Without this project, if you try to follow the steps I describe above, your kind will most likely end up selecting all page numbers, instead of just the next one. This is because this project contains a set of JavaScript gatherers that gather properties from elements that are located before the element you select when creating a kind, which on this case means that it will gather properties from the current page number when you select the next page number. Since the current page number is different that all other page numbers (i.e. is not a clickable link), Helium Scraper will be able to distinguish between the next page number and any other page number.
I've tested it on several search engines and sites and it worked OK. But if you notice that your kind starts selecting every page number instead of just the next one, let me know and I'll add whatever JavaScript gatherer is necessary to have it working properly.