How to "Dynamically Set X in the Repeat X Times" option?

Questions and answers about anything related to Helium Scraper
Post Reply
acecorp
Posts: 5
Joined: Mon Feb 11, 2013 3:03 am

How to "Dynamically Set X in the Repeat X Times" option?

Post by acecorp » Mon Feb 11, 2013 3:49 am

I am trying to scrape inventory data from a website.

The website does not have a next button and I am using the premade listed under 'Turning pages without "Next" button'

Currently, I have repeat set to 10 times.

My understanding of how this works is that the process starts, navigates to each link, extracts the data, and then turns the page.

Then, repeats so the repeat counter is now 2/10.

The number of pages could be as small as 24 or as large as 1000 or more, so setting the repeat to 10 will prevent me from getting all the inventory. However if I set it to 100000, it might cycle through the inventory 100 times and over write or duplicate inventory.

Is there a way to set this up so it only goes through the inventory on all the pages until it gets to the end?

I updated my current (updated project file) if looking at it is helpful.
Attachments
masanoproject_rev1.zip
Project Rev1
(59.83 KiB) Downloaded 191 times

acecorp
Posts: 5
Joined: Mon Feb 11, 2013 3:03 am

Re: How to "Dynamically Set X in the Repeat X Times" option?

Post by acecorp » Mon Feb 11, 2013 4:48 am

I got this working. This is fixed using the "Go through all pages" premade.

Post Reply