run crawlers in batch mode?

Questions and answers about anything related to Helium Scraper
Post Reply
Posts: 1
Joined: Mon Jun 06, 2011 6:45 am

run crawlers in batch mode?

Post by fredrik » Mon Jun 06, 2011 6:48 am

I want to use Helium Scraper to run on hundreds of sites each day. What is the best way to achieve this, can I run the crawlers in batch mode? i.e. put all my crawler files in a folder and create a script that runs all of them one after the other or several at a time?



Site Admin
Posts: 496
Joined: Mon Dec 06, 2010 8:39 am

Re: run crawlers in batch mode?

Post by webmaster » Mon Jun 06, 2011 5:33 pm

Hi Fred,

Helium Scraper doesn't have that functionality. However, if these sites have similar structure and you want to extract the same content, then the "Navigate URLs" action might work for you. Other than that, you can have many instances of Helium Scraper running at the same time and have them minimized to tray.

Let me know if this works for you. Perhaps this could be a functionality to implement on a future version.
Juan Soldi
The Helium Scraper Team

Post Reply