Helium Scraper • URL Variations - Progress

Page 1 of 1

URL Variations - Progress

Posted: Sat Mar 31, 2012 2:11 pm

by Jake

Just bought Helium Scraper - It's an amazing product!

That being said - Using the URL Variations online template is incredibly useful - But the one thing that you can't see is the current page you're scraping when using it.
For example, It would be nice to see something like:

Execute Tree: URL Variations - Page 4 of 10

in the actions tree.

Maybe this is possible through scripting?

Re: URL Variations - Progress

Posted: Sun Apr 01, 2012 7:21 pm

by webmaster

Hi Jake,

This is a cool idea. I've just added it to the our TODO list. For now, though, you can use the log. Here is an example. Just go to the URL Variations tree, open Execute JS (Do Navigations) and replace the existing code by this:

Code: Select all

if(Tree.UserData.currentPage > Tree.UserData.lastPage) 
{
	return false;
}
else
{
	Global.Log("Page " + Tree.UserData.currentPage + " of " + Tree.UserData.lastPage);
	var url = Tree.UserData.url.replace(new RegExp("\\$", "g"), Tree.UserData.currentPage);
	window.location.href = url;
	Tree.UserData.currentPage += Tree.UserData.stepSize;
	return true;	
}

As you can see, this code has just one extra line. You can then see the current page at Project -> View Log. Note that you can use the buttons above to filter the kind of messages shown. You might want to keep only User Events active so that it shows only your current page.