URL Variations

Premades & Resources to be used with Helium Scraper 3
Post Reply
webmaster
Site Admin
Posts: 521
Joined: Mon Dec 06, 2010 8:39 am
Contact:

URL Variations

Post by webmaster » Thu Apr 05, 2018 4:39 am

This is a quick way to generate a list of URLs when a number in the URL is the only thing that changes, typically the page number or the starting item, such as in this example:
The function takes 4 arguements:
  • url: The base URL. Must contain a @ symbol to be replaced with each number.
  • start: The starting number.
  • step: The step between each number.
  • count: The number of URLs to generate.
To use this on your project, follow these instructions:

Code: Select all

function (url start step count)
   Sequence.FromValues
      ·  List.Range
            ·  0
            ·  count
   as index
   return
      ·  String.Replace
            ·  url
            ·  "@"
            ·  Core.ToString
                  ·  +
                        ·  start
                        ·  *
                              ·  step
                              ·  index
To load the output URLs, right click the new action and select Output Result, and then give a descriptive name to the output variable, such as url. Then, add a Browser.Load action and pass this variable as the argument.

The following example will generate the 3 URLs on the list above and navigate to each of them. This example assumes you've already pasted the code above and called it Variations:

Code: Select all

Variations
   ·  "https://www.google.com/search?q=helium+scraper&start=@"
   ·  50
   ·  10
   ·  3
as url
Browser.Load
   ·  url
Juan Soldi
The Helium Scraper Team

sawal86
Posts: 4
Joined: Tue Jun 19, 2018 2:16 pm

Re: URL Variations

Post by sawal86 » Tue Jun 19, 2018 2:27 pm

I have ready list of links and I need to make Helium 3 to navigate each link and extract needed info. I have had a few tries, but couldn't resolve it. I have created database with 2 columns: ID and link. Help please!

webmaster
Site Admin
Posts: 521
Joined: Mon Dec 06, 2010 8:39 am
Contact:

Re: URL Variations

Post by webmaster » Tue Jun 19, 2018 7:17 pm

Juan Soldi
The Helium Scraper Team

Post Reply