Infinite Scroll with a "Load More" Button

Premades & Resources to be used with Helium Scraper 3
Post Reply
webmaster
Site Admin
Posts: 491
Joined: Mon Dec 06, 2010 8:39 am
Contact:

Infinite Scroll with a "Load More" Button

Post by webmaster » Fri Jan 04, 2019 8:11 am

This function handles pages with infinite scroll when a button must be clicked to load more elements. For a quick start, check out this video tutorial on how to quickly import and use this premade from within Helium Scraper.

This function takes the following arguments:
  • itemSelector: A selector that selects the items in the scrollable area, typically each of the results items. After scrolling down, the previous elements will be deleted to minimize memory consumption. Usually these elements can be automatically selected using the Detect List button on the page.
  • buttonSelector: A selector that selects the button that loads more elements.
  • scrollDelay: A wait time in milliseconds, to wait for new data to load after scrolling down.
  • removeOldElements: Whether to remove old elements after they've been extracted. Should be true unless removing elements breaks the page's scrolling mechanism.
When the action runs, it selects each of the list items on the browser, so if extracting from a list, the list items should not be selected again above the extract action or below this action.

To use this on your project, follow these instructions:

Code: Select all

function (itemSelector buttonSelector scrollDelay removeOldElements)
   Browser.ScrollLoop
      ·  itemSelector
      ·  buttonSelector
         Browser.Click
         Browser.Wait
            ·  scrollDelay
      ·  removeOldElements
The maximum number of elements to be extracted can be limited using the Sequence.Take function as in the following example (this example assumes the project already contains a selector called ListItem that selects the list items, another selector called LoadMoreButton that selects the button that loads more elements, and the code above was pasted under a global called InfiniteScrollButton):

Code: Select all

Sequence.Take
   ·  500
   ·  InfiniteScrollButton
         ·  Select.ListItem
         ·  Select.LoadMoreButton
         ·  100
         ·  true         
Note that the example above would extract 500 list items, each of which will usually occur many times per page, as opposed to 500 pages.
Juan Soldi
The Helium Scraper Team

ABkeeper
Posts: 3
Joined: Fri Jan 04, 2019 1:57 pm

Re: Infinite Scroll with a "Load More" Button

Post by ABkeeper » Fri Jan 04, 2019 3:00 pm

Is it possible to have a more detailed description of scraping in Helium 3 sites like:

https://www.topdrawer.co.uk/exhibitors#/

How to continue scraping if there is such a site structure (the button SEE MORE).

Thank you.

P.S. It would be great if you added new videos to the video tutorial. 8 videos since the appearance of Helium 3 is not too much.

webmaster
Site Admin
Posts: 491
Joined: Mon Dec 06, 2010 8:39 am
Contact:

Re: Infinite Scroll with a "Load More" Button

Post by webmaster » Fri Jan 04, 2019 4:18 pm

I will soon upload an infinite scroll tutorial.

I think you're referring to the tutorials playlist. We've recently added a samples and fundamentals playlists. The fundamentals playlist is one I'd really recommend watching in full to get a clearer idea of how the whole extraction process works in Helium.

Let me know if there's any other particular kind of extraction you'd like a tutorial about.
Juan Soldi
The Helium Scraper Team

webmaster
Site Admin
Posts: 491
Joined: Mon Dec 06, 2010 8:39 am
Contact:

Re: Infinite Scroll with a "Load More" Button

Post by webmaster » Tue Jan 08, 2019 9:09 pm

I've added tutorials for this and the "Infinite Scroll" premades on top of the description.
Juan Soldi
The Helium Scraper Team

Post Reply