Infinite scroll stop after a couple scrolls

Questions & Answers about Helium Scraper 3
Post Reply
Stark8026
Posts: 2
Joined: Fri Mar 10, 2023 9:19 pm

Infinite scroll stop after a couple scrolls

Post by Stark8026 » Fri Mar 10, 2023 10:03 pm

Trying to get wine info on this page but Infinite scroll stop after a couple scrolls.
https://www.vivino.com/explore?e=eJwNxE ... ficYG1M%3D

It helps increasing the scrollDelay but over a certain milliseconds delay, wathever the delay specified, it stops. I tried to increase navigation timeout, remove image handling and increasing max retries in project settings.

Same same, I cant get more than 175 lines (items).

I think it may be a protection on the page itself, because the offscreen browser show a page without loading new paage scroll button at bottom.

Here my code :

https://i.imgur.com/Df4tgZ6.png
Attachments
Df4tgZ6.png
Df4tgZ6.png (78.14 KiB) Viewed 6804 times
Last edited by Stark8026 on Sat Mar 11, 2023 8:38 pm, edited 1 time in total.

Stark8026
Posts: 2
Joined: Fri Mar 10, 2023 9:19 pm

Re: Infinite scroll stop after a couple scrolls

Post by Stark8026 » Sat Mar 11, 2023 8:38 pm

I also tried with slowscroll script. Let it go for 5 hours. It slow as **8 and I think there a bug again cause 7000 results divided / 25 = 280 scrolls whihc is not so long. In the ends no results were collected.

See attached screens for settings i used for the Scrollto bottom tryout
.
Attachments
Capture d’écran 2023-03-11 153529.png
Capture d’écran 2023-03-11 153529.png (84.9 KiB) Viewed 6805 times

webmaster
Site Admin
Posts: 521
Joined: Mon Dec 06, 2010 8:39 am
Contact:

Re: Infinite scroll stop after a couple scrolls

Post by webmaster » Wed May 10, 2023 8:44 am

The slow scroll premade won't help here since that scrolls just once. I've noticed that the site sometimes won't load more items after scrolling to the bottom, even when manually scrolling, so when this happens Helium will think there are no more items and stop scrolling.

Note sure if there's much that can be done in this case other than figuring out which requests the site is making and getting the data from there. Check this video for an example on how to do this, or if you know some TypeScript you can also use this feature.
Juan Soldi
The Helium Scraper Team

Post Reply