Incredible memory leak when not showing pictures
Posted: Thu May 03, 2012 1:46 pm
Well, this took me by surprise. In an attempt to save bandwidth, I disabled 'Show Pictures' in Internet Explorer (since Helium Scraper uses IE as the web browser). After a 20 minutes, Helium came back with a "Low on Resources!" error, which was interesting. Repeated tests with browser emulation - 7, 8, 9... all cause RAM usage to shoot up constantly and consistently. I'm running the Scraper now with images on and it's only reached 120MB, whereas with Images off it would have been 230MB right now.
I don't know where which component is at fault (Internet Explorer is very very bad though, so I like to point fingers at it first), but have you considered a 'Low Bandwidth' usage mode? I'm assuming your program was written in a .NET language (only assuming, from the installer), would something like HTMLAgilityPack work for running the actual scraping? I don't really need to view the pages when I've completed testing.
I don't know where which component is at fault (Internet Explorer is very very bad though, so I like to point fingers at it first), but have you considered a 'Low Bandwidth' usage mode? I'm assuming your program was written in a .NET language (only assuming, from the installer), would something like HTMLAgilityPack work for running the actual scraping? I don't really need to view the pages when I've completed testing.