Memory/Resources problem with new version

Let us know if anything goes wrong with our baby :)
jafuli
Posts: 17
Joined: Fri Aug 19, 2011 8:26 am

Memory/Resources problem with new version

Post by jafuli » Fri Mar 02, 2012 12:38 pm

I am running a script developped with former version. I get messages like 'out of resources' and must pause the actions. Memory usage is very large. I had run this script before, on same amount of data, with no problem.

jafuli
Posts: 17
Joined: Fri Aug 19, 2011 8:26 am

Re: Memory/Resources problem with new version

Post by jafuli » Fri Mar 02, 2012 12:39 pm

details of said problem :
<?xml version="1.0" encoding="utf-16"?>
<ErrorReport xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<TimeStamp>2012-03-02T13:39:09.8679915+01:00</TimeStamp>
<Message>Ressources système insuffisantes.</Message>
<StackTrace> à System.Data.OleDb.OleDbCommand.ExecuteCommandTextErrorHandling(OleDbHResult hr)
à System.Data.OleDb.OleDbCommand.ExecuteCommandTextForSingleResult(tagDBPARAMS dbParams, Object& executeResult)
à System.Data.OleDb.OleDbCommand.ExecuteCommandText(Object& executeResult)
à System.Data.OleDb.OleDbCommand.ExecuteCommand(CommandBehavior behavior, Object& executeResult)
à System.Data.OleDb.OleDbCommand.ExecuteReaderInternal(CommandBehavior behavior, String method)
à System.Data.OleDb.OleDbCommand.ExecuteNonQuery()
à Player.Data.DataBaseFile.ExecuteNonQuery(String sqlQuery)
à Player.DataBaseManager.ExecuteNonQuery(String sqlQuery)
à m.b(Dictionary`2 A_0, Boolean A_1)
à Player.Actions.ExtractParallel.b.e()
à Player.Actions.ExecutionData.f.b()</StackTrace>
<TypeName>System.Data.OleDb.OleDbException</TypeName>
<Version>2.3.1.2</Version>
</ErrorReport>

webmaster
Site Admin
Posts: 521
Joined: Mon Dec 06, 2010 8:39 am
Contact:

Re: Memory/Resources problem with new version

Post by webmaster » Sat Mar 03, 2012 1:51 am

Hi,

Any chance you can send us this script?
Juan Soldi
The Helium Scraper Team

jafuli
Posts: 17
Joined: Fri Aug 19, 2011 8:26 am

Re: Memory/Resources problem with new version

Post by jafuli » Mon Mar 05, 2012 10:30 am

I am sending you a mail on webmaster@...

jafuli
Posts: 17
Joined: Fri Aug 19, 2011 8:26 am

Re: Memory/Resources problem with new version

Post by jafuli » Mon Mar 05, 2012 2:00 pm

I sent you a mail ...

Also : I was able to get all the data by running the project twice, at two different points in the scrapped webpage (launching the software anew everytime). The error message comes after some time (I have some 475 links to scrap ; first pass blocks around 250, and it does not seem to block everytime at the same point ... )

chillmen
Posts: 3
Joined: Mon May 09, 2011 8:25 am

Re: Memory/Resources problem with new version

Post by chillmen » Wed Aug 08, 2012 8:36 pm

Same problem here, but i'm not using any script.
I receive messages like out of memory, during scrapping and i need to pause and play every 10 or 13 pages.
Also i can't save projects i receive the same message "out of memory"
Thanks for your help!

derba
Posts: 27
Joined: Tue May 15, 2012 8:03 pm

Re: Memory/Resources problem with new version

Post by derba » Sat Sep 01, 2012 9:07 am

Hello,

I have the same problem than Chilmen :
"
I receive messages like out of memory, during scrapping and i need to pause and play every 10 or 13 pages.
Also i can't save projects i receive the same message "out of memory"
"


I have noticed the following :
When I run helium scrapper on a simple project :
follow a list of urls, find a link inside the url and download the html file out of the link.
After a few hundred urls, Helium must stop as my computer starts having some memory problem.

But for some reasons the file : "pagefile.sys" becomes huge (thanks to http://windirstat.info/, then look under "files" folder on win7 64bits) ! ??
2012-09-01_105211.jpg
2012-09-01_105211.jpg (13.33 KiB) Viewed 27476 times
notes:
- Helium was the only program to run ! And I started to run out of space on C:\ .
- the size of all the html files downloaded is only 45 MB in total !

A question : why ?
Is Helium not able to free the computer memory usage ( a little like Firefox last update - v15 - where addins memory usage were not freed when not used anymore) ??

Thanks in advance ;)

scraper
Posts: 1
Joined: Sat Sep 07, 2013 7:42 am

Re: Memory/Resources problem with new version

Post by scraper » Sat Sep 07, 2013 7:46 am

Hi,
I am having the same problem. Bought Helium Scraper a few days ago and cannot do any large project w/o getting the error described above.

Can I please get a refund or the link to a download of an older version that does not have the issue.

Urgently, please, I have a large scraping job that needs to be finished asap.

Thanks

derba
Posts: 27
Joined: Tue May 15, 2012 8:03 pm

Re: Memory/Resources problem with new version

Post by derba » Sat Sep 07, 2013 8:11 am

Hello,

I wrote my message one year ago and never get an answer. So I don't use this software anymore ! ;(

I wanted to encourage its code writer for making it better but in fact I bought it for nothing. ;(

Good luck !

webmaster
Site Admin
Posts: 521
Joined: Mon Dec 06, 2010 8:39 am
Contact:

Re: Memory/Resources problem with new version

Post by webmaster » Mon Sep 23, 2013 1:23 am

Guys,

First of all, installing Internet Explorer 10 (which is already available for Windows 7) will drastically improve memory usage. This memory problem is related to a component used by Helium Scraper which is updated when you update Internet Explorer, and the problem is fixed for most sites with IE 10. Now, if a site still causes a memory leak even with IE 10 installed, you'll need to use multi-process extraction to prevent this from happening. Here is a video that shows how to turn a single process extraction into a multi-process one.
Juan Soldi
The Helium Scraper Team

Post Reply