Helium Scraper

So, lets me first state that this program is awesome. I give a big hand to the developers for coming up with such an affordable and user friendly software. It is definitely just as easy to handle as Mozenda (once you figure it out) without the hefty per page scraping fees, and shows a lot of potential. I will be making a purchase so long as this thing can get these two basic jobs taken care of. I have a couple of issues though...

My first project entails me to acquire a digital product catalog from a specific website. I'm trying to program helium to navigate one specific parent category and then all its children categories, and finally each product listing scraping details such as price, description, name, specs, image, etc...

Most importantly i cant seem to get any of my images to pull, I've done as directed by applying the SrcAttribute property to it, but still dos'nt work. Tried to get the direct link, no go. I would be satisfied with a direct link as that would make my job easier.

I'm having a couple issues with regards to how I'm suppose to implement the "If/While" premade, as some categories extend deeper than others. I'm a bit confused on how the tree needs to be constructed?

My tree looks something like this... but were do i insert the if/while and is the following Navigation directive suppose to be listed as the If/Whiles child? I'm confused!
-----------------------------------------------------------------
-Repeat (1) Times (1/1)
--Navigate Each "Top Cat"
---Naviatge Each "Sub Cat"
----Navigate Each "Sub Sub Cat"
-----Naviate Each "Sub Sub Sub Cat"
-------Extract "Table Data"
-------Execute "Go Through All Pages"
------------------------------------------------------------------

I have tried to upload my .hsp but it tells me the file is to big, It does contain 16 kinds, If/While & Go Through All Pages Action

A little more additional information...

Ex...

http://scene7.samsclub.com/is/image/sam ... _sharpen=1 is the actual src attribute. The parameters (?wid=206&hei=200&op_sharpen=1) following the filename (0004000001231_A") are unnecessary and I would like to exclude these... so i can download the big picture.

<IMG style="BORDER-BOTTOM: #cccccc 1px solid; BORDER-LEFT: #cccccc 1px solid; PADDING-BOTTOM: 3px; PADDING-LEFT: 3px; PADDING-RIGHT: 3px; BORDER-TOP: #cccccc 1px solid; BORDER-RIGHT: #cccccc 1px solid; PADDING-TOP: 3px" id=imgElm title="M & M's® Milk Chocolate - 48 / 1.69 oz. pk." alt="M & M's® Milk Chocolate - 48 / 1.69 oz. pk." src="http://scene7.samsclub.com/is/image/sam ... _sharpen=1" width=206 height=200>

is the main element i get when selecting the picture. I've tried selecting the parent, (up to 3 levels up) to no avail.

While stepping away from this job, I tended to another... I have another catlog a website that is built with frames, how can access the content and navigate it within a frame. Im not sure if it's an Iframe but the syntax just says <frame>.

The webisites in reference are grocery.walmart.com & samsclub.com. Walmart has the frame trickery and sams club the image trickery

Ok, got the images thing fixed. I was making it harder than it had to be. Turns out I needed to deligate more than just a couple images in one category to the kind. After about 20 elements defined to the kind, they started rolling off the assembly line.

Im still a little confused how to implement the if/while case if my category depth dosnt excend the maximum depth in the tree, & also this iframe thing.

Direction on both matters would be highly appreciated! Great software again, who needs to rent Mozenda when you can own your own!

I retrack the last post, 26 hours of pure frustrations. Everytime i shutdown the software and reload a scrape, I seem to have to reset all of the elements pertaiingin to my kinds. The images are downloading but its downloading 5 copies at a time, and also inserting 5 copies into the database with only one copy within the database retaining all the data. This things got a mind of its own, good conceptual view.... But sure aint ready for todays web!

Sorry, my frustrations are getting to me! Cant wait to meet Juan, he gots all the answers!

Hi,

I cannot tell what the problem is without looking at the project. If the forum doesn't allow you to send it, try using a service such as this. My guess is, since you have so many levels and Navigate Each actions, one of these is not navigating inside each link. What I do when the project gets this complex is break it apart into smaller tasks by extracting the top categories URLs to a table and then using a Navigate URLs action to navigate inside them and then do the same with inner levels.

Juan,

Thanks for the reply Juan, I knew you would eventually be here... In reference to samsclub.com strip, i did successfully get this to work as intended. I needed to break it down into multiple scrapes with deeper navigate eaches for certain categories. Tedious but it worked and sure worked better than copy and pasting.

The only issue i have left is grocery.walmart.com... This site is broken up into frames and I cant seem to figure out how to break into the main content frame and extract data. I have tried using a what i think is a direct url obtained from the source code to no avail but i may be doing it wrong. You think you can guide me?

These scrapes will need to be performed on a bi-weekly basis so, If i can get this thing to work both sites correctly I will definitely be purchasing.

Hi,

Seems like the iFrame has a script that goes back to the original page whenever is loaded in the main window. Usually you can follow the src attribute of the iFrame. I'll see if there is anything that can be done from Helium Scraper itself and provide an update as soon as I find a solution.

Great, cant wait!