Which CPU is best for scraping?
-
bankratechrisHS
- Posts: 3
- Joined: Wed May 29, 2013 4:03 am
Which CPU is best for scraping?
I plan to ask for a new computer that will be optimized for web scraping. Any suggestions?
-
crookedleaf
- Posts: 38
- Joined: Tue Dec 11, 2012 6:44 pm
Re: Which CPU is best for scraping?
it really depends on how intense your scrape is. for example, if you are going to be using the "multi-process" feature, which starts up multiple instances of helium to complete a scrape, you will definitely want a higher-end processor. if it is just basic or only using one process, you can get away with a lower end processor.bankratechrisHS wrote:I plan to ask for a new computer that will be optimized for web scraping. Any suggestions?
for example, i have 3 separate computers for different levels of scrapes. the most high-powered one runs on a 4th generation intel i7 with 16 GB of ram, and is running 36 instances of helium simultaneously at a time, pretty much 24/7. my mid-range computer is a 3rd generation intel i5 and 6 GB of ram, and runs 16 instances simultaneously at a time for a total duration of 30 minutes, every 2 hours for 1 hours a day. my low-end computer is an amd fx-4100 with 6 GB of ram that runs 3 virtual machines, each virtual machine running a single process scrape every 5 minutes 24/7.
to be honest, i'd get a higher-end processor just in case. you can get a computer tower by itself (no monitor, no keyboard, no mouse) with 16GB of ram, a 4th generation intel i7, and 1 TB hard drive for around $600 if you buy online. so pretty cheap.