Which CPU is best for scraping?
Posted: Thu Jun 27, 2013 8:02 pm
I plan to ask for a new computer that will be optimized for web scraping. Any suggestions?
it really depends on how intense your scrape is. for example, if you are going to be using the "multi-process" feature, which starts up multiple instances of helium to complete a scrape, you will definitely want a higher-end processor. if it is just basic or only using one process, you can get away with a lower end processor.bankratechrisHS wrote:I plan to ask for a new computer that will be optimized for web scraping. Any suggestions?
for example, i have 3 separate computers for different levels of scrapes. the most high-powered one runs on a 4th generation intel i7 with 16 GB of ram, and is running 36 instances of helium simultaneously at a time, pretty much 24/7. my mid-range computer is a 3rd generation intel i5 and 6 GB of ram, and runs 16 instances simultaneously at a time for a total duration of 30 minutes, every 2 hours for 1 hours a day. my low-end computer is an amd fx-4100 with 6 GB of ram that runs 3 virtual machines, each virtual machine running a single process scrape every 5 minutes 24/7.
to be honest, i'd get a higher-end processor just in case. you can get a computer tower by itself (no monitor, no keyboard, no mouse) with 16GB of ram, a 4th generation intel i7, and 1 TB hard drive for around $600 if you buy online. so pretty cheap.