I need an experienced freelancer or team in scrapping (not intermediaries!), to implement a scrapping architecture that will assume that all target sites are protected.
The implementation of the scrapers will use proxies (we need to discuss the best solution with rotating proxies), and use multi-threading with multiple proxies to highly improve the speed of scraping.
It will be a MAYOR plus if you already have many proxies and are able to test a basic scraper of the first site (only grab basic details like Price and Surface for all listings) and determine if you are capable to fulfill the time requirements before we move forward.
Amount of scrapers to implement: 3 (their URLs are in the attached file called "[url removed, login to view]")
Maximum time expected for each scraper run to take: 14-24 hours.
Technology to use: I'm open minded here, as soon as achieves the best results
Database to use: MySQL
General architecture details
- Must be always multi-threading (and must use each of its threads with a different proxy to highly increase scraper performance)
- Each scraper is separate and can be run at any time independent of the others
- Make a simple Admin panel to allow to manage the different scrapers (attached image "[url removed, login to view]"). Example of the table style used: [url removed, login to view]
- Scrappers Steps:
+ Initial validation (to check if the target site changed and stop the run if it fails)
+ First "Job" that will scrape only the surface of Search Results (to obtain only all the IDs on the target website without scraping the inner details)
+ Second "Job" that will use the result of the first one, to compare the IDs obtained with the ones we already have and scrape only the ones we need (this comparison will tell us which IDs to scrape more in details)
I will provide the detailed specification for each scraper when I discuss with freelancers under consideration. We can set a milestone per scraper.
I will only release the milestone for each scraper when is tested on my side and checked it works fine as expected.
Please only apply if you have good experience in high performance scrapping on protected websites.
33 freelancers are bidding on average $598 for this job
I am ready to get started right away.... Can we discuss the project details? My distinction, payment after your complete satisfaction with the resulted task.