Scraper/webscrawler written in python

Hi, I can help you fixing your script and I can also use my resources to get your data within 1 day, not 3-4 :) let me know what site you scraping so I'll estimate how much that will cost. Thanks

Hi sir, I am scraping expert, I have did too many similar projects, please check my feedback then you will know. Can you tell me more details? then I will provide demo data for you. Thanks, Kimi

Hello I can try to help you. But I need more details. I have an experinence of creating python scripts to extract data from various websites. Waitng for your response.

Hi, I am a professional web data scraper specialized using Python program, PHP script, .Net program, Crawler and Bot. My tool can search data and get information from Aa to Zz with an existing lists of english words. Below is the link for your reference as a sample related to my tool being developed. This demo will capture doctor's name, address, zip, phone, ratings and reviews in 4 different sites. The final output will be save in *.XLSX format or as your quirement.I can start as early possible depending on your approval and acceptance. In relation to this application, I can rest assured I will impart a high quality and reliable, efficient and accurate with the output. Give me a try and I will try to get the best results and finish the project far before the deadline. Thanks,Ferdous

Hello! We are a small team of Python/JS developers. We have experience in developing custom webscrapers and other standalone scripts. We have some question about your project. What kind of pages do you scrape? 300000 per hour is rather high rate. What environment do you use? Is it ok for you if script will run multiple threads. How do you launch the script and pass pages to it? Do you need to connect it with any kind of UI? We will be happy to help you! Thanks and Regards!

A proposal has not yet been provided

Hello. I am a professional Python programmer with several years of web scraping experience. I can optimize the webcrawler you already have, although I would prefer to build my own from scratch if that would be okay. Please contact me so that we may speak further, and I hope to work with you soon.

Hello! I'm web scraping expert. I use python scrapy framework. My scripts can run on windows or linux, but linux is preferably. I can schedule scripts on server if it is required. I have a lot of finish projects (google scraping, facebook scraping, yellow pages, linkedinIn, amazon, webshops and other sites with lists of any items). I can scrape secured and protected sites, my crawlers can enter into login form, emulate ajax requests etc. If site block IP i can use proxy or TOR. I can try avoid captha on site in avtomatic or manual mode. I can export data into json, csv (excel), mysql, mongodb.

Hi, Could you make the source code available? I am not sure if I am understanding the requirements correctly: do you want to have the data scraped within 3-4 days? Does that include the project completion time? Could you provide a bit more detail in that aspect? Looking forward to your reply, Artur

We have a good amount of experience in webscraping using Python,Django and nodejs. This is our latest project on webscraping using python: Scraping using Python: Electronics Parts Intelligence Processing eProductScrapper is mostly scraping & data-mining oriented project, which is based on scrapy and lxml plugins, along with Celery distributed environment via redis. This is mostly focused on electronics parts to fetch information like product details, sku, technical datasheet(pdf), product stock, price history. which will be used to make product life-cycle in a highly presentable manner to make non-authorized seller, brokers, after market sellers more aware of the market requirements of the products. Technology & Framework Used: Python, django, celery, scrapy, nodejs, mongodb, mysql. We would love to have ongoing relationships with your team and ready to work on your time schedule 40-50 hrs per week as per requirements. Thanks

hello : python will be used as you like . i have done crawler with python + scrapy + selunium etc .. maybe you can let take a look at the script to see if i could do it. btw , my skill is python2 ty

Python developer. Have experience in parsing sites using various technologies : BeautifulSoup,Selenium etc. Please provide me the url of the site so I can look if I can make your script running faster.

Hello, i can change the script to use multiprocess if it should run on the same server and making several processs using all cores of the cpu or if using different server make it using celery to spread the task to multiple server and also using all cores. Best regards, Thorsten Sanders

I got 7+years work experience in Data Collection,Bulk Email Campaign,Excel VBA and Internet Research in IT companies here.I can do write and modify script and scrap datas from site using C++,Python and Perl coding as per your requirements in excel with multiple ip rotations.I have dealt with US,UK and Australia companies President,Directors and Managers for web design and development projects successfully and I have Good Communication with writing skills.I am well versed in Internet,MS Office Applications and Phone Etiquette manners with latest Technologies.I can accept your payment terms.

I am developing/modifying web scrapers for quite some time. I am also familiar with requests library if that's what you think of in your project desc. In order to be ore precise with the deadline I need more details (and a specification of tasks too) on the project. Cheers

Hi, expert programmer and web/data scraper here with over 19 years experience in programming and RDBMS. Please see my reviews. I'm using Python or Perl for this kind of jobs. My offer is 20 usd (without fees) per 1 000 records I'm able to extract data fast.

Dear Sirs, I'm a Python web scraping specialist. I know and can manage the selenium PhantomJs framework. But I also use other environments such as scrapy, urllib, beautifulsoup, requests. Etc. I believe I can help you to improve the extraction rate and propose alternative methods. Just give me an opportunity. Thank you Alejandro.

Hi, I have done exact same optimization for another client on Upwork already. The job was to scrape all books information for trade in from Amazon (it was around 500,000) and the script based on phantomjs worked too slow - the average scraping speed was around 1 book per second. When I redesigned it to use "urllib" module (similar to "requests" module), the scraping speed increased to 8 books per second, so all the 500,000 books were scraped in 19 hours. Both designs included multiprocessing, however urllib turned out to be much faster and robust than phantomjs. Please, contact me if you have any additional questions. Regards, Maksym

Hi, I have masters degree in parallel programming and experience with Python. I think I can help you. I would create a parallel code that creates a streaming of fetched pages. It could scale with your connection. I would also create a pool of consumers that scrap the content and save wherever you need. I would use Redis to centralize everything. As you have 300k items, the data probably fits in memory. If no, we can use a persistent solution like mongo, or even the filesystem. I don't know if you have a list of urls or if it's built from the previous scrapped pages. If so, redis could also help to create a list of pages to be consumed, and the downloaders would act as consumers of this queue. We can use requests to download and lxml to parse the html. I think this architecture would optimize the two bottlenecks, the page fetch and the html scrapping.

I can write it to requests & xpath, if run the webpage javascript is unnecessary , using requests maybe a better way, and using multiprocessing to run multi instance on a machine or several servers, if run on several servers , using Redis as message queue maybe a good way.