I need to scrape data from 10 websites. Targeted data types are – links, text and numeric. After scraping I need to rearrange, sorting and recalculate all data before a show to my website. Targeted websites add/change/update their data around 2-3 times per hour. So, I need those scrapers run/scrape automatically 4-5 times per hour.
Similar data found on multiple sites should be overwritten/skip/updated while scraping. Similar numeric data should be overwritten/skip/updated or replaced by higher/lower value while scraping. Some data should be overwritten/skip/updated by early/latest date and time.
There are some custom calculators are present on targeted websites. If those calculators can’t be scraped then those calculators have to be built with collected numeric data. Some numeric data need to change by 5%-10% before showing on the website.
Hello, I have experience in web scraping with Python. I can use Selenium, Scrapy, BeautifulSoup and Requests to make the best web scrapers! I hope to work with you!