Find Jobs
Hire Freelancers

Improve webscrapping

$30-250 USD

Closed
Posted over 3 years ago

$30-250 USD

Paid on delivery
I have a Java program to scrap information from a website. The architecture of the solution involves: 1) using Java Selenium to send requests to the webpage via Chrome Webdriver to trigger authentication and authenticated requests; 2) routing the requests from Chrome (headless) to Java BrowserMobProxy to capture three HTTP headers (Authorization, X-CSRF-TOKEN, and Cookie) and one query string; and 3) use these 4 elements in HTTPs requests from Java directly to the webpage (i.e. without Selenium, Chrome, and BrowserMobProxy involved) to retrieve the desired information. This program does the basic functionality of extracting the information but has a few problems: It depends on an external non-Java component: Chrome WebDriver It depends on Java Selenium and Java BrowserMobProxy, two dependencies that I would like to remove It is not optimized (too much refresh and too long sleep periods) relatively to the limit upon which the Webpage (Cloudfare) starts responding 429 errors. Thus, the retrieval of the information is taking much more time than needed. Deliverables You will get the current program Java code and you will need to solve the problems above. To do so, you will need to: A. Find out how to authenticate and refresh the 3 headers and the query string without depending on Selenium, Chrome Webdriver, and BrowserMobProxy. As most of this data is likely generated in JavaScript, you will need knowledge about JavaScript and how to execute JavaScript from within Java or convert the JavaScript code to Java (preferable solution). B. You will need to identify the limit upon which the Webpage (behind Cloudfare) starts responding 429 errors. You will need to tune the refresh frequency of the headers and sleep periods to the limit identified. You will need to demonstrate the benefits of your changes by extracting the information currently extracted by the program and measuring how long it takes. Note: you will need to create your own login/password in the webpage. No additional requirements exist to register.
Project ID: 26802694

About the project

12 proposals
Remote project
Active 4 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
12 freelancers are bidding on average $173 USD for this job
User Avatar
kindly provide the site details and which data you want to scrape. I will build a scraper that is not based on selenium nor chrome driver. Thanks
$300 USD in 3 days
5.0 (225 reviews)
8.0
8.0
User Avatar
I understand your code is working on server, RIGHT? I thinl php curl is a best method for your scraping, please contact.........
$200 USD in 3 days
4.9 (52 reviews)
5.7
5.7
User Avatar
Hi there, Let’s have a quick chat to discuss this project. I am expert in Python, PHP, JavaScript,Web Scraping,MYSQL.I do have expertise for this project. You can check my portfolio here:- https://www.freelancer.com/u/PoojaRautela417?page=portfolio&w=f&ngsw-bypass= Looking forward to hear from you soon. Regards Pooja Bohra
$200 USD in 2 days
4.8 (38 reviews)
5.3
5.3
User Avatar
Hello Sir! I am a web scrping expert, I think I'm a great fit for this project. because I have an interest in your project and can deliver on time, according to your specifications Thanks
$140 USD in 7 days
4.9 (8 reviews)
4.4
4.4
User Avatar
Ready to start the work to improve your web scrapper with all needs , please check my recent similar work ,we can discuss more over chat, thanks regards kanta singh. Scrapping projects link : https://www.freelancer.com/projects/python/Data-Scraping-Real-Time-for/ https://www.freelancer.com/projects/php/web-scrap-betradar/
$100 USD in 7 days
4.0 (18 reviews)
5.6
5.6
User Avatar
Hello Sir/Madam, i can make soluction for your problems just with make it with PHP and cURL. You will no have dependancy and will work very fast. My last 2 project was for same think you can check my reviews. Feel free to ask me any questions that you have. Best Regards!
$200 USD in 3 days
4.9 (22 reviews)
4.2
4.2
User Avatar
Hello, I'd like to take a look at this. Can you send me the website in question and the information you want to extract? I'll try to mimic the behavior of Selenium by sending the appropriate headers, then parse out the contents with Jsoup.
$170 USD in 7 days
5.0 (6 reviews)
3.6
3.6
User Avatar
Thanks for project posting   and I respect it  I recently worked on the project like yours and can provide you demo work as well  Do you want free demo ? ping me in freelancer message board  Thanks and Regards,      
$140 USD in 7 days
0.0 (0 reviews)
0.0
0.0
User Avatar
WEBSITE DESIGN & Web Development () ======================== Hello. I'm a WEB design,PHP, Shopify,Wordpress expert I have 5+ years of experience with Website development. ✅ PHP, Shopify,Wordpress Specialise Area ------------- ✅ Expert in HTML, CSS, Bootstrap, Javascript, jQuery, PHP, MySQL, and many types of API integration. ✅ Shopify Responsive design,theme adding custom options using customize plugin & Jquery,PSD to Shopify, ✅ Shopify custom theme designs based on your individual ideals and requirements. ✅ Shopify store development ,existing themes modification ✅ Shopify App Development ✅ Logo design ✅ Graphic design ✅ If interested i will share you some reference /my previous design work. I am confident I will deliver perfect results. Please feel free to contact me. Thank you.
$250 USD in 7 days
0.0 (1 review)
0.0
0.0
User Avatar
hello I hope you are doing well. I am working on data scraping using python using selenium and beautiful.
$30 USD in 2 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Good day. I'm interested in your project. I have about 3 years of scraping and 6 years of python programming experience. A big plus of using python is that everything will be automated and that I can write a program quickly. I use all modern scraping libraries like: beautifulsoap, selenium, request, scrapy and so on. I can deliver you the result in a form convenient for you: json, csv, txt, sql database. I also worked on large projects, and scraped large sites such as: Amazon, Alibaba, YouTube and so on, so I know how to work with large amounts of data. If I suit you as a specialist, we can discuss the project in more detail.
$140 USD in 7 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of ROMANIA
Băilești, Romania
5.0
1
Member since Mar 8, 2020

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.