Closed

Scraping Expert for XML Sitemap Generator

I would like to build an XML sitemap generator in either Python or other language that is preferred by the developer for this type of project.

Here are the features I would like to have for the generator:

- Pattern-matching to check for parallel pages in other languages/locale

- Configurable patterns based on the site URLs

- Ability to configure to crawl only specific directories

- Ability to disallow from crawling parameters

- Cloud-based GUI to add URL and start crawl (I can host on my VPS)

- Menu to export crawl results, configure crawl URLs, disallow parameters, etc. (See Screaming Frog)

- Crawl large sites up to 1,000,000 pages

- Check http status codes and ensure all 200 status

- Only add https URLs to sitemap and if http, check https for a page to add

- Check box for adding video, images, etc. to sitemap either combined or separate sitemaps

- Build sitemap index for large sites and then link off to each associated locale sitemap

- Option to divide sitemap based on URL patterns (Have a certain directory or part of the site in each sitemap)

- No more than 50,000 URLs per sitemap

Skills: Amazon Web Services, Linux, Python, Software Architecture, Web Scraping

See more: xml feed generator phpnuke, xml file generator, generate xml sitemap script, build script asp will give xml sitemap, sitemapclassphp google xml sitemap, buy sitemap generator php script, google xml sitemap admin cre loaded, xml pdf generator web fop, xml document generator, php web scraping expert needed, google xml sitemap fehler, google xml sitemap, xml sitemap generator, standalone xml sitemap generator page, inurlphp xml sitemap generator, standalone xml sitemap generator, premium xml sitemap generator industrial, xml sitemap wordpress plugin php generator

About the Employer:
( 0 reviews ) Novosibirsk, Russian Federation

Project ID: #20284625

15 freelancers are bidding on average $484 for this job

developer2581

We can take your work for fulltime, as we can provide you with the dedicated react native developer for your project. We are ready to start as soon as we discuss the details. We can provide you with the quality remote More

$500 USD in 7 days
(21 Reviews)
6.5
sky930320

Dear Sir. Glad to meet you. I'm Web developer specializing in web scraping crawling and indexing web pages. Skills: python, scrapy, selenium, requests, beautifulsoup, mechanize, lxml, urllib2, automation, bots, spide More

$500 USD in 7 days
(24 Reviews)
6.3
adampohp79

Hi. Glad to meet you. I'm Web developer specializing in web scraping crawling and indexing web pages. Skills: python, scrapy, selenium, requests, beautifulsoup, mechanize, lxml, urllib2, automation, bots, spiders I wor More

$500 USD in 7 days
(66 Reviews)
6.1
mituld

Hello! I am Scraping Expert for XML Sitemap Generator developing expert and have good experiences with Android, iPhone, Mobile App Development... This project is very interesting for me and perfect fit with my skill More

$400 USD in 15 days
(41 Reviews)
6.2
farooq4161

Greetings, I am an experienced professional scrapper and have done similar projects in the past. Same can be verified from my profile. Let me allow to assist you with your requirements. Thanks

$500 USD in 6 days
(66 Reviews)
6.2
zeke

Dear Customer! I have built many scrapers. This is my favorite type of job. I am absolutely confident I can complete your project easily and on time. Ready to begin immediately and finish as soon as possible. Please co More

$500 USD in 7 days
(59 Reviews)
6.9
hassanalvi95

Hi sir, I am expert in scrapping data and made scrappers for Amazon, eBay, Adidas, and Nike. I made successful scrappers which are running 24/7 and downloading data for my clients. I need some more information from yo More

$500 USD in 7 days
(38 Reviews)
5.8
smsaurabhv

‌Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHO More

$555 USD in 7 days
(73 Reviews)
5.6
coolbratan

Hi, I am 10 years experienced scraping and developing expert. I have done hundreds of similar projects. Accuracy is my main goal! Hire me and you will be satisfied for 100%!

$700 USD in 7 days
(26 Reviews)
5.6
shivansh121

Hi Employer, I can also extract any data from any [login to view URL] I have scraped the data from these websites & stored scraped data into CSVs and excels as per my clients requirement & I have made desktop applicatio More

$261 USD in 6 days
(11 Reviews)
4.1
athminteam

Hi, This is Vipin.I have 6+ years of experience in customized software,web scrapping, web and mobile app development and has expertise in Android, iOS, Python, Magento, PHP, HTML, Java, Angular and Ionic framework. More

$500 USD in 7 days
(7 Reviews)
3.3
micepapai

Hi, I'm a full-stack software developer specialized in Python and PHP for the backend and Javascript for the frontend. Why do you want to use a spider/scraper to generate the sitemap? If it's your site, it should be e More

$350 USD in 3 days
(1 Review)
2.7
techlinesols6

Dear Prospect Hiring Manager. Thank you for giving me a chance to bid on your project. i am a serious bidder here and i have already worked on a similar project before and can deliver as u have mentioned "I can do th More

$500 USD in 7 days
(6 Reviews)
0.0
snaciri81

Hi, I am a data scientist with strong skills in business intelligence, WEB scraping and data analysis tools including Python with Spark and hadoop knowledge. More than 8 years in BI and data analytics and more than 1 More

$500 USD in 7 days
(0 Reviews)
0.0
myappsdevelopers

Hello Project manager I have more than 7 years of experience in Android, Iphone, Mobile Applications,Python , PHP, Ionic ,React native ,Angular JS, WORDPRESS, , HTML5,CSS, Ajax, jquery, JavaScript, Mysql, Graphic Des More

$500 USD in 12 days
(0 Reviews)
0.0