Crawler regex jobs
I'm in need of a web crawler that can extract specific data from a large online ads website. The extracted data should then be neatly exported to a Google Sheet. Key requirements: - The crawler should not be intrusive to the host website. - I want the crawler to crawl current and expired ads based on links present in the site map, as historic data is also of importance to me - You should be able to provide a way for me to run this crawler myself at any time - ideally on a monthly basis. I haven't yet decided on the precise fields to be extracted. I'm looking for someone with the skills to recommend suitable fields that could be beneficial for my project. It's important that the extracted data is organized and easily readable once exporte...
I'm in need of a talented professional who can help me with a custom web crawler project. Key Requirements: - **Web Crawler Customization:** The ideal candidate will need to adapt an existing web crawler code to include specific features I require. This includes the ability to extract and filter data from multiple websites. - **Data Management:** I expect the total scale of the data to be relatively small, so the freelancer should have experience handling data sets of this size. - **Data Analysis:** After the information is extracted, I intend to use it primarily for data analysis. The freelancer should have a good understanding of how to structure data for this purpose. Ideal Skills & Experience: - Proven experience in customizing web crawlers and managing...
...telethon api - **Expected Behavior**: The scraping script is supposed to collect data that matches specific regex patterns. This means that it's not just supposed to collect all messages or only new messages since the last run, but rather those that align with the predefined regex matches. It should give me the option to select which groups get scraped, how far back it crawls back and download/read/regex match/save data. I'm looking for a Python guy who can help me rectify these problems and ensure the script runs efficiently. Experience with Telegram/telethon API, regex, and web scraping is crucial for this task. So if you're a Python developer with previous experience in fixing script errors, particularly in the context of web scraping
We are seeking a skilled developer or development team to create a comprehensive web crawler system. This system should scrape, structure, and update information from web pages, handle large projects, and be accessible via a powerful API. The system should offer user-friendly features for both individuals and businesses and include flexible user management for both customers and employees. Project Requirements 1. Web Crawling and Data Extraction -Develop a system capable of scraping entire websites and extracting data from each page. -The system should divide the scraped information into sections to facilitate easy reading and understanding. -For e-commerce sites, the system should identify and structure the following information: --Product category (including breadcrumbs for hiera...
I'm in need of a proficient coder who specializes in regular expressions (regex). The goal is to extract specific data from a log file. Here are the task details: - Develop a regex expression to parse a log file - Specifically, I need to retrieve software name, version and install date Ideal Skills: - Strong experience with regex expressions - Understanding of log file structures - Attention to detail In your proposal, please include examples of previous similar projects.
I am looking for a Python experienced person capable of creating a web scrapper that can extract text content from Aliexpress and then categorize these extracted data into a Google Sheet in a specific format. Crawler in python format. Thanks. Please provide a sample
I'm looking for a skilled developer who can create a coupon code aggregator website similar to retailmenot.com. The key feature of this project is the integration of a web crawler that will search and gather coupon codes from various online sources. Key Requirements: - Development of a coupon code aggregator website - Integration of a web crawler that will search for coupon codes on a daily basis - A search bar where users can enter websites like they are browsing normally and coupon codes will be applied similar to how microsoft edge or Honey does it - The website should have the functionality to automatically apply coupon codes to a shopping cart - Ability for users to sign up and receive daily coupon codes and deals - Ability for the website admin to be able to add ...
...algorithms Proficiency in data integration and ETL processes Familiarity with MongoDB and data storage + M365 Integration Strong analytical and problem-solving skills Excellent communication and documentation skills Task Breakdown: Develop a fuzzy matching module to compare data across 15-20 points of data Implement additional matching algorithms (e.g., exact matching, regex pattern matching) to handle various data formats Develop 15*3 ETL processes to handle different data formats and request codes Integrate the developed modules with our Azure/M365 environment Configure data storage, authentication, and authorization Perform thorough testing and debugging of the developed modules and ETL processes Deliverables: Fu...
i need a backend preferably in nodejs. one endpoint. that submit some birh date data. then the backend should request the html from anotherwebsite, fill its form click on submit get the data and send them
...- on incoming message the forwarding is done by a configurable delay in milliseconds milestones: MS1: implement a basic hardcoded forwarder which detects new SMS (event) and forwards them via e-mail to a hardcoded e-mail address (android and ios) MS2: design & implement a UI, which has a settings dialog for the rules: - add/delete/update/display (CRUD) operations on rules - user can choose a regex pattern to detect specific phone numbers (from) - user can choose the destination(s) (a list of destinations) with whatsapp, telegram, sms, e-mail - implementation in this MS on the UI only for e-mail (but user can select all the named options) - user can choose to use macros like: - sender phone (smsSenderPhonenumber) - sender name (smsSenderDisplayName) - senders text ...
I think I'm on the right track other times to recognize the patterns of the text which are similar to the elements of a command I would have used regex and other but with the AI I can submit the command to it and it will be able to extract the elements. data that interests me will you be able to support me in this project access to an AI with APIs ? my preference is for a free AI, a free API and with the possibility of sending it a PDF file
...automation to download PDFs (invoices) from different portals. The solution shall be a microservice app. Key Requirements: - OOP based programming - shall be able to login, navigate to the billing section and download the invoices into a individual folder along with details of the download - the download shall be able to work with ...- date periods (fetch invoices: after a date, between dates) ...- regex on filenames ...- downloaded invoice is represented by a invoice class, which properties to represent the full invoice details like (invoice date, invoice number, filename, filepath, sha512 fingerprint ) ...- the fetched data from the website to analyse is mapped into a page model class (page object model) ...- the finally fetched data of the invoices, which
I think I'm on the right track other times to recognize the patterns of the text which are similar to the elements of a command I would have used regex and other but with the AI I can submit the command to it and it will be able to extract the elements. data that interests me will you be able to support me in this project access to an AI with APIs or use a local AI?
I'm looking for an experienced crawler specialist to help me crawl databases, and specifically collect certain datasets. Key Project Requirements: - Crawler Type: The focus for this project will be on databases, not websites or APIs. - Objective: The primary goal is to extract specific datasets from these databases. - Data Format: The extracted data should be delivered in CSV format. Ideal Skills and Experience: - Proficiency in database crawling and data extraction is a must. - Previous experience in collecting specific datasets from databases is highly preferred. - Familiarity with working with CSV data format. Please share relevant experience when bidding.
I'm seeking a meticulous and experienced professional in Octoparse to construct a responsive web crawling template. This crawler will be solely focused on an eCommerce platform, aiming to capture extensive product information. Key Responsibilities: - Capture specifics like the URL, category, and SKU for every product. Ideal Candidate: - Knowledgeable in Octoparse. - Proven experience in data mining and web crawling. - Detailed understanding of eCommerce platforms and product specifications. - Ability to maintain precision across multiple product categories and large datasets. I look forward to seeing your proposals and discussing this project further with you.
I'm looking for an experienced web crawler developer to build a tool for me. This tool needs to be able to extract text content and links from various websites. I need data from the whale watch section of The full crypto address within the links should be associated with their name counter part in a spreadsheet or organized file. I have included a highlted picture demostrating the fields I need. They should be done for the full 6 pages. Requirements: - Experience: Please detail your experience in web crawling and provide examples of previous work. - Delivery: The extracted data should be delivered in a spreadsheet file. It's crucial for the freelancer to have in-depth knowledge of web scraping and data extraction. The tool should be efficient and robust, capable
I am seeking an expert to build a web crawler specifically suited for extracting a broad range of data from social media platforms. The unique objective of this project is to efficiently and accurately amass textual content, images, links and more from social media sources. Specifically, the project will revolve around: - Gathering user posts and related content - Documenting user interactions, encompassing likes, comments, and shares - Extracting comprehensive details from user profile Ideal candidates for this project will have: - Proven experience in Data Extraction and Web Crawling - Knowhow of Social Media platforms and their structure - Proficiency in programming languages used for web crawling - Excellent attention to detail and a commitment to accuracy. If ...
Mail server is getting attacked, I need to create a windows service constantly checking a .txt log file for preset keywords like "Failed SMTP login" or "Relay attempt" to auto block IP's. This service should auto a...good example or a similar concept could be found here: Some of the settings that I need to control: – Duration to ban ip address (min) – Number of failed login attempts before ban – Whitelist of comma separated ip addresses to never ban – Blacklist of comma separated ip addresses to always ban – Custom keywords, XPath and Regex to parse logs for failed login attempts – No need to restart the service when we change settings – Highly configurable, ban anything that comes through a log file.
...(online store mock up) almost done (on my local computer). But few issues I still have with the Contact Us form (email validation. With the header container (the hamburger and logo image touch each other). Rules for the form: 1. All fields must be filled in (validation works) 2. autofill (doesn't work) 3. phone number in 10 digit format (check works) 4. The email input must follow the rules of RegEx for email validation and must be matched with the second email entered (does not work) 5. email again. But email passes without problems in the format sad@df and tyr@df, which is wrong 6. Enter your date of birth. The user must be 18+ (working) 7. Comments: at least 3 characters. 8. Check the box that you agree with our rules, Terms, etc. (works) If you have ti...
I am seeking an expert capable of creating a highly detailed and intricate graphic line art rendition of a lattice boom crawler crane with a luffing jib (two independently controlled booms). This rendition must strictly capture the exterior features of the equipment ( there's a lot). Your focus should also be on all the components that make up the crane. I am open to starting with a more basic line art (outline) and seeing what we need to add. This would be the first design for a construction apparel line You need to have a meticulous eye for detail and at least have a portfolio that showcases any automotive art.
I'm looking for a proficient developer capable of creating a customized web crawler for eBay and AliExpress. The primary function for this crawler will be to accept a list of keywords of any length and return specific data about those products. Here's what I need in detail: • Find top hot selling products at any given time • Crawl eBay and AliExpress for specific product data based on entered keywords. • The data retrieved should include product names, quantities sold and locations of sellers, images, • The crawler should be capable of providing this information for a variable timescale, ranging from 1 to 30 days in the past. • The results should be sortable. Php or python preferably The task would best suit a freelancer wit...
...with Upwork, so this would be phase 1 of our 3 part plan. Once we are through with it and have good experience with the dev, we will be putting up another order in late May or early June. - Should have hashtags support, like in Twitter/Insta, if we type hashtag, it should automatically convert it into a link (I will take care of the backend) - Autocomplete links: Shouldn't be difficult, but using regex or whatever you like, when you see a text URL auto convert it into hyperlink - Block tags: Auto block and remove certain tags, like someone is trying to insert scripts, block 'em. Preferrably Inline editor like Gutenberg. The current setup of everything being at top is a bit old fashioned though completely functional. A block-based system with modern inline editor would b...
...- on incoming message the forwarding is done by a configurable delay in milliseconds milestones: MS1: implement a basic hardcoded forwarder which detects new SMS (event) and forwards them via e-mail to a hardcoded e-mail address (android and ios) MS2: design & implement a UI, which has a settings dialog for the rules: - add/delete/update/display (CRUD) operations on rules - user can choose a regex pattern to detect specific phone numbers (from) - user can choose the destination(s) (a list of destinations) with whatsapp, telegram, sms, e-mail - implementation in this MS on the UI only for e-mail (but user can select all the named options) - user can choose to use macros like: - sender phone (smsSenderPhonenumber) - sender name (smsSenderDisplayName) - senders text ...
I'm looking for a skilled professional who can crop the complete content of a website and save it in a zip file to be deployed on a new server. The website is a small Wordpress with about 41 pages (as per sitemap). Content must not be saved as Wordpress, but static html to be deployed on another server. Key Requirements: - Crop all website content including subpages. - Save the cropped content in a compressed zip file. - The purpose of this cropping is to save the content for future use. - all files must be included, esprecially but not limited to images, js, css, ... - inner paths must be relative paths. If you're confident in your ability to deliver this and can ensure all the content is correctly saved, please bid accordingly. IMPORTANT READ THIS: Auto-generated bids wil...
See details per the word document. Goal is to get a regex that ensures only valid Azure BLOB access urls are returned.
...- on incoming message the forwarding is done by a configurable delay in milliseconds milestones: MS1: implement a basic hardcoded forwarder which detects new SMS (event) and forwards them via e-mail to a hardcoded e-mail address (android and ios) MS2: design & implement a UI, which has a settings dialog for the rules: - add/delete/update/display (CRUD) operations on rules - user can choose a regex pattern to detect specific phone numbers (from) - user can choose the destination(s) (a list of destinations) with whatsapp, telegram, sms, e-mail - implementation in this MS on the UI only for e-mail (but user can select all the named options) - user can choose to use macros like: - sender phone (smsSenderPhonenumber) - sender name (smsSenderDisplayName) - senders text ...
I'm looking for an expert to design a web crawler to extract email addresses from specific company websites. The ideal candidate should be experienced both in web crawling and email data extraction. Key Tasks: - Crawl targeted company websites for email data extraction - Ensure the accuracy and integrity of all extracted emails Skills and Experience: - Experience with web crawling and data extraction technologies - Proven track record in email data extraction - Excellent analytical and problem-solving abilities - Familiarity with common web interrogatories for more efficient data capture is an asset - Strong attention to detail in order to capture complete and valid email addresses The successful candidate will deliver a list of valid emails extracted from the indicated c...
As the project lead, I'm seeking a highly skilled developer to construct a Scrapy web crawler for multiple sites. Although I omitted website specifics for now, rest assured we will provide the necessary details post-engagement. The ideal candidate is capable of: - Building an optimized web crawler with Scrapy - Extracting text data proficiently from various web pages, accurately and efficiently For the application, please provide: - Detailed project proposals outlining your approach Experience in web crawling design and Scrapy framework are a must for this project. Your technical capacity to generate the explained particulars is critical for success here. This is a golden opportunity to exhibit your proficiencies and contribute to an exciting project. - The sites...
I'm seeking an expert in writing Splunk queries to detect threats, efficiently querying from primarily CrowdStrike EDR telemetry data, but also other log sources. Key requirements from your expertise include: - Proven skills in extracting and analyzing data from logs - The ability to use Regex queries to extract additional, nuanced information - Proficient querying from CrowdStrike EDR telemetry. The information to be extracted comprises of EDR fields like: - Process Name - File Hash - Command Line I look forward to working with a detail oriented and security savvy professional who will utilize this data to help us identify potential security threats and ensure system integrity.
...the Saleforce CPQ Quote has it's document generated. I want to add the auto-approval to this flow. 5. In order to use the stuff from (1), Inside the flow, we have to get the approvalRequestId into a variable. This is the part where I got stuck. The article mentions suggests using this method to get it, but there may be others I want to resolve that issue and any others that come up afterward to get it working in Sandbox and then confirm working when moved into Production. To simplify getting this working as proof of concept, let's make a test flow which is triggered off a field edit in SBQQ_Quote. Then to test the flow/Apex, we'll: a. Submit the the Quote into the AP. b. Move it along by manually approving
...create an iconic logo for our American Football League. I attached our current design in the attachements is a football and our state called Baja California Mexico but this design is to simple made. I am in a bit of a rush as it's required ASAP. Here are the details of the project. Logo Words: LIFA BC Competition Logo: ://://www.facebook.com/oefabc/&docid=bW04nRkl5sarEM&w=1631&h=1631&hl=en-US&source=sh/x/im/m1/3&kgs=944815503abde356&shem=abme,trie Key Design Elements: - Main Colors: Blue, Yellow, and White. Certainly, the combination should reflect intensity, energy and teamwork. Ideal Candidate: The perfect freelancer will have a significant
I require an IAM role with specific permissions to interact with s3, glue, and Athena on AWS. This role will allow for efficient management of the AWS stack, specifically enabling the creation of a crawler to populate a glue catalog database and tables, and also allowing interaction with an Athena workgroup. What I need: - Read-only access: This permission should allow the IAM role to view S3 bucket contents, view Glue catalog information, and view Athena workgroup details. - Write access: This level of permission should enable the IAM role to create or update tables in the Glue catalog and modify Athena workgroup settings. Ideal Experience and Skills: - Solid understanding of IAM roles and permissions on AWS - Experience with AWS s3, Glue, and Athena - Capability to work on AWS I...
I need an experienced and proficient website crawler who can help me extract Text content from Database Tables on my website. Key requirements: • Acing in finding, fetching, and structuring data. • Proficient in delivering the retrieved data in Plain text format. • Emphasis on fetching Text content from the Database Table, thus understanding of database structures is a must. Ideal skills include web scraping, data mining, and database understanding. It would be best if you had an eye for detail as accuracy is key in this project. With your expertise, you should be able to navigate through complex website layers to get to the desired content.
We are in need of a knowledgeable and skillful programmer who can edit on GoLang code. The main task involves Ethereum network crawling. We already have a running code to crawling the Ethereum network. However, we still need to do some changes on GoLang code (designing crawler), such as partition the ID_space into sections, and crawl using multiple crawlers in Ethereum not only single crawler as what we have. In addition, there might be more few changes on the code. The ideal candidate will: - Understand DHT-based network and Kademlia - Knows a little about Ethereum (blockchain) - Have experience in GoLang - Possess a deep understanding of DHT network crawling (p2p) - Have previous experience with crawling networks Your responsibility will be building on top of running code...
Google Jobs Scraper project requiring skills in .NET, Python, Selenium and SQL. The task is to create a crawler to extract job listings based on predefined queries, manage the data in a structured manner, and provide an interface for controlling and monitoring the scraping process. The solution requires adept handling of rate limits and the use of rotating proxies for efficient data scraping. Data To Be Scraped: - Scrape all specified job listing data points. - Store and manage the scraped data in a SQL database for effective access and organization.
I'm currently facing issues with my web crawler code. Specifically, it's not managing to scrape all the required data and it's throwing errors inconsistently. This code has been built to scrape information from certain e-commerce platforms. The ideal freelancer for this task should show: - Proficiency in Node.js - Extensive experience with Puppeteer and web crawling, especially focusing on e-commerce platforms - Exceptional debugging skills Your challenge will be to inspect the code, identify the underlying issues and then fix them to ensure a comprehensive and seamless data scraping process. I look forward to collaborating with someone who can effectively resolve the issues and optimize the performance of this web crawler.
I'm looking for a sharp developer highly skilled in C#, specifically with regex, to create validation for a string. This string can be alpha and numeric like "123test123" or "test" but cannot be all numbers like "1209562783" or a date like "12/11/2000". The date must match any date format as well (MM/DD/YYYY and DD/MM/YYYY, including using a "-" or "/" as a separator). This code should be in a method that gets the string passed in, does the above validation and returns a bool value of true is it passes (meaning it is like "123test123" or "test") or false if it fails (matches all numbers or is a date)
your website is causing the load spikes in question. You can consider reviewing this on your side additionally. We have also noticed that there is a lot of bot/crawler access to the website. Access requests originating from bots are registered every 2 to 5 seconds.
Hi, Today i have created a Scrapy crawler to crawl the website , but i'm not able to crawl the website because when i run the crawler, any request i do to the website, i got the 403 error to me. I just neeed you fix my code to do a request at someway the website return the code 200 instead of 403. I'm sending my code attached. Send me a print screen in the chat showing you received code 200 using the same url, then i will hire you to send me your code.
Looking for a seasoned AWS expert to assist in optimizing EC2 instances, S3 storage, and Lambda functions for improved performance. The ideal candidate will have: Mainly working on Aws glue jobs , S3,lambda, Crawler, step functions, dynamo DB ,Clouldwatchs, Appflow, parameter store , API Gateways, ETL Jobs,splunk logs CloudFormation - important big bucket,Spunk, Jenkins pipelines* CI/CD * , YAML for creating templates and manifest files * Sql mandatory for date base * using Postman for API calls * Python and spark Musk know
...thread 7) Add new property 'Regex' for each subreddit added - For example, filter out threads that have a specific word in their title 8) Move from JSON to SQL using () 9) Add discard button for threads so they are no longer visible at all (thread id would be saved in db and never allow that thread to be shown) 10) Archive button for threads so they are no longer visible in the subreddit browser but visible in an Archive section - Save it as property "Archived" - Add Archive section to left navigation bar - Record the comment count as property "CCAA" in db for the thread at the time of archiving 11) Record comment count of thread when it was first loaded - Save it as property "CAFL" in db All values including 'Regex...
...developer to create a specialized web crawler focused on extracting vital data from e-commerce websites, primarily targeting ''. Key Requirements: The web crawler must adeptly parse text-based content and efficiently navigate the site's structure to extract pertinent information and links. Extensive expertise in Web crawling, Web scraping, and Data mining is essential for this project's success. Prior successful engagements in web scraping projects centered on e-commerce platforms would be greatly advantageous. Proficiency in programming Python is required. Familiarity with web scraping frameworks and libraries like BeautifulSoup, Scrapy, or Selenium is highly desirable. While the initial focus is on '', the developed crawler sho...
..."react-json-to-csv": "^1.0.4", "react-querybuilder": "^3.12.1", "react-redux": "6.0.0", "react-responsive": "^8.1.0", "react-router-dom": "^5.2.0", "react-select": "^4.3.1", "redux": "4.0.1", "redux-saga": "1.0.2", "reselect": "4.0.0", "": "8.0.0", "styled-components": "5.0.0", "ua-parser-js": "^0.7.24", "uglifyjs-webpack-plugin": "^2.2.0", "url-regex": "^5.0.0", "xlsx": "^0.16.9" }, "devDependencies": { "@b...
I'm in need of assistance with the creation of 301 redirects on my WordPress website. My project involves: - Managing redirects of moderate complexity, consisting URL structures with subdirectories - The prime objective is to move content to new URLs successfully I am seeking an experienced developer who is well-versed in handling 301 redirects and is experienced in using regular expressions. I'm using htaccess, but I'm open to other methods. The challenge is that there are variations in the original URLs (random characters preceding or appended to the URL). I will provide you with 10 URLs to redirect.
Skills required: Good English, HTML, CSS, JS, JSON, API. 1) Change theme to match from ...controls 4) Add support for reddit hosted videos - no auto-play - with controls 5) Add support for reddit image gallery 6) When a thread isn't loaded widen thread list so it takes up 100% width - When a thread is loaded slide it in from the right to the left - Above Tweet button place X button to close thread 7) Add new property 'Regex' for each subreddit added - For example, filter out threads that have a specific word in their title All values including 'Regex' need to be able to be imported/exported via "Import Data" and "Export Data" button on left bottom navbar Project must remain as a portable html that does not require building or...
...system for users to create new accounts and log in as desired. 2. Chat functionality powered by GPT: A well-equipped Chat-bot that's capable of handling inquiries based on proprietary documents. 3. Comprehensive integration with LLM (GPT): A system that is able to process different types of documents (text, web-pages, PDFs etc.), inferring from both internal and external sources. 4. Web-link Web-Crawler: An engine that can gather data from the web and cross-link it with Azure. 5. Advanced SEO: Optimization techniques to help improve website ranking and visibility. 6. Google Maps integration: An option for website users to connect with Google Maps. 7. Multi-platform accessibility: The site will need to function seamlessly on both PC browsers and mobile devices (Android and iO...
I'm looking for a Google Domains admin to help me fix an issue with page indexing and device permissions for my in-domain email addresses. Tasks: - Analyzing and resolving an issue with page indexing (the Google crawler is complaining about something but I can't tell what) - Setting device permissions for in-domain email addresses (I'm having trouble because users seem to be currently prohibited from logging in on their own devices and I can't find a setting to allow better functionality) Experience and Skills: - Expert knowledge of Google Domains - Proven track record of resolving similar issues Applications: - Only applications showcasing past work in Google domain administration will be considered. Please include the details of previous projects you have ...
I need a seasoned Python expert to assist in the refinement of a script I've developed. The script, currently less than 1 MB in size, needs overhaul in terms of usage, execution speed, and code readability: - Usage: The existing script works, but is full of mistakes. Your primary task will be to modify it to reduce redundancies without compromising the the actual logical str...the the actual logical structure. - Code readability: Also, the script requires restructuring to increase code readability, making it clean and well-organized for easier future revisions. An ideal freelancer for this project should not only have extensive Python programming skills, but also experience in script optimization, code restructuring and performance improvement in Python, focused on Regex an...
