Find Jobs
Hire Freelancers

Big Data Processing AWS EMR or Redshift

$250-750 AUD

Closed
Posted about 8 years ago

$250-750 AUD

Paid on delivery
Hi All, Thanks for taking time to bid on the project. I have large amount of log file data that I need to analyse. This data is stored on AWS S3 in .gz txt files that are tab delimited . It contains the following fields (some optional) TIMESTAMP UID GEO URL CATEGORIES USERAGENT META_KEYWORDS KEY_TERMS ENTITIES Sample file is attached - File sizes are from KB to 10 MB. Requirement: 1: To load and analyse the data (Via EMR or Redshift on AWS), this choice is based on keeping costs lowest. Performance is not the main criteria. 2: Calculate high level metrics (By time period) including: A: Domain Name based counts B: Domain to Key Terms frequency C: Useragent frequencies D: Entities Frequencies E: Categories Frequencies F: List of Domains based on Categories G: list of Domains based on Key Terms Please ask questions before you bid not after. I am open to suggestions. Regards Happy Bidding
Project ID: 9406922

About the project

9 proposals
Remote project
Active 8 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
9 freelancers are bidding on average $827 AUD for this job
User Avatar
Hi. How are you? what need you do with this data? maybe i can put on topics to apache kafa (a queue services with data persistence) and make micro services to route to destiny of data. Is ok?
$1,111 AUD in 5 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Hello! Can do this task for you very quickly. Have experience using Amazon EMR in old project. I have wide experience in writing utilities on C++/C#/Python/R/PHP (including client-servers scripts, web scraping, working with databases, monitoring and control systems, and so on). May start right now. Almost always online, waiting for your answer Thank you.
$650 AUD in 5 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Hi, I have some questions regarding the timeframe and others for this project. Although you've mentioned that performance is no the main criteria, what's your worst case scenario in terms of time for analysis of a 10 MB file and what would be the instance specifications on AWS or Redshift that we'd be working on ?
$700 AUD in 7 days
0.0 (0 reviews)
0.0
0.0
User Avatar
we have a skilled team of machine learning and data mining experts. we have completed several project involving clustering, feature space reduction using algorithms like PCA and data analysis using python, R and Matlab. Our team can help you with this project. Please share more details so we can talk further. final offer and timeline will be decided after discussing the details.
$1,000 AUD in 10 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Hi Team, I am having 4+ years of experience in data analytic and served 15+ clients. As a suggestion : This work could be done using Elasticsearch / Logstash and Kibana. Where reports and dashboard can be generated using Kibana for the mentioned requirement as below : 1: To load and analyse the data (Via EMR or Redshift on AWS), this choice is based on keeping costs lowest. Performance is not the main criteria. : I would suggest to use ELK stack nothing but Elasticsearch , Logstash and Kibana which is open source and can be integrated on AWS 2: Calculate high level metrics (By time period) including: Graph can be plotted to demonstrate the same (for all below metrics). A: Domain Name based counts B: Domain to Key Terms frequency C: Useragent frequencies D: Entities Frequencies E: Categories Frequencies F: List of Domains based on Categories G: list of Domains based on Key Terms Let me know if we can discuss for the same and start ASAP. Also if you want a demo just give me few data say 100 entries , I will do it manually in my environment and come up with a small demo. (One portfolio is attached in my profile as well which is having analysis of my Gmail Data) If you are thinking I do not have any experience on Freelancing or projects so i would suggest you to check my Upwork profile for the work i have done and my portfolio as well, As started bidding on freelancing recently so no portfolio as such.
$727 AUD in 10 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of AUSTRALIA
Australia
4.9
67
Payment method verified
Member since Sep 3, 2003

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.