Find Jobs
Hire Freelancers

Show me how to set up and run a simplified streaming pipeline on AWS - Spark, Kafka, Cassandra ($50-$100)

$50-100 USD

In Progress
Posted about 5 years ago

$50-100 USD

Paid on delivery
Beginning Big Data developer here who knows a little AWS, Hadoop, Spark, Python and Java looking for someone who can take me through the process of setting up a Spark + Kafka + Cassandra (or maybe MongoDB) streaming pipeline on AWS with multiple nodes (say, around 5 nodes, with EMR or EC2 or whatever makes the most sense). This is for getting myself started with setting up and running a semi--semi-realistic streaming pipeline to play around with for my self-education purposes, not for production, so a very basic "hello world" type setup doing the basics is what I am looking for. In other words, Kafka plus Spark streaming to process the input and put it into db, without anything too fancy in addition besides some trivial analytics or computations. You can make up your own problem to solve or re-use something you've already done! Success would be giving me reproducible instructions and very basic code (Java, Python or Scala) which I can take and successfully run myself. I don't have any hard requirements for Kafka and Spark redundancy, but it would be good to know what to do if I want to scale this out more. Pipeline automation like Airflow or NiFi would be a nice bonus but not necessary. Try to give me something that should be simple enough for a software engineer with very little background info to do without requiring more than 2 or 3 emails to get clarifications. Or maybe I'm asking too much for a beginner to take on? If so, let me know something else similar but more reasonable! But it must be a multi-node streaming config ... I can probably do anything simpler on my own. I'm hoping the cost I'm offering will cover the AWS time setting this up and verifying it, but I can be flexible. (Just running the pipeline for a few minutes and verifying it works is good enough). If the consensus is that the fee is too low then I will certainly accept something higher. If we get into debugging the setup, I will pay any additional AWS fees necessary you may incur. Let me know what you think you do!
Project ID: 19082932

About the project

7 proposals
Remote project
Active 5 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
Awarded to:
User Avatar
Hi Sir, I have 5 years + experience in spark kafka hadoop yarn and in other big data technologies. I have done setups for the many enterprises. I can assure you the work and learning as well.
$110 USD in 10 days
0.0 (0 reviews)
0.0
0.0
7 freelancers are bidding on average $177 USD for this job
User Avatar
Hi there, I have read your project description and i'm confident i can do this project for you perfectly.I still have a few questions. please leave a message on my chat so we can discuss the budget and deadline of the project. Thanks . .
$85 USD in 3 days
4.8 (25 reviews)
6.2
6.2
User Avatar
do kindly share the requirements in a zip file. you can reach out to me over chat to discuss further
$166 USD in 5 days
4.8 (26 reviews)
4.5
4.5
User Avatar
Hi, Coincidentally I worked on a similar streaming pipeline for a project that had TBs of data flowing through multiple asynchronous jobs. Would love to talk more about it and also discuss more about your requirements. Thanks, Sajid
$111 USD in 2 days
5.0 (2 reviews)
1.6
1.6
User Avatar
I am working as a data engineer and my current project is creating real time recommendation pipeline with the same technologies you want to learn.
$130 USD in 8 days
0.0 (0 reviews)
0.0
0.0
User Avatar
i am a hadoop bigdata cloud expert..
$83 USD in 10 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of UNITED STATES
Scottsdale, United States
0.0
0
Payment method verified
Member since Mar 29, 2019

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.