AWS - Infrastructure as Code for Dockerized Apache Spark

I am looking for a freelancer who is experienced in AWS and Infrastructure as Code to help me with my project. Project Overview: - I currently use Aurora, S3, EKS, Fargate, Spark, and Docker on AWS. - I have an existing Dockerized Apache Spark application. - My preferred Infrastructure as Code (IaC) tool is Terraform. Skills and Experience Required: - Strong knowledge and experience with AWS services, specifically Aurora, S3, EKS, Fargate, Spark, and Docker. - Proficiency in Terraform for managing AWS infrastructure. - Experience working with Dockerized Apache Spark applications. - Understanding of Infrastructure as Code principles and best practices. Project Deliverables: - Create and manage infrastructure on AWS using Terraform. - Deploy and maintain a Dockerized Apache Spark application. - Ensure high availability, scalability, and security of the infrastructure. - Implement best practices for Infrastructure as Code. If you have the necessary skills and experience, please submit your proposal. **AWS Job Requirements**: 1. **Dockerization**: - Collaborate with our development team to optimize and finalize the Dockerfiles for both the Apache Spark application and the NGINX web application. - Implement a CI/CD pipeline using AWS CodePipeline and CodeBuild. The pipeline should handle building, testing (unit tests, integration tests, security scans), and pushing Docker images to Amazon Elastic Container Registry (ECR). 2. **Infrastructure as Code (IaC) Setup**: - Provide IaC scripts using a specified version of AWS CloudFormation or Terraform. - Define consistent naming conventions, tagging standards, and resource grouping across all AWS resources. - Ensure version control for IaC scripts and support for rollbacks. 3. **EKS + Fargate Deployment for Apache Spark**: - Deploy a scalable EKS cluster adjustable for different environments (development, staging, production). - Implement automated scaling policies using defined metrics such as CPU and memory utilization. - Ensure data persistence for the Spark application using appropriate AWS storage solutions. 4. **Aurora Database Deployment**: - Deploy an Amazon Aurora instance suitable for the application's workload. Configure automatic backups, replication, and scaling as necessary. - Ensure the Aurora instance meets all security best practices, including VPC isolation, encryption at rest, and in transit. 5. **NGINX Web Application Deployment with CI/CD**: - Decide on either: a. Deploying the Dockerized NGINX web application in EKS using Fargate for scalability, integrating seamlessly with the Aurora database and other application components. b. Deploying the NGINX web application on EC2 instances managed by Auto Scaling Groups to ensure scalability and fault tolerance. - Use AWS CodePipeline and CodeBuild (or an equivalent CI/CD tool) to automate the build, test, and deployment processes of the NGINX web application. - Implement security best practices including SSL/TLS termination, DDoS protection, and WAF integration. 6. **Monitoring, Logging & Alerting**: - Set up AWS CloudWatch dashboards to monitor infrastructure (EKS, Fargate, EC2, Aurora) and applications (Spark, NGINX). - Implement centralized logging using services like CloudWatch Logs and ensure long-term archiving with services like AWS S3. - Design multi-channel alerting mechanisms (email, SMS, etc.) and set clear response time expectations for each alert type. 7. **Documentation & Handover**: - Provide comprehensive documentation in a standard format (e.g., Markdown) detailing architecture, setup processes, troubleshooting steps, and disaster recovery. - Conduct handover and training sessions, ensuring they are recorded for future reference. Plan follow-up Q&A sessions. 8. **Compliance & Performance**: - Ensure the entire setup adheres to relevant compliance standards or industry guidelines. - Establish, monitor, and maintain SLAs or performance benchmarks for both infrastructure and applications.