what I need is the following:
- I need a helm script (kubernetes) which "installs" the latest spark version as yarn cluster on kubernetes
- This setup should provide Zeppelin
- All of this has to work with sparks latest structured streaming framework and should be able subscribe a kafka topic
- Confluent is used as kafka provider (and it's security mechanism - [login to view URL] = [login to view URL])
- Zeppelin has to deploy jobs to cluster and NOT execute them standalone
- The kubernetes cluster doesn't use level 4 load balancer, so we can't simply use the following helm chart ([login to view URL])
- Kubernetes Level 7 Loadbalancers are used
- Basic neccessary setup (Nodes needs to have the corresponding spark versions deployed)
- Acceptance criteria: I have a structured streaming script, which we can use to check if setup works, in the meantime you can use your script for development.
- Kubernetes Version is v1.16.3.
- I can't grant you access to the cluster nor to the confluent stream.
Thanks for your time.