[EN] Building a CI/CD pipeline for a Spark project using Github Actions, SBT and AWS S3 — Part 2

Wed, 29 Apr 2020 13:01:24 +0200

In the first article of this series, we talked about how we can set up a CI/CD pipeline for a Spark project using Github Actions, SBT as a build tool and S3 for deployment. Our code once pushed to the [master] branch of our project on Github, triggered an SBT Build command to generate a fat jar, then pushed it to S3 to the chosen bucket.

However, this pipeline still lacks a way to add a logic since it does not allow us to check whether the jar’s version we’re putting to S3 already exists for instance.

Github on Ayoub Fakir

[EN] Building a CI/CD pipeline for a Spark project using Github Actions, SBT and AWS S3 — Part 2