Job Responsibilities:
• Apply DevSecOps to develop state-of-the-art tools and frameworks to automate infrastructure, security components, build scalable and efficient pipelines for data processing, dataset creation, analytical workloads
• Implement automated Big Data services/solutions such as AWS EMR, Apache Spark, Databricks, AWS Redshift, Snowflake, S3, GCP Big Query, Google Cloud Storage and Google Data Proc and general cloud services such as Lambda, SNS, EC2, Airflow, Elastic Cache, Redis, RESTful APIs
• Apply DevSecOps & Agile approaches to deliver holistic and integrated solution to automate infrastructure, security components and Continuous Integration & Continuous delivery for optimal execution of data pipelines (ELT/ETL)
• Develop DevOps pipelines involving Data & Engineering version control, automation of continuous integration & deployment, configuration management, infrastructure orchestration using Jenkins, Ansible, Chef, XL Release and XL Deploy
• Develop CloudOps & Infrastructure solutions to build resiliency in data pipelines with platform health checks, monitoring, and alerting mechanisms so quality, timeliness, recency and accuracy of data delivery are improved. Review system issues, incidents, and alerts to identify root causes and implement features to continuously improve platform performance
• Design & Develop CloudOps, infrastructure automation (Terraform, Cloud Formation, Docker), provisioning cloud services, supporting feature engineering with ETL/ELT pipelines and programming in SQL, Python, Java, Scala-Spark, Bash and other scripting languages
• Liaison and Collaborate with enterprise security, digital engineering, and cloud operations to gain consensus on architecture, solution frameworks
Experience Required: 5-8 Years
