Data Engineer

• 5+ Yrs
• Kolkata
• 22 Oct 2021
• Job Description
• • Extensive experience in Data Engineering, designing and implementing complex large-scale Data Pipelines for processing data from various data sources.
  • Experience in designing, implementing, and optimizing large-scale data pipelines in production tools and technologies listed in the technology stack on the cloud, specifically AWS.
  • Experience in implementing Data Lake on the cloud using Databricks Delta Lake
  • Experience in implementing Data Pipelines using the following technology stack:
    1. Apache Airflow
    2. Apache Spark
    3. AWS S3
    4. Parquet
    5. Delta Lake
  • Process Logging, Error detection, log monitoring, and batch error recovery
  • Expert level experience in Data Pipeline Orchestration using Apache Airflow for large scale production deployment
  • Experience in visualizing data from various tasks in the data pipeline using Apache Zeppelin/Plotly or any other visualization library.
  • Implementation of custom Airflow Operators, Hooks, and Sensors
  • Airflow log management and log monitoring using ELK/Grafana
  • Scalable Elastic deployment architecture and solution for Apache Airflow on the Kubernetes cluster
  • Containerization of DAGs
  • Git Hub Integration
• Desired Candidate Profile

• Five (5+) years of experience in Data Engineering

• Key Skills
• • Apache Spark
  • Spark SQL
  • SparkR
  • Scala
  • Apache Athena
  • Apache Airflow
  • Python
  • PySpark
  • Apache Zeppelin
  • Plotly
  • AWS EMR
  • EC2
  • S3
  • Kubernetes
  • ELK
  • Grafana
  • HDFS/Apache
  • Arrow/Parquet
  • Java
• Salary
• Based on experience and ability/potential
• Industry
• Information technology
• Functional Area
• Data Engineer
• Role
• Data Engineer
• Employment Type
• Full time