Senior data engineer

Pune 5-7 year Day On-site
Create job alert

Job Description

Our company is looking to fill the role of a senior data engineer. To join our growing team, please review the list of responsibilities and qualifications.

Skills:

    • A Bachelor’s degree and a minimum of 5 years relevant experience as a data engineer.
    • Hands-on deployment experience with Hadoop/Spark, Scala, MySQL, Redshift, and Amazon AWS or other cloud base systems.
    • Comfortable writing code in python, ruby, perl, or equivalent scripting language.
    • Experience with Cosmos/Scope, SQL, or Hadoop.
    • At least 3 years of professional work experience programming in Python, Java or Scala.
    • 2+ years of Distributed Computing frameworks such as Apache Spark, Hadoop.

Responsibilities:

    • Design and develop ETL (extract-transform-load) processes to validate and transform data, calculate metrics and attributes, and populate data models, using HADOOP, Spark, SQL, and other technologies.
    • Lead by example, demonstrating best practices for code development and optimization, unit testing, CI/CD, performance testing, capacity planning, documentation, monitoring, alerting, and incident response in order to ensure data availability, data quality, usability and required performance.
    • Use programming languages such as SAS, R, Python, and SQL to create automated processes for data gathering, cleansing, reporting, and visualization.
    • Implement systems for tracking data quality, usage, and consistency.
    • Design and develop new data products, using languages.
    • Monitor and maintain system health and security.
    • Oversee administration and improvements to source control and deployment process.
    • Prepare unit tests for all work to be released to our live environment (including data validation scripts for data sets releases or changes).
    • Implement performance tuning on the databases based on monitoring.
    • Design and implement data products using Hadoop technologies.
    • Clear documentation of process flow diagrams and best practices.

Qualification

    • BS or MS in Computer Science, Computer Engineering, Data Science, or related discipline