We are seeking a skilled Hadoop Developer to join our data engineering team. The ideal candidate will have experience in building, managing, and optimizing large-scale Hadoop clusters. You will work on designing and developing solutions that allow for efficient data processing, transformation, and storage. This role requires a deep understanding of Hadoop ecosystem components and their integration with other data processing tools.
Key Responsibilities:
- Hadoop Development : Design, develop, and maintain Hadoop applications and solutions using various tools within the Hadoop ecosystem, such as HDFS, MapReduce, Pig, Hive, HBase, and others.
- Data Processing : Build and optimize data processing pipelines using Hadoop technologies to handle large volumes of structured and unstructured data.
- Data Integration : Integrate Hadoop applications with other data sources, databases, and data warehouses.
- Performance Optimization : Optimize Hadoop clusters for performance, reliability, and scalability, including tuning MapReduce jobs, HDFS, and other Hadoop components.
- Data Security : Implement data security and encryption within Hadoop clusters to ensure data protection and compliance with regulatory requirements.
- Collaboration : Work closely with data scientists, analysts, and other developers to understand data requirements and provide appropriate Hadoop-based solutions.
- Monitoring and Troubleshooting : Monitor Hadoop clusters and applications, troubleshoot issues, and ensure high availability and reliability of the system.
- Documentation : Create and maintain technical documentation related to Hadoop processes, systems architecture, and workflows.
Required Skills and Qualifications:
Education : Bachelor’s or master’s degree in computer science, Information Technology, Engineering, or a related field.
Experience :
- Minimum of 5 years of experience in Hadoop development.
- Hands-on experience with Hadoop components such as HDFS, MapReduce, Hive, Pig, HBase, and YARN.
- Experience with data integration tools like Sqoop, Flume, or Kafka.
- Strong experience with scripting languages like Python, Perl, or Shell scripting.
- Proficiency in SQL and NoSQL databases.
#J-18808-Ljbffr