Big Data - Spark & Scala Developer

Ahmedabad   |  Job Code : AHRS643

Job Description

  • In depth understanding/knowledge of Hadoop & spark Architecture and its components such as HDFS, Job Tracker, Task Tracker,executor cores and memory params.
  • Experience in Hadoop development, working experience on SPARK, SCALA is mandatory and Database exposure is must
  • Hands on experience in Spark and Spark Streaming creating RDD's, applying operations -Transformation and Actions.
  • Experience in code optimize to fine tune the applications.
  • Expertise in writing Hadoop/spark Jobs for analyzing data using Spark, Scala, Hive, Kafka and Python
  • Experienced with streaming work flow operations.
  • Experience with developing large-scale distributed applications and developing solutions to analyze large data sets efficiently.
  • Integration with Hadoop/HDFS, Real-Time Systems, Data Warehouses, and Analytics solutions. Experience in Data Warehousing and ETL processes.
  • Strong database, SQL, ETL and data analysis skills.
  • Knowledge in Maintaining the Log and Audit information in Hive/SQL tables, Experienced in providing Logging, Error Handling
  • Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
  • Configured Spark Streaming to receive real time data from the Apache Kafka and store the stream data to HDFS using Scala.
  • Experienced Scheduling jobs using Airflow/ESP.
  • Experienced in developing scripts for doing transformations using Scala.
  • Involved in developing Shell scripts to orchestrate execution of all other scripts and move the data files within and outside of HDFS.
  • Using Kafka on publish-subscribe messaging as a distributed commit log, have experienced in its fast, scalable and durability.
  • Implemented Kerberos Security Authentication protocol for existing cluster
  • To be able to understand existing business logic & implement the business changes
  • Working with business and peers to define, estimate, and deliver functionality.
  • Should have gone through entire development life cycle including system integration testing
  • Provide Production Support for any issues raised by the Business and experience on Production/Integration release management
  • Experience with GIT, Jenkins, test frameworks.
Soft Skills:
  • Right communication skills to be able to work with Business.
  • Good logical reasoning, analytical abilities.Recommend changes to client software based on information such as client feedback, language changes and new technology.
  • Experience working with remote teams.
  • Work to become the trusted partner of the Architects by regular engagement and proposals on best development practices which can be implemented and would bring change to the process
  • Excellent Performance and Technical Skills, Attend required training and support peers by sharing knowledge and mentoring.
  • Taking up training programs within your team to ensure understanding of BDF.


Agile skills:

  • Experienced in Agile development methodology.
  • Create the required technical tasks in backlog and update status regularly.
  • Participate in all Agile ceremony meetings and update task status and adher to timelines and quality standards for the task.
  • Hadoop/Big Data
  • HDFS
  • Scala
  • Spark
  • Hive
  • Kafka
  • Shell Scripting
  • Java experience is a plus
  • GIT Version control tool knowledge,Jenkins and Scheduling tools Airflow & ESP
Job Opportunities

Apply For Your Dream Job