PySpark Big Data Developer

Citi — Pune Maharashtra India

We are seeking a highly skilled and experienced BigData/PySpark Engineer to join our dynamic Big Data Analytics team. This role is pivotal in designing, developing, and optimizing robust, scalable data pipelines for large-scale data processing and analytics. Key Responsibilities: Design & Development: Create and optimize scalable ETL (Extraction, Transformation, Loading) pipelines using PySpark for massive datasets. Coding & Engineering: Write clean, efficient, well-documented code primarily in Python (PySpark) often leveraging frameworks/tools. Collaboration: Work with cross-functional teams (senior developers, data engineers, analysts, business partners) to understand data requirements and ensure seamless solution integration. Troubleshooting & Optimization: Debug and resolve data processing issues and performance bottlenecks in Spark applications and other big data technologies. Full SDLC Involvement: Participate in the entire software development lifecycle, from requirements analysis and design to testing, deployment, and operations. Data Integrity: Ensure high data quality and integrity throughout the data lifecycle. This candidate possesses 4-8 years of experience in developing and managing Enterprise Applications, demonstrating a robust foundation in Big Data technologies and a strong grasp of software development principles. Key Experience & Expertise: Enterprise Application Development: 4-8 years in developing and managing enterprise-grade applications. Object-Oriented Programming (OOP): Solid foundation in OOP concepts. Big Data Development: Expertise in PySpark, HDFS, Hive, Sqoop, and Hadoop for Big Data environments. Database Technologies: Good exposure to SQL Server and ORACLE databases. Experience with query writing for data validation/manipulation Scripting & Automation: Proficient in Shell Scripting and experience with job scheduling tools like Autosys . BI Reporting Tools: Some exposure to BI tools, specifically Tableau . Tools & Practices: Proficie

View on HireSetu