PySpark Developer

This role is for one of Weekday’s clients

Salary range: Rs 2000000 - Rs 3000000 (ie INR 20 - 30 LPA)

Min Experience: 8+ years

• Location: Bengaluru, Chennai, Tamil Nadu, India
• JobType: full-time
• We are looking for an experienced
• PySpark Developer with 8–13 years of experience in data engineering, Python programming, and large-scale data processing. The ideal candidate will have strong expertise in developing, optimizing, and maintaining scalable data pipelines using PySpark and Apache Spark. This role requires hands-on experience in processing structured and semi-structured data, implementing ETL/ELT workflows, and building reliable data solutions that support business intelligence, analytics, and data-driven decision-making.
• The candidate will collaborate with data engineers, architects, analysts, and other technical stakeholders to design efficient data processing solutions, improve pipeline performance, and ensure data quality across the organization.

Requirements

• Key Responsibilities
• 1. Data Pipeline Development
• Design, develop, and maintain scalable, high-performance data pipelines using PySpark and Python.
• Build robust ETL/ELT workflows to ingest, transform, validate, and load large volumes of data from multiple sources.
• Develop batch and, where required, streaming data processing solutions to support business and analytical requirements.
• Implement reusable data processing frameworks and ensure pipelines are reliable, maintainable, and scalable.
• Integrate data from databases, APIs, cloud storage, and enterprise data platforms.

2. PySpark Development and Optimization

• Develop complex data transformations using PySpark DataFrames, Spark SQL, and distributed processing techniques.
• Optimize Spark jobs by analyzing execution plans, partitioning strategies, caching, and resource utilization.
• Troubleshoot performance bottlenecks, resolve processing failures, and improve job execution efficiency.
• Work with large datasets while ensuring efficient memory management and distributed computing performance.
• Apply best practices for handling joins, aggregations, data skew, and incremental data processing.

3. Python Programming

• Write clean, modular, reusable, and well-documented Python code for data engineering applications.
• Develop utilities, validation scripts, and automation solutions to streamline data workflows.
• Implement exception handling, logging, testing, and debugging practices to improve code quality.
• Collaborate with engineering teams to maintain coding standards and version control practices.

4. Data Quality and Reliability

• Implement data validation, reconciliation, and monitoring mechanisms across pipelines.
• Identify and resolve data inconsistencies, duplication, and transformation errors.
• Maintain technical documentation, data lineage, and operational procedures.
• Support production deployments, incident resolution, and continuous improvement initiatives.

5. Collaboration and Technical Leadership

• Work closely with business stakeholders and technical teams to translate requirements into scalable data solutions.
• Participate in architecture discussions, code reviews, technical design sessions, and development planning.
• Mentor junior developers and contribute to engineering best practices.

Required Skills and Qualifications

• 8–13 years of overall experience in software development, data engineering, or related technical roles.
• Strong hands-on expertise in PySpark, Apache Spark, and Python
• Proven experience designing and maintaining scalable data pipelines and ETL/ELT processes.
• Strong understanding of distributed computing, data partitioning, transformations, and performance optimization.
• Experience working with SQL, relational databases, and large-scale structured or semi-structured datasets.
• Familiarity with data warehousing concepts, data modeling, and data quality practices.
• Experience with Git, debugging, testing, and production support.

Preferred Qualifications

• Experience with cloud data platforms such as AWS, Microsoft Azure, or Google Cloud.
• Exposure to platforms such as Databricks, Delta Lake, Snowflake, or cloud-based data lakes.
• Familiarity with orchestration tools such as Apache Airflow and CI/CD practices.
• Knowledge of Kafka or other streaming technologies is an advantage.