CrawlJobs Logo

Pyspark Module Lead

https://www.soprasteria.com Logo

Sopra Steria

Location Icon

Location:
India , Noida

Category Icon

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

Not provided

Job Description:

We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. As a Data Engineer, you will collaborate closely with our Data Scientists to develop and deploy machine learning models. Responsibilities include working with PySpark, AWS EMR, and S3 for data processing, designing machine learning pipelines, optimizing pipelines for performance, and managing ETL workflows using Streamsets.

Job Responsibility:

  • Work in tandem with Data Scientists to design, develop, and implement machine learning pipelines
  • Utilize PySpark for data processing, transformation, and preparation for model training
  • Leverage AWS EMR and S3 for scalable and efficient data storage and processing
  • Implement and manage ETL workflows using Streamsets for data ingestion and transformation
  • Design and construct pipelines to deliver high-quality training and inference datasets
  • Collaborate with cross-functional teams to ensure smooth deployment and real-time/near real-time inferencing capabilities
  • Optimize and fine-tune pipelines for performance, scalability, and reliability
  • Ensure IAM policies and permissions are appropriately configured for secure data access and management
  • Implement Spark architecture and optimize Spark jobs for scalable data processing

Requirements:

  • Proficiency in Advanced SQL (Window functions), Spark Architecture, Pyspark or Scala with Spark, Hadoop
  • Proven expertise in designing and deploying data pipelines
  • Strong problem-solving skills and ability to work effectively in a collaborative team environment
  • Excellent communication skills and ability to translate technical concepts to non-technical stakeholders

Nice to have:

  • Hands-on experience with Airflow, S3, and Stream sets or similar ETL tools
  • Understanding of real-time or near real-time inferencing architectures
  • Basic knowledge on Kafka, AWS IAM, AWS EMR and Snowflake
What we offer:
  • All positions are open to people with disabilities
  • Commitment to fighting against all forms of discrimination
  • Inclusive and respectful work environment

Additional Information:

Job Posted:
April 26, 2025

Job Link Share:

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Pyspark Module Lead

Pyspark Module Lead

Sopra Steria is seeking a PySpark Module Lead based in Bengaluru, India. The pos...
Location
Location
India , Bengaluru
Salary
Salary:
Not provided
https://www.soprasteria.com Logo
Sopra Steria
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Python
  • PySpark
  • SQL
  • JavaScript
  • Proficient with Skywise platform and tools e.g. Contour, Code-Workbook, Code, Slate and Ontology
Job Responsibility
Job Responsibility
  • Designing and building solutions for Airbus on foundry
  • Data engineering and integration
  • Data analysis and visualization
  • Application development within Skywise Framework
  • Working closely with Product Owners and core Airbus business functions to understand processes, capture requirements, and solve business problems
  • Full delivery lifecycle for designing, implementing, testing, documenting, and supporting applications
  • CI/CD pipeline development and automated deployments
  • Troubleshooting application issues
  • Providing support on developed tools
What we offer
What we offer
  • Inclusive and respectful work environment
  • Open to people with disabilities
  • Fulltime
Read More
Arrow Right

PySpark Module Lead

PySpark Module Lead position at Sopra Steria, a major Tech player in Europe. The...
Location
Location
India , Bengaluru
Salary
Salary:
Not provided
https://www.soprasteria.com Logo
Sopra Steria
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Good experience in applying best coding practices
  • Good understanding and ability to develop simple to medium applications
  • Strong learnability and analytical skills
  • Strong team player with excellent communication skills
  • Mandatory skills: AWS, PySpark, PL SQL, Java
  • AWS Skills are on S3, Lambda, Cloudwatch, Dynamo DB, SQS, SNS, CloudFormation, Stack, Serverless
  • Total Experience Expected: 04-06 years
  • B.Tech, MCA or equivalent degree
What we offer
What we offer
  • Committed to fighting against all forms of discrimination
  • Fosters a work environment that is inclusive and respectful of all differences
  • All positions are open to people with disabilities
Read More
Arrow Right

Senior Software Engineer

The Senior Software Engineer will lead a team of data engineers to design, build...
Location
Location
United States , San Diego
Salary
Salary:
Not provided
gomillenniumsoft.com Logo
MillenniumSoft Inc
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10+ years of experience in developing data solutions
  • Proven experience in designing and implementing large-scale data pipelines
  • Hands-on expertise with AWS cloud technologies and distributed systems
  • Strong programming skills in PySpark, Python, and SQL
  • Data pipeline development using AWS tools (Step Functions, Airflow, etc.)
  • Distributed systems (S3, ADLS), and modern file/table formats (Parquet, Hudi)
  • ETL tools, orchestration technologies, and database design
  • Knowledge of regulatory standards like SOX, PCI, and HIPAA
  • Bachelor’s or Master’s in Computer Science, Information Systems, or related fields
  • AWS certification preferred
Job Responsibility
Job Responsibility
  • Design and implement scalable data pipelines for structured and unstructured data
  • Lead module architecture design for optimal performance, reliability, and maintainability
  • Mentor team members and ensure alignment with business goals
  • Collaborate with cross-functional teams to enable real-time and batch data processing
  • Ensure adherence to data security, privacy, and governance standards
  • Create and maintain technical documentation for architectures and decisions
  • Conduct code reviews and establish best practices
  • Develop monitoring systems for data pipeline reliability
  • Drive adoption of new technologies and workflows
  • Fulltime
Read More
Arrow Right
New

Pharmacy Intern

You’ve invested a lot of time and energy in your education. Now you want the cha...
Location
Location
United States , Wichita
Salary
Salary:
18.00 - 19.75 USD / Hour
https://www.cvshealth.com/ Logo
CVS Health
Expiration Date
March 01, 2026
Flip Icon
Requirements
Requirements
  • Accepted into, or actively enrolled in, an ACPE accredited college or school of pharmacy
  • Must possess, or be in the process of obtaining, valid intern and/or technician licensure as required
  • Regular and predictable attendance, including nights and weekends
  • Ability to complete required training within designated timeframe
  • Attention and Focus: Ability to concentrate on a task over a period of time
  • Ability to pivot quickly from one task to another to meet patient and business needs
  • Ability to confirm prescription information and label accuracy, ensuring patient safety
  • Customer Service and Team Orientation: Actively look for ways to help people, and do so in a friendly manner
  • Notice and understand patients’ reactions, and respond appropriately
  • Communication Skills: Use and understand verbal and written communication to interact with patients and colleagues
Job Responsibility
Job Responsibility
  • Living our purpose by following all company SOPs at each workstation to help our Pharmacists and Technicians manage and improve patient health
  • Following pharmacy workflow procedures at each pharmacy workstation (i.e., production, pick-up, drive-thru, and drop-off) for safe and accurate prescription fulfillment
  • Contributing to positive patient experiences showing empathy and genuine care: creating heartfelt and personalized moments while serving patients at pick-up, drive-thru, and over the phone
  • keeping patients healthy by offering immunizations and other services at the register and over the phone
  • and demonstrating compassionate care by solving or escalating patient problems
  • Offering to counsel, fielding medical questions, and soliciting information on a patient’s medical history to provide optimal care, when appropriate under the direct supervision of a licensed pharmacist
  • Taking telephonic prescriptions from the prescriber, and calling the prescriber to clarify prescriptions or facilitate medication changes, where allowed by state regulation
  • Maintaining the highest level of self-awareness and providing in-the-moment coaching, training, and mentoring to pharmacy team members while sharing best practices
  • Completing basic inventory activities, as permitted by law, and as directed by the pharmacy leadership team, such as accurately putting away medication deliveries and completing cycle counts, returns-to-stocks, waiting bin inventories, etc.
  • Contributing to a high-performing team, embracing a growth mindset, and being receptive to feedback
Read More
Arrow Right
New

Energy Centre Technician

D200 Energy Centre is a Boots owned, purpose-built Combined Heat and Power (CHP)...
Location
Location
United Kingdom , Nottingham
Salary
Salary:
Not provided
boots.com Logo
Boots
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Time-served electrical apprenticeship
  • C&G Level 2 and 3 Diploma Electrical Installation (or equivalent)
  • 18th edition
  • Significant mechanical experience
  • Knowledge and understanding of High Voltage systems
  • Understanding and appreciation of industrial processes
  • Understanding and appreciation of steam systems
  • Knowledge and understanding of Health & Safety obligations in an engineering environment and safe systems of work
  • Must be prepared to work out of hours and shifts when required
  • Good communication and organisation skills
Job Responsibility
Job Responsibility
  • Carry out planned and reactive maintenance activities within D200 and on the Boots Beeston site whilst following safe systems of work
  • Responsible for the management of statutory maintenance of the Beeston site High Voltage network, carrying out AP/SAP duties
  • Use expertise, experience & professionalism to make strong well-informed decisions to prioritise tasks to ensure safe, reliable and efficient supply of utilities across site
  • Provide support and guidance to other team members and technical assistance as and when required
  • Positively respond to abnormal or emergency situations, making prompt decisions to minimise potential disruptions to stakeholders
  • Carry out first line fault finding on the plant, identifying defective equipment and replace/repair as required
  • Participate in the continuing development and review of standard operating procedures and safe systems of work
  • Be flexible with regards to the nature of any task, being effective, professional and presentable at all times
  • Undertake shift relief for Energy Centre Shift Operations Controllers
What we offer
What we offer
  • Competitive salary
  • Maternity / Paternity / Adoption leave pay
  • Boots Pension Scheme
  • Competitive holiday allowance
  • Generous staff discount (Boots, Boot Opticians and Boots Hearing Care)
  • £100 gift card for colleagues expecting/adopting a baby
  • Access to corporate discounts, including retail, cinema, holidays etc (UK only)
  • Discretionary annual bonus scheme
  • Access to our Employee Assistance Programme
  • Flexible benefits scheme (via salary sacrifice) including financial wellbeing support, gym membership, dental cover, life assurance, restaurant discount cards, activity passes, holiday buying and much more
  • Fulltime
Read More
Arrow Right
New

Oracle Integration Cloud Developer

Bright Vision Technologies is looking for a skilled Oracle Integration Cloud (OI...
Location
Location
United States
Salary
Salary:
Not provided
bvteck.com Logo
Bright Vision Technologies
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience with Oracle Integration Cloud (OIC)
  • Experience with Oracle SOA Suite
  • Experience with REST/SOAP APIs
  • Experience with Oracle ATP
  • Experience with Oracle ERP Cloud Integration
  • Experience with File Server (SFTP)
  • Experience with XML/XSD/XSLT
  • Experience with BPEL
  • Experience with ICS Packages
  • Experience with Security & Certificates
What we offer
What we offer
  • H-1B sponsorship for the 2026 quota
  • H-1B filing with level 4 prevailing wage
  • Fulltime
Read More
Arrow Right
New

Style Agent / Housekeeper

As "Talent", you represent W lifestyle to our guests. You are the soul behind th...
Location
Location
Switzerland , Verbier
Salary
Salary:
Not provided
https://www.marriott.com Logo
Marriott Bonvoy
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Prior experience in a 5* hotel in housekeeping
  • Highly motivated
  • Fluent in French and English
  • Detail oriented
  • Passion for giving guests an exceptional experience
  • Real interest in the worlds of taste, sound, body, scene and stance
  • Must be from Switzerland or the European Union (due to visa restrictions)
Job Responsibility
Job Responsibility
  • Clean guest rooms in accordance with established quality and performance standards
  • Clean occupied guest rooms
  • Clean and reset guest bedroom and bathroom areas according to established standards
  • Organize and stock cleaning cart
  • Organize linen closets on floors assigned
  • Fulltime
Read More
Arrow Right
New

Shipping and Receiving Clerk

The Shipping and Receiving Clerk is responsible for the proper receiving, labeli...
Location
Location
United States , Oakland
Salary
Salary:
Not provided
ajinomotofoods.com Logo
Ajinomoto Foods North America, Inc.
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Basic math and organization skills
  • Ability to bend and lift 50 pounds or more
  • Detail-oriented mindset
  • Forklift certification
  • Ability to work on feet for long hours
  • Excellent communication skills
  • Understanding of safety protocols and requirements
  • Strict adherence to all safety measures
  • Excellent work ethic and a Lean mindset
  • Valid Driver’s License, clean driving record
Job Responsibility
Job Responsibility
  • Inspect inbound shipments for damage and correct quantities
  • Report damages or shortages to the appropriate person
  • Record receiving data in company databases
  • Ensures all pallets are tagged with PID labels and Pallet Flags
  • Processes shipments in shipping software
  • Ensure product is accurately pulled and labeled for shipment
  • Responsible for scanning and filling of Shipping and Receiving paperwork
  • Meet or exceed productivity standards in all functions performed during assigned shift
  • Check-in/out delivery drivers and liaise with office staff for paperwork
  • Fulltime
Read More
Arrow Right