Envision Technology Solutions
Job Title – Pyspark Lead
Location – Irving, TX – Onsite
Hire Type – Fulltime
Key Responsibilities
- Lead the design and development of scalable data pipelines using PySpark and distributed computing frameworks.
- Build and optimize ETL/ELT workflows for large-scale banking datasets.
- Collaborate with business stakeholders, data architects, analysts, and application teams to understand requirements.
- Ensure high performance, fault tolerance, and scalability of big data applications.
- Work with structured and unstructured datasets from multiple banking systems.
- Implement data quality checks, governance, and security standards.
- Mentor and guide junior developers and data engineers.
- Participate in architecture discussions and recommend best practices.
- Integrate data solutions with cloud platforms and enterprise data lakes.
- Support production deployments, troubleshooting, and performance tuning.
- Coordinate with offshore/onshore teams and manage delivery timelines.
Required Skills
Technical Skills
- Strong hands-on experience in PySpark and Apache Spark
- Expertise in Python programming
- Experience with Hadoop ecosystem components:
- HDFS
- Hive
- Sqoop
- Kafka
- Strong SQL and data warehousing concepts
- Experience with ETL pipeline development
- Knowledge of Airflow/Oozie for workflow orchestration
- Experience with cloud platforms such as:
- AWS
- Azure
- GCP
- Familiarity with Delta Lake, Databricks, or Snowflake
- Version control using Git
- CI/CD pipeline understanding
Banking Domain Skills
- Good understanding of banking data domains:
- Retail Banking
- Corporate Banking
- Risk & Compliance
- AML/KYC
- Payments/Transactions
- Understanding of regulatory and compliance requirements in banking environments.
Preferred Qualifications
- Bachelor’s/Master’s degree in Computer Science, Engineering, or related field
- Spark/Databricks/AWS certifications preferred
- Experience working in Agile/Scrum environments
- Exposure to real-time streaming and event-driven architectures
Soft Skills
- Strong communication and stakeholder management
- Leadership and team mentoring abilities
- Problem-solving and analytical mindset
- Ability to work in a fast-paced banking environment