Information Technology
Python / Spark / AI Developer
Job ID: #
Python / Spark / AI Developer
Key Responsibilities:
Build Python & PySpark pipelines on Delta Lake
Replatform legacy T-SQL to Spark SQL
Develop tested, type-safe Python with CI quality gates
Build FastAPI REST APIs with authentication & job tracking
Validate migrated pipelines against legacy systems
Work spec-first, following design documents & ADRs
Location:
Cape Town, Remote/Hybrid or In office candidate can decide
Job Type:
Permanent
Date:
2 days ago
Requirements
4+ years Python development
2+ years production Spark/PySpark
Strong Python 3.12, pytest, Pydantic & modern tooling
Spark SQL, DataFrames, performance tuning & distributed architecture
Delta Lake / Iceberg / Hudi
Strong SQL & T-SQL migration experience
FastAPI / REST API development
Docker, Git & CI/CD
Able to work independently in a spec-driven environment
⭐ Nice to Have
LLM/AI Integration • Data Privacy & Governance • Azure/ADLS/Synapse • Hive/Trino/Ranger •
Observability • Data Migration & Parity Testing • Databricks Certification
Qualifications:
Bachelor’s Degree in Computer Science, Engineering or equivalent experience
Spark/Databricks certification advantageous