Project Description
The Lead Data Platform Engineer plays a crucial role in developing and enhancing a cloud-native Data Platform for a large-scale e-commerce marketplace. The primary objective is to improve developer experience and streamline data processes through automation and standardization, ultimately enabling data-driven decisions and supporting rapid product growth.
Responsibilities
- Design and develop tools that standardize, automate, and simplify Data Platform operations.
- Build and maintain internal CLI tools for common platform and Data Product management tasks.
- Develop and evolve Data Product templates based on modern data engineering practices.
- Design and implement visualizations and operational insights within Backstage.
- Collect and analyze platform usage metrics, audit data, and adoption statistics.
- Design and implement Agentic AI workflows and developer-assistance capabilities.
- Develop reusable AI skills and automation components that standardize work with data products and data assets.
- Create mechanisms for distribution, lifecycle management, and monitoring of AI skills.
- Maintain and enhance a shared Apache Airflow platform based on Cloud Composer.
- Build reusable libraries, operators, and common components for Airflow DAG development.
- Develop and maintain infrastructure-as-code assets and Terraform modules.
- Contribute to GitOps adoption initiatives using tools such as Backstage, GitHub, ArgoCD, and Crossplane.
- Collaborate with platform, data engineering, analytics, and machine learning teams to improve platform usability and engineering efficiency.
Requirements
- Strong commercial experience with Python development.
- Hands-on experience with Google Cloud Platform (GCP).
- Practical experience with BigQuery and cloud-based data platforms.
- Experience with Apache Airflow, preferably Cloud Composer.
- Experience building platform engineering, developer tooling, or internal self-service solutions.
- Knowledge of Infrastructure as Code practices and Terraform.
- Experience with GitOps principles and modern software delivery practices.
- Good understanding of data engineering concepts and Data Product lifecycle management.
- Experience with PySpark and/or Apache Spark ecosystems.
- Familiarity with FastAPI, Pydantic, and modern Python tooling.
- Knowledge of CI/CD processes and source control best practices.
- Experience working in Agile development environments.
- Strong problem-solving skills and ability to work independently.
- Effective communication skills and ability to collaborate with cross-functional teams.
- Professional proficiency in Polish and English.
Nice to Have
- Experience with Vertex AI or other GenAI platforms.
- Hands-on experience with GitHub Copilot, Copilot extensions, plugins, or AI-assisted development solutions.
- Experience developing Agentic AI workflows or AI automation capabilities.
- Knowledge of Backstage plugin development.
- Experience with Dataproc.
- Familiarity with dbt and Kedro.
- Experience with ArgoCD and Crossplane.
- Experience building observability, telemetry, or platform analytics solutions.
- Experience working in large-scale data environments supporting analytics and machine learning workloads.
RO
