Improving · Posted 2w ago
Big Data Engineer (Databricks)
Job relayed by Get on Board, its source: you apply there.
- Location
- Remote
- Role
- Data engineering
- Source
- Get on Board
Stack
- Databricks
- Spark
- Delta Lake
- Unity Catalog
- Azure
- CI/CD
The posting
• Develop and maintain data pipelines using Databricks.
• Support the design and optimization of Delta Lake tables.
• Work with Unity Catalog to support data governance and access management.
• Participate in CI/CD processes using Azure DevOps and Databricks Asset Bundles (DAB).
• Implement incremental data processing and support schema evolution requirements.
• Collaborate with team members to troubleshoot data quality and performance issues.
• Assist with data modeling activities and platform optimization efforts.
• Document solutions and follow engineering best practices.
• Ideally 5 YOE
• Strong expertise in Apache Spark, including delivery on Databricks
• Experience with Databricks Asset Bundles (DAB).
• Experience using Azure DevOps.
• Knowledge of incremental processing and schema evolution.
• Healthcare or Insurance industry experience.
• Experience with cloud-based data platforms.
• Azure Databricks certifications.
• Exposure to Medallion Architecture.
• Data quality and monitoring framework