Databricks Platform Administration 

• Deploy, configure, and maintain Azure Databricks workspaces
• Manage clusters, pools, and jobs for optimal performance.
• Handle Databricks Runtime upgrades and compatibility testing.
• Configure Databricks notebooks, Repos, and Delta Lake for efficient workflows.

 

Security & Access Control:

• Implement Identity and Access Management (IAM) using Azure AD / Azure IAM / SCIM.
• Set up and enforce Role-Based Access Control (RBAC) and network security.
• Ensure compliance with data governance policies (Unity Catalog, creation of Catalogs, schema etc.).
• Applying grants of catalogs and schemas
• Manage personal access tokens (PATs) and OAuth-based authentication.

 

Performance Monitoring & Optimization:

• Monitor cluster performance, auto-scaling, and cost optimization.
• Analyze and fine-tune Spark jobs to improve execution efficiency.
• Use Databricks SQL for querying and performance diagnostics.

 

Integration & Automation:

• Integrate Databricks with Azure Data Factory, Synapse, Snowflake, Kafka, and other data sources.
• Automate cluster management using Terraform.
• Implement CI/CD pipelines for Databricks notebooks using GitHub or Azure DevOps.

 

Troubleshooting & Support:

• Investigate and resolve job failures, cluster crashes, and query performance issues.
• Provide L2/L3 support for Databricks infrastructure issues.
• Collaborate with Databricks support for bug fixes and escalations.