If you are looking for a new Data Engineering role and you have Databricks experience and Machine Learning experience, then I have a great opportunity for you.
The role is remote
You must be based in the Netherlands
6 months +
Key Responsibilities:
Machine Learning at Scale
- Scaling ML Models: Ensuring models can handle increasing data and usage by implementing techniques such as distributed training, model parallelism, and optimizing infrastructure.
- Model Optimization: Focused on making sure models run efficiently at scale by improving performance, resource utilization, and latency.
- Retraining Models: Setting up processes for continuous retraining based on new data. This could involve automation of training pipelines to ensure models stay updated with fresh data.
Training and Retraining Models
- Automated Pipelines for Training: Setting up and managing pipelines to automatically train and retrain models based on new data or changes in the dataset.
- Model Validation: Implementing checks to validate models at each stage of training and retraining to ensure they perform as expected.
Machine Learning Pipelines
- Pipeline Development: Building and maintaining efficient, scalable ML pipelines. These might include data preprocessing, feature engineering, model training, and deployment.
- Orchestration: Using tools like Apache Airflow or Databricks Jobs for pipeline orchestration, ensuring the pipeline runs seamlessly from data ingestion to model deployment.
Infrastructure and Automation
- Infrastructure Management: Setting up the environment (e.g., on Databricks, cloud infrastructure like AWS, Azure) to ensure scalability, availability, and reliability of ML models.
- Automation: Implementing automation for various aspects of model development, deployment, and monitoring. This may involve automating model retraining, scaling, and deployment.
CI/CD for Machine Learning (MLOps)
- Continuous Integration/Continuous Deployment: Establishing CI/CD pipelines specifically for machine learning workloads, ensuring that models can be continuously integrated, tested, and deployed with minimal manual intervention.
- Versioning: Managing model versioning to handle updates and rollbacks efficiently.
Technical Skills:
- Databricks: Expertise in leveraging Databricks for distributed model training and managing ML workflows.
- Machine Learning Frameworks: Proficiency in tools like TensorFlow, PyTorch, Scikit-Learn, etc.
- Automation Tools: Familiarity with tools like Terraform, Ansible, or other infrastructure automation solutions.
- CI/CD Pipelines: Experience with Jenkins, GitLab CI, or other CI/CD platforms.
- Cloud Infrastructure: AWS, Azure, or GCP for scalable ML deployments.
Key Focus Areas:
- Scalable machine learning solutions that can handle large datasets.
- Automation of model training and retraining to ensure the most up-to-date models are being used.
- Building robust ML pipelines and integrating them with CI/CD tools for continuous deployment.
- Infrastructure setup to enable smooth ML operations.
If this is of interest to you please do not hesitate to contact me
Darwin Recruitment is acting as an Employment Business in relation to this vacancy.
Victoria Howes