GOALS

• Build and deploy machine learning models into production efficiently and reliably using Dagster's orchestration capabilities
• Leverage Dagster's software-defined assets (SDAs) to ensure data integrity and reproducibility in model training and evaluation
• Collaborate effectively with data engineers on building end-to-end ML workflows that are both performant and maintainable.

Frustrations

• Slow iteration cycles for deploying ML models due to inefficient handoffs between data science and engineering teams
• Difficulty in tracking data lineage and ensuring data quality for model training, leading to unreliable results
• Lack of transparency and reproducibility in the model deployment process hinders model governance and debugging efforts.

GOALS

• Empower the data team to adopt a unified and standardized approach to building and managing data pipelines across the organization
• Improve the observability and monitoring of data workflows to reduce downtime and ensure data quality for business-critical applications
• Optimize resource utilization and manage costs associated with data infrastructure by leveraging Dagster's advanced features.

Frustrations

• Fragmented data engineering practices across different teams lead to inconsistencies and knowledge silos
• Limited visibility into the performance and health of data pipelines makes it challenging to proactively identify and resolve issues
• Difficulty in controlling and optimizing cloud infrastructure costs associated with running data pipelines at scale.