Developing AI systems for complex real-world settings requires aligning technical development with domain-specific needs. However, a gap often exists between stakeholders and developers; stakeholders may lack technical expertise to express their needs clearly, whereas developers may lack domain knowledge to identify relevant tasks. This thesis aims to bridge that gap by exploring how decision support systems can address complex real-world tasks through tailored technical solutions and evaluation procedures.

The work includes a qualitative multiple case study with make-to-order companies to identify and prioritise AI tasks for system development, along with experimental studies that address gaps in intermittent demand forecasting using a novel timing-aware model and evaluation metric. We also conduct a remote sensing ditch detection study for environmental planning. Both cases highlight the need to align models and evaluation procedures with task-specific challenges such as data sparsity, noise, and class imbalance.

Our findings show that make-to-order manufacturers prioritise tasks that improve customer understanding, such as demand forecasting and decision risk estimation, as well as production-related tasks like quality inspection and predictive maintenance. Demand forecasting emerged as the most important task, with challenges linked to heterogeneous data stemming from intermittent patterns and numerous unique items. Our experiments show that decomposing demand into timing and magnitude improves forecasting performance, and that timing-aware metrics are essential for fair evaluation on a global scale. The ditch detection case similarly underscores the value of domain-aligned design and evaluation. The thesis contributes empirical insights on industry priorities and technical advances in forecasting and evaluation, emphasising the importance of grounding AI development in real-world conditions.