ABSTRACT
Data-intensive flows are increasingly encountered in various settings, including business intelligence and scientific scenarios. As the data flows become more and more complex and operate in a highly dynamic environment, we argue that we need to resort to automated cost-based optimization solutions rather than relying on efficient designs by human experts. In this talk, we are going to discuss three complementary aspects of dataflow optimization. First, motivated by the fact that current approaches tend to employ multiple execution engines, we discuss state-of-art solutions to the problem of allocating flow activities to specific heterogeneous and interdependent execution engines while minimizing the flow execution cost. Second, we are going to discuss novel approaches to automatically define the execution order of the constituent tasks in a flow, thus relieving the designer from the burden of manually deciding the exact execution plan in full detail. Third, we narrow our focus on MapReduce-like systems and their descendants like Spark, and we discuss trade-offs between individual executor load and data transmission over the network during shuffling.

Conference information