Data Systems Seminar Series

With the emergence of Large Language Models (LLMs), such as OpenAI’s GPT, the role of humans in traditional data pipelines, including data integration and discovery, has been reconsidered. Despite significant advancements, many still believe that human intelligence is irreplaceable. Humans possess the unique ability to generate multiple, diverse ideas and solve problems through spontaneous, free-flowing thinking. Accordingly, human-in-the-loop solutions, tailored to the current state of research, are essential for the development of future intelligent systems. Human-in-the-loop generally refers to leveraging human intelligence within data science pipelines. However, the reliability and proficiency of these human annotators can vary, potentially impairing the quality of the generated results. It is therefore crucial to understand human behavior and identify effective ways to utilize their input. As contemporary research increasingly employs LLMs, human involvement must be adapted accordingly.

In this talk, I will discuss the merits of human-in-the-loop data integration and discovery, with a focus on the challenges of achieving effective human matching and validation. To illustrate how machine learning can support human-in-the-loop processes, I will present a novel perspective on using human input as labels, acknowledging that humans can make errors. I will show how using a behavioral profile, we can calibrate their input to improve labeling quality. To support these claims, I will present two proof-of-concepts: one for data integration and one for data discovery. For integration, I will present results from three different matching tasks: schema matching, entity matching, and text matching, demonstrating that our method enhances label quality across multiple settings, including cross-domain applications. For discovery, I will share a preliminary study we conducted on human involvement in table unionability.