News

Abstract: Actions in many real-world domains cannot be executed exactly. An agent’s performance in these domains is influenced by two critical factors: the ability to select effective actions (decision-making skill), and how precisely they can execute those selected actions (execution skill). For an AI to make effective action recommendations to a person, knowledge of their execution skill is required. This talk addresses the problem of estimating both the execution and decision-making skill of an agent, given observations. Several execution skill estimation methods will be presented, each of which utilize different information from the observations and make assumptions about the agent’s decision-making ability. A final novel method forgoes these assumptions about decision-making and instead estimates the execution and decision-making skills simultaneously under a single Bayesian framework. Experimental results in several domains evaluate the estimation accuracy of the estimators, especially focusing on how robust they are as agents and their decision-making methods are varied. These results demonstrate that reasoning about both types of skill together significantly improves the robustness and accuracy of execution skill estimation. A case study using the proposed methods to estimate the skill of Major League Baseball pitchers is presented, along with a complete AI system that utilizes this estimate to provide pitch recommendations. This case study demonstrates how these skill estimation methods can be applied to real-world data sources and provide effective action recommendations to humans.