NEW ADVANCES ON BAYESIAN AND DECISION-THEORETIC APPROACHES FOR INTERACTIVE MACHINE LEARNING

Abstract

THE EXPLORATION-EXPLOITATION TRADE-OFF IS A FUNDAMENTAL DILEMMA IN MANY INTERACTIVE LEARNING SCENARIOS, WHICH INCLUDE BOTH ASPECTS OF REINFORCEMENT LEARNING (RL) AND ACTIVE LEARNING (AL). MORE OFTEN, LEARNING STRATEGIES THAT IGNORE EXPLORATION WILL EXHIBIT SUB-OPTIMAL PERFORMANCE WHILE, CONVERSELY, THOSE THAT FOCUS ON EXPLORATION MIGHT SUFFER THE COST OF LEARNING WITHOUT BENEFITTING FROM IT. UNFORTUNATELY, WHILE THIS TRADE-OFF HAS BEEN WELL-RECOGNIZED SINCE THE EARLY DAYS OF RL, THE STUDIES OF EXPLORATION-EXPLOITATION HAVE MOSTLY BEEN DEVELOPED FOR THEORETICAL SETTINGS AND, PERHAPS SURPRISINGLY, GLOSSED OVER IN THE EXISTING AL LITERATURE. FROM A PRACTICAL POINT OF VIEW, WE SEE THREE LIMITING FACTORS:

1. PREVIOUS WORKS ADDRESSING THIS TRADE-OFF IN RL HAVE ONLY FOCUSED ON SIMPLE CHOICES OF THE ENVIRONMENT MODEL AND CONSEQUENTLY, ARE NOT PRACTICAL ENOUGH TO ACCOMMODATE MORE COMPLICATED ENVIRONMENT STRUCTURES.

2. EXISTING WORKS IN THE AL LITERATURE OFTEN ADVOCATE THE USE OF MYOPIC