EFFICIENT HIERARCHICAL REINFORCEMENT LEARNING THROUGH CORE TASK ABSTRACTION AND CONTEXT REASONING

Abstract

IN HIERARCHICAL REINFORCEMENT LEARNING (HRL), AN AUTONOMOUS AGENT ADOPTS A DIVIDE-AND-CONQUER APPROACH TO SOLVE LARGE, COMPLEX PROBLEMS BY RECURSIVELY DECOMPOSING THE ROOT PROBLEM INTO SMALLER TASKS, AND SOLVING THEM SYSTEMATICALLY. WE PROPOSE CONTEXT SENSITIVE REINFORCEMENT LEARNING (CSRL), A NEW MODEL-BASED APPROACH TO HRL THAT EXPLOITS SHARED KNOWLEDGE AND SELECTIVE EXECUTION AT DIFFERENT LEVELS OF ABSTRACTION TO EFFICIENTLY SOLVE LARGE, COMPLEX PROBLEMS. CSRL HAS THE FOLLOWING ADVANTAGES OVER EXISTING HRL METHODS. FIRST, CSRL DOES NOT REQUIRE THE FULL SET OF TASKS AND PRIMITIVE ACTIONS TO BE SPECIFIED. SECOND, CSRL FACILITATES EFFICIENT EXPERIENCE SHARING BETWEEN SIMILAR SUBTASKS OR TASKS WITH OVERLAPPING FEATURES. THIRD, CSRL CAN HANDLE PROBLEMS WHERE MULTIPLE TASKS ARE ACTIVE AT THE SAME TIME. WE TEST THE FRAMEWORK ON COMMON BENCHMARK PROBLEMS AND COMPLEX SIMULATED ROBOTIC ENVIRONMENTS. IT COMPARES FAVORABLY AGAINST THE STATE-OF-THE-ART ALGORITHMS, AND SCALES WELL IN VERY LARGE P