Solving the Semantic Gap via the Symbol Detachment Problem

Source: Deep Learning on Medium

Solving the Semantic Gap via the Symbol Detachment Problem

Photo by Chris Barbalis on Unsplash

In my previously described Capability Maturity Model for Deep Learning, I discussed hybrid systems at Level II. Dual Process (or Hybrid) systems are systems that employ traditional computer science algorithms in combination with Deep Learning networks. Traditional computer science algorithms (where GOFAI is included in this set) are human-engineered algorithms. An open research problem is how can we gain the benefits of both symbolist (GOFAI) and deep learning (DL) approaches. This is known as bridging the Semantic Gap. The brittleness of GOFAI’s symbolic approach is that symbols are not grounded to semantics. If we are to leverage the symbolic rules we need to be able to ground it with the semantics discovered by deep learning systems. This fusion will be a major step towards more capable artificial intelligence system.

In my approach towards more general intelligent systems, I followed the prescription that demanded a bottom-up approach. That is one where an agent is embodied in its environment and learns through interaction (see: Embodied Learning). This is known in cognitive psychology as Enactivism (as opposed to Cognitivism). The current model is known as 4E:

Embodied involving more than the brain, including a more general involvement of bodily structures and processes.

Embedded functioning only in a related external environment.

Enacted involving not only neural processes, but also things an organism does.

Extended into the organism’s environment.

Subscribing to Enactivism requires that symbols do arise from the bottom up and not a jerry-rigged through a yet to be discovered ‘symbol grounding’ problem. The Enactivist approach to this problem is what is known as the ‘symbol detachment’ problem:

Giovanni Pezzulo and Cristiano Castelfranchi propose that “anticipation” plays a crucial role in the detachment process. That is, the representations for anticipation (i.e. action specification) is intrinsically detached from the original sensorimotor cycle. These representations are what is exaptated for use as symbols. The basic idea is that there is an internal action model that is imagined (i.e. generated model) by the agent. This action model is argued to be detachable from the sensorimotor cycle:

Paul Cisek explores this from from the perspective of the evolution of cognitive skills:

Cisek argues that an alternative taxonomy that is inspired by evolutionary process is a more informed one over the conventional taxonomy of behavior that divided into perception, cognition and action. He proposes the following taxonomy that is based on evolutionary development:

https://link.springer.com/article/10.3758%2Fs13414-019-01760-1

Another explanation of the symbol detachment is explained by Terrence Deacon. Deacon explains that indexical relations, which are intrinsically grounded are captured by the grammar and syntax of our language. This is what is grounded and ungrounded symbols are bound to sentences and thus given grounding (i.e. meaning).

The key to solving the symbol detachment problem is through an understanding of semiotics (see: Deep Learning and Semiotics). Deep Learning systems are able to extract the indexical and iconic representations of the world. Language captures navigational instruction, this involves what kind of actions to perform given an encountered landmark. It’s the responsibility of the generating agent (i.e. the speaker) to express the given action and to describe the landmark. It’s the responsibility of the listener to resolve any ambiguity when executing the navigational instructions. Like most navigation instructions, a listener doesn’t discover symbol grounding until landmarks are identified.

Thus, robust interpretation of navigation requires late symbol binding.