The Intuition of Distributed Representations

This post describes the nature of data representations in artificial intelligence. Representations include quantifying real objects, concepts, objectives or policies in a numerical way. Representation is central to the question of, ‘How do you accurately quantify reality, such that you can then perform useful modelling of it?’

I argue that the challenge for machine intelligence fields is to fully transition from the belief in discrete, local representations into continuous, distributed representations. In some techniques in AI, there is a vestigial belief that strongly bounded object exist, essentially related to the belief in a metaphysical essence.

What I believe is consistent with the success in AI so far is the same principle that makes neural networks themselves successful. That is, the distinction between low-dimensional symbolic representations and high-dimensional distributed representations.

To illustrate how distributed representations are the most accurate way to quantify anything in reality, consider the image below.

Figure 1. Apple and pear visualization (Source: iStockPhoto)

In symbolic AI, or techniques which make use of low-dimensional vectors, the two objects might be classified as [0 0 … 0 1 0] for apple and [0 0 … 0 0 1] for pear. The problem with representations like this is that it contains no metrics for variations nor dimensions/continuums/spectrums for anything to be understood on. What you want instead is many dimensions of metrics to describe the data.

You can demonstrate the utility of a distributed representation of the above apple and pear through imagining that the features of the apple start to change along continuums such that it begins to resemble the pear. If the values of it’s parameters in a representational vector continue to change such that it’s appearance becomes identical to the pear, the values of the apple and pear vectors will also be identical.

So now the apple has become a pear. But this is a demonstration that it was never really an apple to begin with. Nor is the pear itself some bounded category. There is no such thing really as an apple or pear. Likewise for any category in reality.

On the back-end of distributed systems, and likewise in the brain, there is no such thing as discrete classes of apple, pear, table or chair. There are only these unstructured, distributed representations in many dimensions. The unique combination of these metrics forms the only understanding the system has of differences between real objects.

The figure below demonstrates quantifying personality phenomena in 11 dimensions as a more accurate approximation of truth than simply ‘extrovert’ or ‘introvert’.

Figure 2. Quantifying personality through the use of 11 dimensions (Source:

The counter-intuitive nature of this concept is that the more distributed and unstructured the representation becomes, the less insightful it becomes for a human to perceive. We think what we want to give a machine system is the most insightful answer, so we provide it with the most reduced, 1-dimensional answer. However by doing this you corrupt the system of really understanding the totality of real phenomena.

In order for a system to be general enough to conceive of many objects, and to perform useful operations between these, all representations have to contain the same set of dimensions that any real object will be understood in terms of. The limit of a system’s understanding, in some sense, is the set of dimensions of it’s representational vectors.

You want to be able to represent all objects accurately with the same complete set of dimensions, such that no unique phenomena is represented by a non-unique vector. For example, if [color: 5, utility: 3, social: 6] could mean both a pizza or a tennis ball, the system will have poor understanding. The next step here would be to increase the number of dimensions such that differentiating metrics between a pizza and a tennis ball would emerge.

All objects and phenomena will then be integrated, and the model can then conceive of the possibility of any matter becoming anything else, which is what can occur in reality. Overall, this is consistent with the general aims of scientific theories, which is, broadly, to unify a variety of phenomena under the same continuous mechanisms.

Source: Deep Learning on Medium