A Generative Model for Discovering Unknowables

Source: Deep Learning on Medium

Photo by Martino Pietropoli on Unsplash

In a previous article I wrote about an actor’s state of knowing. In this article, I will explore the limits of knowing to better understand that nature of the attainability or unattainability of knowledge. This has pragmatic importance in that a framework tells what is within the realm of possibility and what is not. This prunes our search space for knowledge to that what is known to be attainable.

Here’s a schema of knowledge attainability:

Nescience defined here as “unattainable knowledge”.

Universality is classified here under ‘unknowable knows’ in that systems that exhibit universality (i.e. Formal Systems, Turing Machines, Cellular Automata) are known to have dynamics that are unknowable. That is, we know what is unknowable! The paper “Self-referential basis of undecidable dynamics: from The Liar Paradox and The Halting Problem to The Edge of Chaos” discussed the emergence of undecidable dynamics. This undecidability as a consequence of the self-referential features of computational systems:

Self-referential basis of undecidable dynamics: from The Liar Paradox and The Halting Problem to The Edge of Chaos

Pragmatically, we exclusively seek a solution to the problem of “knowable unknowns” using of course systems that exhibit universality. How does knowledge that is unknown become known? To illustrate, prior to Gödel, it was believed that a consistent and complete formal systems can be discovered. Gödel proved the existence of knowledge that will always be unknowable. Beyond the edge of what is known, how does one discover the unknown?

My thinking about navigating uncertainty or knowledge discovery is inspired by Stuart Kauffman’s “patterns of evolution”. It’s best to adopt his vocabulary to better articulate my ideas on this. I do recommend that you read:

I propose a post-entailing law explanatory framework in which Actuals arise in evolution that constitute new boundary conditions that are enabling constraints that create new, typically unprestatable, adjacent possible opportunities for further evolution, in which new Actuals arise, in a persistent becoming. Evolution flows into a typically unprestatable succession of adjacent possibles.

Knowledge discovery also employs the same post-entailing law proposed by Kauffman, where in the new in the universe of knowledge are enabling constraints that create new adjacent possible knowledge. Then when some new knowledge is discovered in the previously unknown adjacent possible then new knowledge becomes emergent in a novel way that could not have been predicted previously. Radical emergence arises in evolution and in learning that is impossible to capture with present day mathematics. This is because the boundary conditions keep changing with each new discovery of knowledge.

There is currently some mathematical understanding about the distribution of innovation. This is discussed in “Mathematical Model Reveals the Patterns of How Innovations Arises”.


In the cited paper, that innovation is enabled by “the adjacent possible”. That is those patterns that are one step away from existing learned patterns. So rather than developing patterns that have no connection, new patterns are realized through existing patterns and the thus new areas of unexplored patterns are discovered:

by providing the first quantitative characterization of the dynamics of correlated novelties, could be a starting point for a deeper understanding of the different nature of triggering events (timeliness, scales, spreading, individual vs. collective properties) along with the signatures of the adjacent possible at the individual and collective level, its structure and its restructuring under individual innovative events.

The conjecture here however is that the class of unseen patterns are either of the class that is “easily imagined and expected” or even better “an entirely unexpected and hard to imagine” class. That is novelties versus innovations. The paper above seems to indicate that the mechanisms to discover the latter is the same mechanism as that of the former:

The same model accounts for both phenomenon. It seems that the pattern behind the way we discover novelties — new songs, books, etc. — is the same as the pattern behind the way innovations emerge from the adjacent possible.

However, the above research only provides a descriptive model of innovation. We can classify distributed models of the world into three classes:

[1810.04261] A Tale of Three Probabilistic Families

A discriminative model is what a machine learning system generates when learning a classifier. It discovers how to recognize patterns that associates a input signal with a class label. A descriptive model specifies the probability distribution of a signal extracted from the descriptive feature statistics extracted from the signal. This is what is discovered in Thermodynamics in its bulk measurements of systems with a large collection of particles. The purpose of fields like Statistical Mechanics is to derive the distributions from first principles. Probabilistic graph models attempt to generate distributions, not from first principles, but rather from an intelligent selection of prior distributions. A generative model begins with the dynamics of many subcomponents and simulates this to generate bulk behavior.

This is classification is pragmatically important because too many people seem to conflate these three models as being the same (see Latent Variable Models).

The question I seek is, how can we create generative models that exhibit intuitive ingenuity. Previously, I wrote about ingenuity and brought to attention the ideas of Christopher Alexander regarding “generative” systems. Generative systems leads to holistic system with emergent properties where the sum of parts is greater than the whole. Emergence exists because as each new innovative component is discovered, an incremental new set of combinations is introduced that can lead to new capabilities previously non-existent prior to its discovery.

What are the intrinsic characteristics of a generative component?

Here are three characteristics that I’ve identified:

  1. Adaptive by default. Is valuable as is.
  2. Reconfigurable. Subcomponents can be recombined to be valuable in different contexts.
  3. Generative. Combinable with other components to create new kinds of Generative components.

So as an example of a generative system, we can look at programs. Programs can be characterized as having a few components. That is assignment (assigning values to memory), sequencing (steps of instruction), conditionals (selecting instruction paths) and loops (repetition). The reconfiguration of these components lead to valuable programs. In addition, as Computer Science has advanced, we have added other conveniences such as call stacks (subroutines), naming (variables) etc. There are all however easily emulated by the core generative components. That is, higher generative components are created out of the core generative components. This kind of design modularization is the key to the rapid advances in Computer Science.

However, what is the generative system for discovering unknowables? What is the generative system that can lead to automation that can automate knowledge discovery?