Hello, once again this is the second part of the “Demystifying Generative Models” posts so if you haven’t read Part 1 yet, I really urge you to do so here.

In the previous post, we discussed the differences between discriminative and generative models, took a peek to the fascinating world of probabilities and used that knowledge to develop a working Naive Bayes that generates passwords for us. Now, we will change our methodologies a little bit and explore how Deep Learning can help us when probabilities fail.

Not so useful Naive-Bayes

Assuming that you have the context knowledge, of part 1, I will jump straight to the point. The Multinomial Naive Bayes model that was developed was based on the assumption that each feature is independent of each other and it worked quite well! But let’s change the problem formulation slightly and observe how it behaves. Instead of having 4 features, fixed order, and a limited amount of choices per features, let’s just define an alphabet with all the letters, digits and special characters that appear in our initial feature choices. These would be:

To make this new problem as close to the previous one as possible, we will constrain the maximum length of the password to be 15 since this is the max length from our passwords dataset. Additionally, let’s define another imaginary character called <NULL>. Since the model that will be described below will have to fill all 15 positions of the password the <NULL> character is just a way for the model to say put nothing here. Therefore from our perspective, any <NULL> characters will be ignored.

With all these changes, the alphabet now contains a total of 19+8+6+4 + 1= 38 characters (uppercase + lowercase + digits + special characters + null char). As a result, there are 38¹⁵ possible combinations in our sample space! Of course, this is a significantly larger sample space than previously but let’s observe how the Naive-Bayes model behaves in these circumstances.

The list below illustrates passwords generated using the old Naive-Bayes model in the current problem context: