NLP News Cypher | 01.19.20

Source: Deep Learning on Medium

And…. we’re back! What a week!

First, we are happy to announce we are now publishing our weekly blog on Towards AI’s platform💪💪! Happy for this publishing partnership as we intend to bring NLP trends to a more global audience from developers in NYC to business professionals in Hong Kong.

And talking about global…

In case you were busy this past week: we dropped the “The Big Bad NLP Database”, a large collection of datasets for ML and NLP developers! The database continues to grow and we have already received excellent recommendations from our users. Updates coming very soon!

Wednesday’s Announcement Article:

Database:

Thank you for all of your support!

declassified

This Week:

The Reformer Transformer

Speaking Thoughts and Minds

Recognizing Facebook’s Real-Time Speech

Deep Hacks

Wolfram Webinars on Data Analytics

CoNLL Meet Spacy

I Recommend Research Papers

Dataset of the Week: ReCord

Meanwhile, Back at the Vegas Ranch…

The Reformer Transformer

“A Transformer model designed to handle context windows of up to 1 million words, all on a single accelerator and using only 16GB of memory.”

Google started the new year with a bang and a new transformer. Google’s new model wants to solve 2 problems weighing on transformers with large input sequences: attention and memory.

Attention is difficult to scale under a large number of words, and as a result, Google introduced a hashing technique allowing the model to efficiently “connect” similar vectors together and dividing them into chunks. After applying attention over these segments, it leads to a reduction of computational load.

The memory problem arises in a multi-layered model because of the requirement to save the activation at each layer for the backward pass. This can lead to your GPU’s memory exploding aka OOM errors.

LINK

To mitigate this issue, Google turned to reversible layers. (This technique is discussed in the paper above). It avoids storing the activation of each layer in memory and instead computes them on the backwards pass through a clever technique.

Blog:

Colab for Text Generation:

Speaking Hearts and Minds

New research from Facebook’s Conversational AI Research group ParlAI, brings together a single model that performs well on several image-grounded conversational tasks (aka multi-modal). Below you can find a few example outputs:

Paper:

LINK

Recognizing Facebook’s Real-Time Speech

Facebook open-sourced their wav2letter@anywhere speech recognition framework. The take-away here is that inference for this framework is geared for real-time performance. Also, it achieves SOTA performance on the LibriSpeech dataset!

Blog:

GitHub:

Deep Hacks

When engineering, the cookie crumbles on the micro-level. Yes, you can achieve great results from a large fine-tuned transformer but you still need to hone your skills on the classics (import re). Priyansh Trivedi drops a few jewels on this topic in his blog:

Wolfram Webinars on Data Analytics

There are three 90 min. Wolfram webinars coming up that will be highlight custom-built Twitter analytics, data mining of imaginary maps and how to create an automated reporting system (and much more). If the Wolfram language and these subjects interest you, register can here.

Blog:

CoNLL Meet Spacy

Well now. Special shout-out to Bram. He updated the spacy_conll repo which allows you to parse your text into CoNLL-U format. The plugin can now be used as a custom pipeline in command line or in a python script.

What’s CoNLL you may ask?

> python -m spacy_conll --input_str "I like cookies . What about you ?" --is_tokenized --include_headers
# sent_id = 1
# text = I like cookies .
1 I -PRON- PRON PRP PronType=prs 2 nsubj _ _
2 like like VERB VBP VerbForm=fin|Tense=pres 0 ROOT _ _
3 cookies cookie NOUN NNS Number=plur 2 dobj _ _
4 . . PUNCT . PunctType=peri 2 punct _ _

# sent_id = 2
# text = What about you ?
1 What what NOUN WP PronType=int|rel 2 dep _ _
2 about about ADP IN _ 0 ROOT _ _
3 you -PRON- PRON PRP PronType=prs 2 pobj _ _
4 ? ? PUNCT . PunctType=peri 2 punct _ _

I Recommend Research Papers

Santosh created a recommendation engine for searching research papers using natural language called Natural Language Recommendations! It was trained on abstracts, so the longer the description the better the search results.

Check out his GitHub which also includes a Colab for trialing the engine.

Dataset of the Week: ReCoRD

What is it:

“A reading comprehension dataset which requires commonsense reasoning.”

Sample:

Where is it?

Meanwhile, Back at the Vegas Ranch…

Last but not least, I can’t believe this is true but apparently indeed.com has a job posting calling for an escort with a Department of Defense “TOP-SECRET” clearance?!?!? 🤣🤣🤣 Are you qualified? Post below: