Curious Resources — Volume #1

Interesting finds in interactive technologies — covering the Web, Unity, Mixed Reality, Deep Learning, JavaScript, C#, 3D, Audio, Art & more

Hi! I’m starting a series of posts that will collect some of the interesting repositories and articles that I’ve come across lately. I’ll also dig up references from my ever growing toolbelt of Github stars. I love research, exploring patterns, making connections, and creating new possibilities.

This week we’re covering:
– Spatial Audio (Ambisonics) for 360/ VR / AR / XR
– GraphQL For Faster Front-End Iterations
– Live Streaming 360 Video to YouTube From Inside of Unity
– Assorted Deep Learning Finds: Poetry, Audio, Hi-Res Images, Unity Agents

I own a company that specializes in experiences — get in touch with me on LinkedIn!

Spatial Audio (Ambisonics) for 360 / VR / AR / XR

I’ve been thinking of applying to Google’s “Jump Start” program to get my hands on their new YI HALO 360 camera rigs for a series of creative shoots involving magical creatures in abandoned places. One of the questions on the application asks about spatial audio aka ambisonics, which remains somewhat unexplored compared to reactive visual systems. I went back through some of my finds:

Google has a spatial audio SDK for ambisonics called Resonance
https://developers.google.com/resonance-audio/

The Unity package is on the Releases page here
https://github.com/resonance-audio/resonance-audio-unity-sdk

Web devs can use this library
https://github.com/resonance-audio/resonance-audio-web-sdk

The engine behind the Web SDK
https://github.com/GoogleChrome/omnitone

For advanced users, you can edit in your favorite DAW as a VST
https://github.com/resonance-audio/resonance-audio-daw-tools

Google’s encoding specs for audio and video
https://github.com/google/spatial-media

Facebook has some slick software, but its source is nowhere to be found
https://facebook360.fb.com/spatial-workstation/

An Ableton-specific ambisonics solution
https://github.com/EnvelopSound/EnvelopForLive

GraphQL For Faster Front-End Iterations

Front-end developers building client applications can sometimes get blocked waiting for back-end API changes to keep up with design.

What if you could just go ahead change that API from the front end? What if you didn’t have to worry about whether a particular endpoint exists — if the data is there, you can construct one yourself. Even better if you could keep that app trim by only getting the data you want when you make a request, instead of a ton of bloat.

To that end, I’ve been considering whether to recommend a GraphQL API instead of a REST API to an e-commerce client. It’s important that they be able to move quickly. While doing my research, I came across some interesting resources:

Starting from this excellent article
https://time2hack.com/2018/02/introduction-quick-guide-to-graphql-for-backend-frontend/

GraphQL query visualizer
https://github.com/graphql/graphiql

GraphQL server for Express
https://github.com/apollographql/apollo-server

Interesting how GraphQL is just an interface on top of whatever database you want to use. For example, you can use it with MongoDB https://www.compose.com/articles/using-graphql-with-mongodb/

Or PostgreSQL
https://github.com/postgraphql/postgraphql

The spec itself, although I’d start with articles and tutorials instead
https://github.com/facebook/graphql

Some front end tools for interacting with GraphQL
https://github.com/apollographql/apollo-client

Specifically for React
https://github.com/apollographql/react-apollo

A whole framework from Facebook that couples React with GraphQL
https://github.com/facebook/relay

Modifying control flow of queries and results
https://github.com/apollographql/apollo-link

Shorthand for writing queries
https://github.com/apollographql/graphql-tag

Live Streaming 360 Video to YouTube From Inside of Unity

I’ve been trying to sort out a pipeline to live stream equirectangular video captured from a camera inside of a Unity scene to YouTube 360. I want viewers to be able to see inside of live game worlds as they’re happening. I’ve found a couple of useful leads:

Here’s one to output using FFMPEG (which could then stream to YouTube)
via the amazing Keijiro
https://github.com/keijiro/FFmpegOut

But at what speed? The new scriptable render pipeline in Unity 2018 might enable a faster solution
https://github.com/keijiro/AsyncCaptureTest

I also evaluated a paid option
https://github.com/SystemFriend/GPUVideoEncoder-Trial

The paid option makes use of this very interesting concurrency library
https://assetstore.unity.com/packages/tools/spicy-pixel-concurrency-kit-3586

Assorted Deep Learning Finds

Generate poetry from images using CNNS and RNNs
https://github.com/rossgoodwin/neuralsnap

Get audio into a Pytorch tensor — more audio Deep Learning projects, please!
https://github.com/pytorch/audio

Speech at super-resolution could lead to clearer communications.
https://github.com/jhetherly/EnglishSpeechUpsampler

Make music alongside a Google AI — a new generation of instruments and live performances may be collaborations with a computer
https://github.com/tensorflow/magenta

Future Alert: with the right kind of sensors, deep learning should enable subvocal speech recognition — the ability to communicate without making sound, which would come in handy in smart gear for divers or firefighters
https://github.com/Mohamed-Elbially/ThesisCodes

Many state-of the art networks have been trained on extremely small images (300px by 300px) due to GPU memory size constraints— however, they have been very accurate despite this limitation. I’m excited about anything trained on high resolution images, since it will help us understand whether smaller training images are sufficient or only scratching the surface of what’s possible. More pixels!
https://github.com/NVIDIA/pix2pixHD

Unity is a great environment to train deep learning agents — especially multi-agent environments.
https://github.com/Unity-Technologies/ml-agents

Here’s a hide/escape example
https://github.com/HappySlice/hide-escape

Here’s one with pirates!
https://github.com/HugoCMU/pirateAI

Source: Deep Learning on Medium