Curious Resources — Volume #1

Interesting finds in interactive technologies — covering the Web, Unity, Mixed Reality, Deep Learning, JavaScript, C#, 3D, Audio, Art & more

Hi! I’m starting a series of posts that will collect some of the interesting repositories and articles that I’ve come across lately. I’ll also dig up references from my ever growing toolbelt of Github stars. I love research, exploring patterns, making connections, and creating new possibilities.

This week we’re covering:
– Spatial Audio (Ambisonics) for 360/ VR / AR / XR
– GraphQL For Faster Front-End Iterations
– Live Streaming 360 Video to YouTube From Inside of Unity
– Assorted Deep Learning Finds: Poetry, Audio, Hi-Res Images, Unity Agents

I own a company that specializes in experiences — get in touch with me on LinkedIn!

Spatial Audio (Ambisonics) for 360 / VR / AR / XR

I’ve been thinking of applying to Google’s “Jump Start” program to get my hands on their new YI HALO 360 camera rigs for a series of creative shoots involving magical creatures in abandoned places. One of the questions on the application asks about spatial audio aka ambisonics, which remains somewhat unexplored compared to reactive visual systems. I went back through some of my finds:

Google has a spatial audio SDK for ambisonics called Resonance

The Unity package is on the Releases page here

Web devs can use this library

The engine behind the Web SDK

For advanced users, you can edit in your favorite DAW as a VST

Google’s encoding specs for audio and video

Facebook has some slick software, but its source is nowhere to be found

An Ableton-specific ambisonics solution

GraphQL For Faster Front-End Iterations

Front-end developers building client applications can sometimes get blocked waiting for back-end API changes to keep up with design.

What if you could just go ahead change that API from the front end? What if you didn’t have to worry about whether a particular endpoint exists — if the data is there, you can construct one yourself. Even better if you could keep that app trim by only getting the data you want when you make a request, instead of a ton of bloat.

To that end, I’ve been considering whether to recommend a GraphQL API instead of a REST API to an e-commerce client. It’s important that they be able to move quickly. While doing my research, I came across some interesting resources:

Starting from this excellent article

GraphQL query visualizer

GraphQL server for Express

Interesting how GraphQL is just an interface on top of whatever database you want to use. For example, you can use it with MongoDB

Or PostgreSQL

The spec itself, although I’d start with articles and tutorials instead

Some front end tools for interacting with GraphQL

Specifically for React

A whole framework from Facebook that couples React with GraphQL

Modifying control flow of queries and results

Shorthand for writing queries

Live Streaming 360 Video to YouTube From Inside of Unity

I’ve been trying to sort out a pipeline to live stream equirectangular video captured from a camera inside of a Unity scene to YouTube 360. I want viewers to be able to see inside of live game worlds as they’re happening. I’ve found a couple of useful leads:

Here’s one to output using FFMPEG (which could then stream to YouTube)
via the amazing Keijiro

But at what speed? The new scriptable render pipeline in Unity 2018 might enable a faster solution

I also evaluated a paid option

The paid option makes use of this very interesting concurrency library

Assorted Deep Learning Finds

Generate poetry from images using CNNS and RNNs

Get audio into a Pytorch tensor — more audio Deep Learning projects, please!

Speech at super-resolution could lead to clearer communications.

Make music alongside a Google AI — a new generation of instruments and live performances may be collaborations with a computer

Future Alert: with the right kind of sensors, deep learning should enable subvocal speech recognition — the ability to communicate without making sound, which would come in handy in smart gear for divers or firefighters

Many state-of the art networks have been trained on extremely small images (300px by 300px) due to GPU memory size constraints— however, they have been very accurate despite this limitation. I’m excited about anything trained on high resolution images, since it will help us understand whether smaller training images are sufficient or only scratching the surface of what’s possible. More pixels!

Unity is a great environment to train deep learning agents — especially multi-agent environments.

Here’s a hide/escape example

Here’s one with pirates!

Source: Deep Learning on Medium