Hi! I’m starting a series of posts that will collect some of the interesting repositories and articles that I’ve come across lately. I’ll also dig up references from my ever growing toolbelt of Github stars. I love research, exploring patterns, making connections, and creating new possibilities.
This week we’re covering:
– Spatial Audio (Ambisonics) for 360/ VR / AR / XR
– GraphQL For Faster Front-End Iterations
– Live Streaming 360 Video to YouTube From Inside of Unity
– Assorted Deep Learning Finds: Poetry, Audio, Hi-Res Images, Unity Agents
I own a company that specializes in experiences — get in touch with me on LinkedIn!
Spatial Audio (Ambisonics) for 360 / VR / AR / XR
I’ve been thinking of applying to Google’s “Jump Start” program to get my hands on their new YI HALO 360 camera rigs for a series of creative shoots involving magical creatures in abandoned places. One of the questions on the application asks about spatial audio aka ambisonics, which remains somewhat unexplored compared to reactive visual systems. I went back through some of my finds:
Google has a spatial audio SDK for ambisonics called Resonance
The Unity package is on the Releases page here
Web devs can use this library
The engine behind the Web SDK
For advanced users, you can edit in your favorite DAW as a VST
Google’s encoding specs for audio and video
Facebook has some slick software, but its source is nowhere to be found
An Ableton-specific ambisonics solution
GraphQL For Faster Front-End Iterations
Front-end developers building client applications can sometimes get blocked waiting for back-end API changes to keep up with design.
What if you could just go ahead change that API from the front end? What if you didn’t have to worry about whether a particular endpoint exists — if the data is there, you can construct one yourself. Even better if you could keep that app trim by only getting the data you want when you make a request, instead of a ton of bloat.
To that end, I’ve been considering whether to recommend a GraphQL API instead of a REST API to an e-commerce client. It’s important that they be able to move quickly. While doing my research, I came across some interesting resources:
Starting from this excellent article
GraphQL query visualizer
GraphQL server for Express
Interesting how GraphQL is just an interface on top of whatever database you want to use. For example, you can use it with MongoDB https://www.compose.com/articles/using-graphql-with-mongodb/
The spec itself, although I’d start with articles and tutorials instead
Some front end tools for interacting with GraphQL
Specifically for React
A whole framework from Facebook that couples React with GraphQL
Modifying control flow of queries and results
Shorthand for writing queries
Live Streaming 360 Video to YouTube From Inside of Unity
I’ve been trying to sort out a pipeline to live stream equirectangular video captured from a camera inside of a Unity scene to YouTube 360. I want viewers to be able to see inside of live game worlds as they’re happening. I’ve found a couple of useful leads:
Here’s one to output using FFMPEG (which could then stream to YouTube)
via the amazing Keijiro
But at what speed? The new scriptable render pipeline in Unity 2018 might enable a faster solution
I also evaluated a paid option
The paid option makes use of this very interesting concurrency library
Assorted Deep Learning Finds
Generate poetry from images using CNNS and RNNs
Get audio into a Pytorch tensor — more audio Deep Learning projects, please!
Speech at super-resolution could lead to clearer communications.
Make music alongside a Google AI — a new generation of instruments and live performances may be collaborations with a computer
Future Alert: with the right kind of sensors, deep learning should enable subvocal speech recognition — the ability to communicate without making sound, which would come in handy in smart gear for divers or firefighters
Many state-of the art networks have been trained on extremely small images (300px by 300px) due to GPU memory size constraints— however, they have been very accurate despite this limitation. I’m excited about anything trained on high resolution images, since it will help us understand whether smaller training images are sufficient or only scratching the surface of what’s possible. More pixels!
Unity is a great environment to train deep learning agents — especially multi-agent environments.
Here’s a hide/escape example
Here’s one with pirates!
Source: Deep Learning on Medium