Sony image sensors with AI unveiled

Original article was published on Artificial Intelligence on Medium

Sony image sensors with AI unveiled

Makes cloud computing more accessible and local processing more powerful

Sony has announced two new image sensors — the IMX500 and IMX501 — that stack a traditional sensor on top of an AI-powered chip designed to process video and images. On paper, the idea isn’t all that new, and companies like Apple and Google have been doing it with their smartphones for years.

From dimly lit restaurants to moonlit beaches, the new Night mode uses intelligent software and A13 Bionic to deliver low‑light shots never before possible on iPhone. And it all happens automatically. You can also experiment with manual controls to dial in even more detail and less noise.

What’s different here, however, is the stacked nature of these chips, which reduces precious component space inside of phones, freeing up room for larger batteries or other chipsets that tackle different tasks, like AR. If you’ve ever seen photos¹ from a phone’s innards, you’ll know that nearly every inch of it is accounted for. Freeing up space on its own can have enormous implications for smartphones, but what the IMX500 and IMX501 image sensors offer goes beyond gained space inside of our phones and has use cases in other devices as well.

Devin Coldewey from TechCrunch:

To be clear, image sensors already have companion processors that do the usual work of sorting pixels, compressing them into a JPEG, and so on. But they’re very focused on performing a handful of common tasks very quickly.

The Sony chip, as the company explains it, is capable of more sophisticated processes and outputs. For instance, if the exposure is of a dog in a field, the chip could immediately analyze it for objects, and instead of sending on the full image, simply report “dog,” “grass” and anything else it recognizes.

It also could also perform essentially improvisational edits, such as cropping out everything in the photo but parts it recognizes and has been told to report — only the flowers, but never the stems, say.

The benefit of such a system is that it can discard all kinds of unnecessary or unwanted data before that data ever goes into the main device’s storage or processing pipeline. That means less processor power is used, for one thing, but it may also be safer and more secure.