Sony to Release World’s First Intelligent Vision Sensors with AI Processing Functionality

Written by Ben

May 16, 2020

TOKYO, May 14, 2020 /PRNewswire/ — Sony Corporation today announced the upcoming release of two models of intelligent vision sensors, the first image sensors in the world to be equipped with AI processing functionality.*1 Including AI processing functionality on the image sensor itself enables high-speed edge AI processing and extraction of only the necessary data, which, when using cloud services, reduces data transmission latency, minimizes any privacy concerns, and reduces power consumption and communication costs.

Intelligent Vision Sensors, Left: IMX500 Right: IMX501
Intelligent Vision Sensors, Left: IMX500 Right: IMX501
Data output format selectable to meet various needs
Data output format selectable to meet various needs
Example of real-time tracking with product and task at a register
Example of real-time tracking with product and task at a register
Example of camera usages in a facility
Example of camera usages in a facility
*1 Among image sensors. According to Sony research (as of announcement on May 14, 2020).

These products expand the opportunities to develop AI-equipped cameras, enabling a diverse range of applications in the retail and industrial equipment industries and contributing to building optimal systems that link with the cloud.

Model name Sample shipment date Sample price (excluding tax)
IMX500 1/2.3-type (7.857 mm diagonal) approx. 12.3 effective megapixel intelligent vision sensor (bare chip product)April 202010,000 JPY
IMX501 1/2.3-type (7.857 mm diagonal) approx. 12.3 effective megapixel intelligent vision sensor (package product)June 2020 (planned)20,000 JPY

The spread of IoT has resulted in all types of devices being connected to the cloud, making commonplace the use of information processing systems where information obtained from such devices is processed via AI on the cloud. On the other hand, the increasing volume of information handled in the cloud poses various problems: increased data transmission latency hindering real-time information processing; security concerns from users associated with storing personally identifiable data in the cloud; and other issues such as the increased power consumption and communication costs cloud services entail.

The new sensor products feature a stacked configuration consisting of a pixel chip and logic chip. They are the world’s first image sensor to be equipped with AI image analysis and processing functionality on the logic chip. The signal acquired by the pixel chip is processed via AI on the sensor, eliminating the need for high-performance processors or external memory, enabling the development of edge AI systems. The sensor outputs metadata (semantic information belonging to image data) instead of image information, making for reduced data volume and minimizing any privacy concerns. Moreover, the AI capability makes it possible to deliver diverse functionality for versatile applications, such as real-time object tracking with high-speed AI processing. Different AI models can also be chosen by rewriting internal memory in accordance with user requirements or the conditions of the location where the system is being used.

Main Features

 World’s first image sensor equipped with AI processing functionality
The pixel chip is back-illuminated and has approximately 12.3 effective megapixels for capturing information across a wide angle of view. In addition to the conventional image sensor operation circuit, the logic chip is equipped with Sony’s original DSP(Digital Signal Processor) dedicated to AI signal processing, and memory for the AI model. This configuration eliminates the need for high-performance processors or external memory, making it ideal for edge AI systems.

■ Metadata output
Signals acquired by the pixel chip are run through an ISP (Image Signal Processor) and AI processing is done in the process stage on the logic chip, and the extracted information is output as metadata, reducing the amount of data handled. Ensuring that image information is not output helps to reduce security risks and minimizing any privacy concerns. In addition to the image recorded by the conventional image sensor, users can select the data output format according to their needs and uses, including ISP format output images (YUV/RGB) and ROI (Region of Interest) specific area extract images.

■ High-speed AI processing
When a video is recorded using a conventional image sensor, it is necessary to send data for each individual output image frame for AI processing, resulting in increased data transmission and making it difficult to deliver real-time performance. The new sensor products from Sony perform ISP processing and high-speed AI processing (3.1 milliseconds processing for MobileNet V1*2) on the logic chip, completing the entire process in a single video frame. This design makes it possible to deliver high-precision, real-time tracking of objects while recording video.

*2 MobileNet V1: An image analysis AI model for object recognition on mobile devices.

■ Selectable AI model
Users can write the AI models of their choice to the embedded memory and can rewrite and update it according to its requirements or the conditions of the location where the system is being used. For example, when multiple cameras employing this product are installed in a retail location, a single type of camera can be used with versatility across different locations, circumstances, times, or purposes. When installed at the entrance to the facility it can be used to count the number of visitors entering the facility; when installed on the shelf of a store it can be used to detect stock shortages; when on the ceiling it can be used for heat mapping store visitors (detecting locations where many people gather), and the like. Furthermore, the AI model in a given camera can be rewritten from one used to detect heat maps to one for identifying consumer behavior, and so on. 

Key Specifications

Model nameIMX500 (bare chip product)IMX501 (package product)
Number of effective pixels4056 (H) × 3040 (V), approx. 12.3 megapixels 
Image sizeDiagonal 7.857 mm (1/2.3 type)
Unit cell size1.55 μm (H) × 1.55 μm (V)
Frame rateFull pixel60 fps
Video4K (4056 × 2288) 60 fps
1080p 240 fps
Full/video+AI processing30fps
Metadata output30fps
Sensitivity (F5.6 standard value)Approx. 250LSB
Sensor saturation signal level (minimum value)Approx. 9610e-
Power supplyAnalog2.7V
Digital0.84V
Interface1.8V
Main functionsAI processing function, ISP, HDR shooting
OutputMIPI D-PHY 1.2 (4 lane) / SPI
Color filter arrayBayer array
Output formatImage (Bayer RAW), ISP output (YUV/RGB), ROI, metadata
PackageCeramic LGA12.5 mm (H) × 15.0 mm (V)

About Sony Corporation
Sony Corporation is a creative entertainment company with a solid foundation of technology. From game and network services to music, pictures, electronics, semiconductors and financial services – the Sony purpose is to fill the world with emotion through the power of creativity and technology.

For more information, visit: http://www.sony.net

SOURCE Sony Electronics Inc.

Related Articles

Contribute to NASA Research on Eclipse Day – and Every Day

Contribute to NASA Research on Eclipse Day – and Every Day

NASA is celebrating the Sun during the Heliophysics Big Year, which extends through the end of 2024. You can get involved to help us learn more about our star and its influence on our planet. With exciting experiments happening during the total solar...

Check out our Amazon Store

Lookin in our Amazon Store and pick up the items we review and talk about (if you purchase something from our store, we earn a small comission)

Join Our Newsletter

Click below and never miss a thing