Building JPEG/Motion JPEG Into The Security Camera

The Imaging Solutions Group (ISG) recently announced the availability of cameras with on-board JPEG/Motion-JPEG compression to reduce bandwidth in the Internet Monitoring, Security and Surveillance Industries and Military Applications.

The Security & Surveillance industry is asking for higher and higher resolution images. The need for JPEG/Motion-JPEG compression is growing rapidly. Individual frame data is preferred to MPEG based algorithms because of its ease of use in reconstructing individual frame data for law enforcement purposes.

The existing infrastructure for the Security & Surveillance industry is based on CCTV standards. They are now beginning to migrate towards 10/100 Ethernet solutions. Both of these standards have limited bandwidth for today''s higher resolution video cameras. While elaborate video systems exist, they are primarily proprietary in nature and very costly. The industry needs solutions that fit within the constraints of existing and near term technologies that are low-cost and easily integrated into today''s infrastructure.

The Imaging Solutions Group is designing and manufacturing solutions that target these markets. The ISG LightWise Camera series is becoming popular in such Security & Surveillance industries.

The compression algorithm in these new cameras from ISG is described below:

General description

The Imaging Solutions Group (ISG) recently introduced a JPEG/Motion-JPEG compression option in their cameras. This JPEG/Motion-JPEG compression enables high-quality high-resolution images to be streamed over industry standard interfaces such as IEEE-1394 and 10/100 Ethernet, etc.

The JPEG algorithm designed for the ISG LightWise Standard Product Cameras is a streamlined version of a 5th generation design. ISG engineers have been compression specialists for two decades and their expertise is available as an engineering service to OEM customers throughout the world. Custom algorithm development is a major part of ISG''s business model.

The ISG Image Processing Pipeline takes in Bayer Data from a Sensor, converts to Y Cb Cr 4:2:2 and compresses according to the Industry Standard JPEG format.

The JPEG frames are stored in a circular buffer in external memory. Upon completion of a JPEG, compressed frame an interrupt is provided to CPU. Registers are provided to indicate the start address of the JPEG image as well as the size. This allows the firmware to easily manage a chain link list of JPEG images for CPU and network access. The image processing core can operate up to 48 MHz . The JPEG core is currently designed to operate at 85.0 MHZ. At this frequency the pipeline will be able to sustain 1280x1024 at 30 fps.

Block Diagram and Detailed Description

BLOCK DIAGRAM

Detailed Description

This streamlined version of the JPEG algorithm design is described below, block-by-block. ISG can custom tailor this algorithms according to specific customer requirements. Each block is individually designed and a variety of versions of each block are available under contract with ISG. The implementation found in the ISG LightWise Standard Product Cameras was designed for simplicity, cost-effectiveness and compatibility.

Sensor Interface Block

The sensor interface block provides programmable inputs for control of multiple CMOS image sensors. The interface provides support for both sync and level type control inputs for vertical. The output of this block will be 10-bit RGB (Bayer Pattern), line valid and frame valid. This block will also be used to define the location of the dark (optically shielded) pixels for use in the Black Level Control block. When dark pixels are present it is necessary to differentiate between dark pixels and valid video pixels by programming the region registers.

White Balance Gain

The White Balance Gain module provides a method of equalizing the Red and Blue pixels so that white areas (Red = Green = Blue) appear white.

Auto Exposure Histogram

This block stores a histogram, which can be used to control exposure (integration time and/or gain). The histogram is a 256-bin implementation, generated from the full frame of image data.

Color Interpolator

The Bayer Pattern interpolation core reconstructs RGB data from an imager with a Bayer color filter array pattern. The input data is up to 10 bits per sample of data captured from an image sensor utilizing a color filter array organized in alternating rows of green/red samples and blue/green samples as shown in Figure 1 below.

The output is 10-bit red, green, and blue data values for each pixel location. The missing green data is computed from a median average of the nearest neighbours. Red and blue data are computed from bilinear interpolation of nearest neighbours of the same color. For image path testing, a bypass mode is available, where the output RGB data is formed by passing the input data to all three channels.

Figure 1 Bayer Pattern

Auto White Balance Assist

Two Auto White Balance Algorithms are supported by the White Balance Assist: Word Gray Assumption (WGA) and White Pixel Discrimination (WPD). WPD is essentially, an enhanced WGA, which will give better results in most cases.

World Gray Assumption: This algorithm involves accumulating Red, Green, and Blue values over the image, or any window of the image, and computing the Red, Green and Blue gains in the CPU based on the hardware accumulated values. This method makes the basic assumption that on a naturally illuminated scene, the "average color" is gray or white, and the light source represents natural white.

White Pixel Discrimination: This algorithm involves a first step of discriminating the white pixels in a scene and only accumulating the pixels which pass a criterion for "white".

Color Correction

The Color Correction module adjusts the relative levels of the Red, Green and Blue pixels so that colors are accurate (i.e. Red appears red).

Gamma Correction

The transfer function of the display device produces output intensity that is proportional to some power (usually about 2.5 for CRT and referred to as gamma) of the input voltage. As a result, high intensity ranges are expanded and low intensity ranges are compressed. By "gamma correcting" the video before being used for display, the intensity output of the display device is roughly linear.

The gamma correction is performed by using Look Up Tables (LUT). Input pixels (10 bit) are used to address into a LUT (one for each color plane) and the contents of the addressed location are output as the new pixel. The tables can be programmed to form any shape of transfer function.

Color Space Converter

The Color Space Converter converts 10 bit RGB pixels into 8 bit YCbCr pixels for use in JPEG compression or generation of NTSC/PAL video.

DCT (Discrete Cosine Transform)

The DCT analyzes the pixel data in the spatial domain for each 8x8 block and classifies it contents by creating coefficients of the frequency domain. These coefficients can be most easily grasped as how much of a particular spatial frequency is created in an 8x8 block. To accomplish this, multiply and addition operations are used to create 64 coefficient. The implementation used for LTI is a full baseline DCT that is pipelined and multipliers shared in such way to allow a Coefficient to be generated every clock cycle.

Quantizer and ZigZag

Quantization is the first of two stages where the storage requirements of the DCT coefficients are reduced in size, and where the most control is offered over output images sizes. Each of the 64 coefficients of an 8x8 block is divided by its own Quantization factor. Two 64 deep, programmable memories are used in the jpeg cores to enhance compression efficiency or image quality. With compression efficiency, image quality is the trade off.

Quantization enhances compression by:

o Reducing the Coefficient magnitude. This will allow the Huffman encode process to use fewer bits to represent the coefficient.

o Increasing the number of coefficients that are Zero in magnitude thus creating larger run-counts.

Zig-Zag enhances compression by:

o Zig-Zag reorders the coefficients by clustering the Zero results together. This also increases the run-counts.

The range of these 64 different Quantization factors (Q factors) is from 1 to 255. Lower Q factors are used on components that are more significant and higher Q factors on less significant components. To allow for this the tables are typically programmed to use one table for the Y component and the second table for the Cb and Cr component.

To show the range of compression efficiency by varying the Q tables, a table is supplied by compressing several images with varying "frequency''s" with varying Q factors. For simplicity, we will have a quality Factor from 1 to 10 where 10 are setting all the Q factors to one. Setting the Q factor to one will create larger output images but with the best image quality.

Example 1: gto.jpg . Quality Factor = 5.0

Example 2: alfa.jpg , Quality factor = 5

Huffman Coding

Once the 64 DCT coefficients are quantized, they are then Huffman encoded. A fixed Huffman table is supplied that duplicates the standard default in the jpeg industry. Huffman encoding essentially combines a zero-run length counter and a method to encode the non-zero quantized coefficient. This produces two pieces of data that must be encoded in the smallest number of bits, whose method is described below.

Count the number of continuous zeros in the 64 coefficients. This is the zero-run length count portion of the Huffman symbol to be coded.

Once a non-zero coefficient is found the coefficient must be coded in such a way to save the sign and value in the least number of bits. Huffman uses a set of variable word-length code symbols that represent these two pieces of information for the AC coefficients. DC coefficient is easier to represent since no run-length information is contained. To accomplish optimal Huffman coding efficiency a Huffman table is included for coding the one DC coefficient and a Huffman table is provided to code the remaining 63 AC coefficients.

Summary

This summary of the ISG LightWise JPEG algorithm describes what takes place to the image data prior to transmission. Transmitting image data over limited bandwidth interfaces is required for high-resolution cameras found in the Internet Monitoring and Security & Surveillance and Military applications.

This algorithm is available on all ISG LightWise Cameras and also available for custom camera development.

JPEG COMPRESSION core features

Conforms to IS 10918-1 and IS 10918 -2 baseline encoding standard.

High bandwidth rates. Core can achieve bandwidths to 85 Mega-Pixel components per second. Huffman encoder will encode data at 85 Mega-code symbols per second under any worst-case image. Multi-frame capture capability for AVI support. Bandwidth monitoring for Motion JPEG. Size limit control enabled by Huffman bit counting. Two programmable and selectable Quantization tables. Two selectable Huffman tables. 4:2:2 sub sampling enabled. Asynchronous data rate support. Input can be throttled by slow output with dataready signal. Input can send data every clock or in bursts. End of page interrupt handling.

********************************************

for more information, contact:

Kerry Van Iseghem

1387 Fairport Road, Suite 890

Fairport, NY 14450

585-388-5220

kerry@isgchips.com