Last week, Insight Media and about 35 other SID members had a chance to visit with Consumer’s Union, the publishers of Consumer Reports (CR). At their facility in Yonkers, NY, we learned more about how they test HDTVs and saw some demos on an array of TVs. It was one of the best-attended SID local chapter meetings in years and we learned some meaningful information.

Chris Chinnock
Senior Analyst and Editor
for Insight Media

First, CR needs to select which TVs to test. To do this, they have a team whose job is to track the models in the market along with planned new models. From selection to printing of test results can easily take 3-4 months, although CR is trying to tighten this time. Therefore, test results need to done on products that are in the market, and likely to be in the market for a while after the review appears on-line or in print. The organization can also do what they call a "Quick Look" at a new technology, product or feature, which can be turned around in days, but is clearly not a comprehensive review.

What I was surprised and delighted to learn is, how much testing goes on behind the scene to arrive at the published ratings. In general, this process consists of doing some basic TV set up tasks, a series of measurements over several inputs, and even a panel review in side by sides with reference TVs. The TVs are always running content in the lab, so every time a tester passes by one and they note a particular artifact or performance level, they double check their ratings to be sure they got it right. This is the "living with the TV" phase, which helps to identify any quirks and helps to validate the test and rating results.

Our group was ushered through the TV testing labs back to a small meeting room in which a number of TVs were set up - all running the same content for easy side by side comparison. As I recall, there was a Pioneer PDP, Panasonic PDP, Panasonic LCD, ViewSonic LCD, Mitsubishi LaserVue DLP, Samsung LCD (with W-LED BLU), the Sony 11" OLED TV and perhaps one more.

CR takes the attitude that they will make adjustments that ordinary consumers can make to the TV, and they will measure and rate the most common items that consumers care about. This seems perfectly appropriate, although it does put them a bit behind the curve in terms of evaluating new features and capabilities.

The basic set up and measurement process consists of first setting the white point to D65 at IRE level 40. Then, using a series of low IRE black squares, the brightness level is set to just see separation in the darkest black levels. Then, the contrast is adjusted to allow the brightest white gray shades to just be discerned. Note, that the "brightness" and "contrast" adjustment terms are used here as these are hold-overs from the CRT era. Getting better terms for these adjustments is way overdue.

The sharpness of the TV is adjusted using a sine wave burst pattern to avoid dark bands in the burst patterns. Color bar patterns are then used to adjust the saturation and hue. Only the user accessible menu controls are used in this phase.

Measuring the viewing angle used to be done using a series of luminance measurements at various horizontal angles from the TV normal. Now, reviewers do a more subjective assessment of the image quality in the up/down and left/right orientations. They are looking for color desaturation, color shifts, image dimming, increasing in the black levels, picture washout, image inversion and more.

In addition, the group also looks at the electronics and how well they handle various sources of content. As a final test, the TVs are benchmarked against others TV and reference TVs and evaluated by a panel of viewers. These multiple measurements and rating are all boiled down to a series of metrics, that are then published by Consumers Union.

As a demo, CR showed a number of clips of various pieces of content. What was very clear from this demo is that each TV and/or TV technology has its strengths and weaknesses and that none excelled in all areas. For example, the Pioneer PDP blew away the competition on one dark scene from the movie "Gladiator."

The Samsung LCD with the LED backlights showed bad haloing with white lettering on a dark background. But it also had an adjustable motion compensation feature that allowed for insertion of motion-compensated images in the 120Hz frame rate sequence. One clip showed film content without this feature on that exhibited typical film motion blur and some judder that was an artifact of the 3:2 pulldown conversion. But with this motion compensation added, the scene motion smoothed out and became decidedly much more three dimensional in the process. I found this much more pleasant to look at, but others complained that it now looked "like video" instead of film. This was true, so it now becomes a matter of personal preference.

Other demos pointed out other flaws or quirks, reinforcing the notion that the right TV choice is very dependent upon the viewing environment, personal preferences and avoidance of quirks and flaws. All in all, a very good evening’s edutainment.