Camera Testing: A How To Guide

Camera testing can be a very personal endeavor as the results one person is after may not be the same results that another person is after. It seems that no matter how carefully a test is done, there will always be critiques of how a test was conducted, and how someone else could have done it better, or that the test should have been done differently. My goal in this post is to show some test scenarios, explain how to set them up, so that you can more effectively test out your camera system for comparison or in preparation for production.


There are two important rules that govern all testing:

• Control everything and limit the variables to one item.
• Good testing is all about controlling everything possible in the setup, and then only introducing one variable. The more variables that get introduced, the higher probability that the other variables will impact the final results - even if you are not testing for that variable. For example, say you have a setup for testing out filtration and the light levels in the scene keep changing. Even though you are not testing for dynamic range of the camera system, the increasing and decreasing light levels of the scene will influence your perception of how the filter behaves. It is only the filter that should change. If you want to see different light levels with the filtration, then those varying light levels should be tested independently so that they can be consistent for each setup. Control, control, control .... 

• Document everything
• Everything should be documented as much as humanly possible. If you do not have detailed notes about the setup, then when you process the footage the results will be meaningless. Additionally, if you find something wrong with the footage and you have not documented everything, you will not know where the problem came from. Keep records, and keep them with the footage for reference when viewing the results.

Test 01: Dynamic Range (Reflected)

Equipment Needs
- Two Diffused Light Sources (Ex. 650w fresnel + 4x4 frame w/Full Grid)
- Sekonic Exposure Profile Chart II
- Light Meter

Setup
- Place two light sources of equal intensity at 45 degrees to the chart.
- The light should fall evenly across the face of the chart. Use the light meter (Incident mode) to verify that the light does not fluctuate more then 1/10 of a stop across the chart.

Overhead Lighting Diagram

Testing Notes To Take (At a minimum)

- Camera

- Lens

- T-Stop of lens

- Camera settings

- Incident reading at the charts position (This determines what 0 EV equals)

- Spot reading of center chip (This helps to calibrate the light meter using the Sekonic software)

Testing
Shoot the chart at the following exposure values: 0 EV*, -3 EV, +3 EV, -6 EV, +6EV. This will cover a total dynamic range of 16 stops (8 over and 8 under), which is more then current digital technology can record. If you want to play it really safe, you can add in -9 EV & +9EV for a total range of 22 stops (11 over and 11 under).

*EV = Exposure value.

For the following example, let's say the chart was lit to F8 at ISO 800, 24 fps, 180 degree shutter.

0 EV = T8 at ISO 800, 24 fps, 180 degree shutter

-3 EV = T22 at ISO 800, 24 fps, 180 degree shutter

+3 EV = T2.8 at ISO 800, 24 fps, 180 degree shutter

Notes
Printed charts like this one has its' limitations due to the capability of printing technology to accurately print a wide range of values. This type of chart is most effective for camera systems capable of capturing around 10 - 12 stops of dynamic range. While there are more effective charts to use for cameras with higher dynamic range, I still find this an effective chart to shoot as it shows how reflected values in a scene will reproduce.

The system you are viewing the footage on will also effect the results. If you are viewing the footage on an uncalibrated system, it is very likely that it could be limiting what you are seeing, effecting the results and conclusions you are drawing. Additionally, current display technology is not capable of displaying high dynamic range (HDR) imagery. How much range you can see on your display will be determined by the specs of that display and it being set up correctly. It is important to know what these limits are when viewing your footage.

Evaluating The Results

Each chip on the chart represents 1/6 of a stop difference. So when you are viewing the chart you can count the chips up and down the exposure range to see the limits of the camera system.  It is also helpful to take the files into an editing / grading application that will allow you to see the results on a waveform. Here you will be able to view a graph of the results and tell more easily where the range starts and stops. The last thing you may want to consider doing, is taking the final image into Photoshop and cutting out the chips to place them immediately next to each other. Sometimes this will make it easier for you to visually see a difference between them. I have also found that there are times where the waveform will show little to no change between 2 stops on the low end, but I can see a difference between the two on the screen. (This extra stop is probably not usable, but it is still there.)

Test 02: Dynamic Range (Transmission)

Equipment Needs
- One Diffused Light Sources (Ex. 650w fresnel + 4x4 frame w/Full Grid)
- Stouffer T4110 Chart
- Black foam core to mount the chart
- Black foam core to create a tunnel to place around the chart
- Light Meter

Setup
- The stouffer chart is a back lit chart, which means that the chart needs to have a soft even light from edge to edge.
- Place the light source behind the chart & use the light meter to verify that the chart is evenly lit. The light should not fluctuate more then 1/10 of a stop. On a chart this small, you should be able to have no fluctuation at all.
- When framing the chart, you will want it to be as large in the frame as possible. The smaller it is, the more difficult it will be to evaluate the results.
- It is best to shoot this chart at the sweet spot of a lens, which is usually 2 stops down from wide open.
- Absolutely NO light can fall on the face of this chart as it will invalidate the results. Be sure to create a black tunnel around the camera, and take any additional steps to ensure no light bounces from the room back onto the chart.

Overhead Lighting Diagram

Testing Notes To Take (At a minimum)

- Camera

- Lens

- T-Stop of lens

- Camera settings

Testing
When the chart is exposed only the very first chip should clip. Adjust the light as needed to ensure that this happens. If you are having trouble determining if only the first chip is clipping, you can slowly roll the aperture up and down and then use the waveform in your editing / grading application to determine the frame where only the first chip clips.

Notes
For best results use a high quality lens, and have it stopped down to prevent lens artifacts from introducing variables. Lower quality lenses can introduce vignetting when they are wide open or stopped down to far. This vignetting will result in skewed results. To verify that the lens is not vignetting, you can pan the camera back and forth across the chart, watching the results on a waveform. The clipped chip should not lower in exposure as it passes across the edge of the lens.

This test will allow you to see a range of 13 1/3 stops. The chart will work well for many camera systems. However, for even more accurate testing over a higher dynamic range I suggest using either the DSC 102Db (18 Stop) Chart or the Arri Dynamic Range Chart.

Evaluating The Results
This chart has 41 steps that are 1/3 of a stop in difference, making for a total range of 13 2/3 stops. The first chip is set to clipping, so it is not counted, as this is the exact clipping point. Take the results into your application of choice, and view the it on a waveform. Starting at chip two, count down the steps until  they all become indistinguishable. Divide this number by 3 and you have the total dynamic range of the camera system. As with the previous test, you can take the image into Photoshop and cut it up to make viewing the results easier. Just be sure to keep everything in its native state, otherwise you could be introducing more variables into the equation.

Test 03: Dynamic Range (Real World)

Equipment Needs
- A Basic Light Kit (One Example)
- 18% grey card (Back of Sekonic Chart)
- X-Rite Passport

- Light Meter

Setup

There are a million and one ways to setup and light this test. How you get there and what tools you choose to use will be a function of what you have available to you and the time you have to run the test. The goals you want to accomplish in this setup are to have areas of texture in underexposure, areas of texture in overexposure, skin tones, mid tone reference, color reference, and areas of high contrast. For the example above, I am wearing a black ribbed shirt for the underexposure texture, and the white shirt and off white cloth have texture in them to give me my overexposure texture. (And my hair, or what is left of it anyway, has some texture / detail in it too.) The 18% grey card gives me my mid tone and the X-Rite Passport gives me the color reference I need. The white and black cloth gives me a line of contrast, and the silver case gives me some more points of highlight in the image.

Overhead Lighting Diagram

(One possible setup of many)

Testing Notes To Take (At a minimum)

- Camera

- Lens

- T-Stop of lens

- Camera settings

- Incident readings of multiple points in the scene

- Spot readings of multiple points in the scene

Testing

This test will involve multiple exposures, the first of which will be at 0 EV. Take a reading using your meter of the key light (incident mode) or off of the grey card (spot mode). Now without adjusting the lights, increase and decrease the exposure levels at consistent increments. (Ex. T2.8 / T2.8 1/3 / T2.8 2/3 T4 ...) For very precise measurements, you'll want to shoot exposures variances of +/- 1/3 EV. Usually +/- 1/2 EV variances will be adequate, and if you are pressed for time, you can do +/- 1 EV, but that will yield less precise results.

Notes

While the previous two dynamic range tests are more precise and accurate for determining the dynamic range of a camera system, I find this kind of test to be more helpful. Personally, I'll use the first two tests to create a profile for my meter, and to calibrate it. I'll then use this test to evaluate how the camera responds to a real life scene, and how it handles under and over exposure. So this is the most practical test in my opinion.

Evaluating The Results

When evaluating the footage from this test, I'll review it through two different passes. The first pass I'll use is uncorrected. I'll use my notes to see how all of the details hold through the range of exposures. Then for the second pass, I'll have the footage corrected back to normal. For example, if the clip is 1 stop over exposed, I'll grade it to match the levels of the correct exposure. When I view this pass, I'll be looking at how the footage holds up after correcting for exposure levels. This helps me to see how well the footage will respond to varying exposure levels and if there are any anomalies that I need to be aware of when over or underexposing the camera.

Test 04: Lens Test (Chart)

Equipment Needs
- Two Diffused Light Sources (Ex. 650w fresnel + 4x4 frame w/Full Grid)
- ISO 12233 Test Chart
- Kodak Color Separation & Grey Scale Large

- Light Meter

Setup

- The Kodak chart should be mounted to the ISO chart as seen in the example above
- Add a dry erase area to the chart to make notes in the frame as seen in the example above
- Place two light sources of equal intensity at 45 degrees to the chart.

- The light should fall evenly across the face of the chart. Use the light meter (Incident mode) to verify that it is even. While it is not critical that the chart be with in a 1/10 of stop tolerance, the less the difference edge to edge, the better. ( I light mine to within 1/10.)
- Focus, check focus, and then check focus again!

Overhead Lighting Diagram

Testing Notes To Take (At a minimum)

- Camera

- Lens

- Lens Serial Number

- T-Stop of lens

- Camera settings

Testing

Shoot the chart at exposure over multiple exposure ranges. For example, the chart shout be lit and exposed correctly for T2.8, then for T4, then T5.6, etc. Full stop increments are fine for this test, unless you want to get super critical in evaluating the performance of the lens at finer increments. Be sure that you are focused on the chart. If you are off, even a little, this will impact the final results.

Notes

If you use the PDF version of the chart, I recommend having Kinko's / FedEx, or somewhere professional print and mount it for you. It needs to be perfectly flat, and printed on a high quality printer. This should run you about $30. However, even if you go this route, this method cannot be used as a reliable resolution test. There are too many factors in the printing process to account for accurate results. If you want accurate resolution results, you will need to buy the official chart ($100+). The home brew version of the chart is good for checking the big picture of edge to edge sharpness, contrast, overall sharpness, and color rendition.

Evaluating The Results

When you evaluate the results, you will want to see them as large as possible, projected would be the best option, a large plasma would be the next best option. If you are on your laptop, then open up the files in photoshop and view them at 100%. When you are looking at the results, you will want to pay attention to how sharp the image is over all, as well as if it is sharp in the corners. As the lens is stopped down pay attention to how the contrast changes by watching the grey scale and paying attention to how sharp the lines feel on the chart. You can use the color chips in the chart to evaluate on a vector scope how the lens renders color, and compare those results to other lenses you have tested.

Test 05: Lens Test (Real World)

(One possible setup of many)

Equipment Needs

- A Basic Light Kit (One Example)

- X-Rite Passport

- Light Meter

Setup

As with the real world dynamic range test there are a million and one ways to stage and light this set up. How you get there and what tools you choose to use will be a function of what you have available to you and the time you have to run the test. Areas that you will want to be able to evaluate are: sharpness, contrast, bokeh, focus fall off, and color rendition. I used news paper, the black and white background, the slate, and the black shirt over a white shirt to help show off sharpness and contrast. The Christmas lights help to evaluate bokeh, the X-Rite Passport helps with color, talent helps with skin tones, and the depth of the scene helps with focus fall off.

Overhead Lighting Diagram

(One possible setup of many)

Testing Notes To Take (At a minimum)

- Camera

- Lens

- Lens Serial Number

- T-Stop of lens

- Camera settings

Testing
Shoot this scene at exposure over multiple exposures just like the previous lens test.

Notes

The quickest way to make this test happen is to not use a matte box. Instead, use a 2x3 solid to flag off the lens to ensure that you are not flared by the light. This will help you change the lenses quicker, cutting down on wasted time. I would also recommend changing the light levels through the use of scrims and nets. Light the scene to the slowest stop, and then add scrims and nets to get to the fastest stop. If you use a dimmer, there will be a color shift introduced to the light which will invalidate the results.

Evaluating The Results
These results need to be seen as large as possible, just like the previous lens test.

Test 06: Compression (High Frequency Detail)

Equipment Needs
- Overcast day with even lighting

- Light Meter

Setup

- Find a scene with high frequency detail in it, like lots of tree branches, for example. Alternatively, you could setup a scene inside with a lot of fine details, but I find exteriors much easier to use.
- Be sure that the light levels do not change while you shoot these clips. (Overcast day)
- The shooting stop should be at least a T5.6
- Focus needs to be set correctly

Overhead Lighting Diagram

Testing Notes To Take (At a minimum)

- Camera

- Lens

- T-Stop of lens

- Camera settings (This includes compression levels)

Testing

Shoot 5 second clips of the same subject at all of the various compression settings that your camera allows.

Notes

Focus and light levels are key. Check, and double check,

Evaluating The Results
These results need to be seen as large as possible, just like the lens tests. Only this time look for how sharp the details of the image are rendered at the various compression levels.

Test 07: Compression (Noise)

Equipment Needs

- A Basic Light Kit (One Example)

- Light Meter

Setup
Create a scene that has multiple levels of underexposure. It doesn't have to be anything fancy, but you do need a range of underexposure values.

Overhead Lighting Diagram

Testing Notes To Take (At a minimum)

- Camera

- Lens

- T-Stop of lens

- Camera settings (This includes compression levels)

Testing

Shoot 5 - 10 second clips at various compression levels.

Notes

If you decide to find a scene, rather then create one, just be sure that the light levels remain consistent from shot to shot.

Evaluating The Results
I run this test through the two pass process. The first pass being the footage as it is unadjusted. The second pass I crank up the exposure in the grade so that I can see the noise more easily. What you are looking for is how the noise is rendered depending on the compression settings. These results also need to be seen as large as possible.

But Wait, There Is More ...

These examples are meant as just a starting point to jump off from. If you have other areas you want to explore, by all means test it out. Pull from these example tests and use as you see fit. Testing can be as much of a personal endeavor as is the creative process of shooting a project. So use what works for you- just be sure to have controls over your testing, and take good notes. If you want more ideas on other types of test, and to see examples of some more stellar testing check out Alfonso Parra, AEC's site, as he is a master of the camera test. (7D Testing here.)

Until Next Time - Get Out There And Shoot!

Ryan E. Walters, Cinematographer