Wednesday 28 March 2012

Validating MTF Mapper accuracy, part 1

Validation of synthetic images generated by mtf_generate_rectangle

The first step in the validation of MTF Mapper accuracy is to generate images with known MTF curves --- this is why I have developed (and included) the mtf_generate_rectangle tool.

But before I can use the mtf_generate_rectangle tool, I have to test whether the images generated by the tool indeed have the desired edge profile. Fortunately, this is relatively straightforward if the synthetic images are generated using a Gaussian Point Spread Function (PSF). The PSF defines the kernel with which each pixel in the synthetic image is sampled. A step edge (such as the black-> white edges used in the slanted edge method) sampled with such a Gaussian PSF will produce an Edge Spread Function (ESF) with a known functional form. The slanted edge method usually computes the PSF by taking the derivative of the ESF, i.e., the ESF must therefore be the integral of the PSF. For a Gaussian PSF, we expect an ESF in the shape of the integral of a Gaussian, also called the error function, or erf. 

To test whether mtf_generate_rectangle produces correct images, all I have to do is measure the empirical ESF, and compare it to the expected analytical erf function. With mtf_generate_rectangle I thus generated a synthetic image with an expected MTF50 value of 0.5, and measured the ESF empirically using mtf_mapper. Then I plotted the measured ESF along with the expected analytical ESF, which will have the form erf(x/0.53002):
At this scale, the measured ESF appears to match the analytical ESF perfectly, but we have to look more closely. A better visualisation is to plot the difference between the two ESFs:
I chose a percentage scale for the y-axis to better convey the magnitude of the differences. We can see from the plot that the difference between the two curves is less than 0.3% in absolute value, relative to range 0.0 to 1.0 used to represent pixel intensities. I expected the random, noisy appearance of the differences in the middle, but the apparently systematic error around -1.5 and 1.5 is slightly worrying.

This difference is slightly larger than I had hoped for, but how much does this affect the MTF50 estimates of mtf_mapper? Take the ESF data sets, and bin them according to the same method used in mtf_mapper, take the finite difference to obtain the PSF, and finally pass this through an FFT. The resulting SFR curves are (again) visually indistinguishable, so here is the difference between the SFR curves instead:
Please note the scale of the plot, and keep in mind that SFR values are in the range 0.0 to 1.0. In the frequency range 0 to 1 cycles per pixel, the maximum absolute error is on the order of 0.13%. The impact of this error on the resulting MTF50 value is also small, resulting in a difference of 0.0566% in MTF50 values between the empirically measured ESF and the expected analytical ESF.

Ok, so what happens at different MTF50 values? I repeated the test by generating a synthetic image with an MTF50 value of 0.06 --- this is a pretty blurry image. The maximum absolute difference between the measured ESF and the analytical ESF came in at 0.356%, and the absolute error in the SFR curve was about 0.8%. The larger errors appear to come from higher frequencies, i.e., because the area under the SFR curve is smaller than that of the SFR curve for the MTF50=0.5 edge above, the relative magnitude of the high-frequency noise (quantization errors?) has increased. Still, the impact on  measured MTF50 was small, differing from the MTF50 derived from the analytical ESF by only 0.066%.

So why do we see errors at all? Why does mtf_generate_rectangle not produce "perfect" results? This is down to the stochastic sampling method used to measure the overlap between a rectangle fragment and the Gaussian PSF centered at each pixel. The default setting is to use 2025 samples per pixel, which is a lot, but not sufficient to ensure perfection. By increasing the number of samples to 31329 per pixel, and repeating the above experiments, the difference between the MTF50 value computed from the measured ESF, and that computed from the analytical ESF is reduced to 0.0147% and 0.0108% for the MTF50=0.5 and MTF50=0.06 cases, respectively. It is therefore reasonable to suppose that these errors will continue to decrease as the number of samples per pixel is increased, eventually converging on the "perfect" case. 

I have an idea on how to change the sampling strategy used by mtf_generate_rectangle (performing numerical integration directly, rather than with a Monte Carlo method), which may turn out to be more accurate at a lower computational cost. But for now, I am keeping the number of samples at 2025, since this appears to offer sufficient accuracy for my purposes.


This experiment demonstrated that the ESF of a synthetic image generated by the mtf_generate_rectangle tool is close to the expected analytical ESF. The small differences observed have no meaningful impact on the accuracy of the resulting SFR curves, nor the computed MTF50 values. In further validation experiments, the synthetic images will be taken as ground truth to measure the accuracy of various slanted edge method implementations.


No comments:

Post a Comment