OpenCV Super Resolution Comparison with Interpolation Methods

Super Resolution is a machine-learning, neural-network based method that promises great improvements for digital image upscaling. In this article I will give a simple and mostly subjective comparison, using GNU/Linux software tools, of two high quality interpolation methods with the super resolution method.

---

What is image interpolation?

Whenever a digital image is enlarged, or even processed in some other way, new information must be created to "fill in the blanks" within the new resultant image. Consider the following representation of an image upscaling, using a 2x2 array of grayscale pixels:

In this example, the original 2x2 array is expanded to a 3x3 array. This operation creates a number of blank areas that must be populated by other grayscale pixels. The exact shade of gray in these unfilled areas must duplicate the intensity relations between the original pixels so as to effect a faithful rendition. But how can this be accomplished?

Many interpolation methods have been developed to create this new information. There is the simple pixel averaging. There are also various ways of fitting the original pixels to a descriptive mathematical curve which can then be applied to create the new information. This latter method, which will be used in this article, usually gives the best upscaling results.

A more recently developed method, called Super Resolution, uses machine learning to create the new information. Many examples of low/high resolution image pairs are first used to "train" a neural network into how upscaling should be accomplished. This "training" is then used to determine the hopefully correct new pixels to be included in an arbitrary upscaled image.

In general, interpolation is usually very fast but super resolution methods can be very slow (not including the preliminary training time), although there are some fast variants available.

In this article I will compare Super Resolution with two high quality interpolation methods: Mitchell and NoHalo. The GNU/Linux FOSS software utilized is listed next:

• 1) Mitchell interpolation: this usually gives excellent results for raster enlargements.
  ImageMagick was used to apply the filter:
  

  convert image.tif -resize 400% -filter Mitchell image-mitchell.tif

• 2) NoHalo interpolation: this is a newly developed filter that is used by the GIMP (GNU Image Manipulation Program) through the GEGL libraries. The NoHalo filter was applied applied using the GIMP.
• 3) Super Resolution: this method is available with OpenCV, the FOSS computer vision package. The particular SuperRes method employed, the pre-trained EDSR model, was applied using the code here. The training models used are available from here. Good background info on SuperRes is available here.

---

Comparison of the Methods

TIFF and PNG format images were gathered from the world wide web. An origin link is given for each image. I chose these formats because they are presumable free of compression artefacts but for some cases it cannot be certain that they are not just converted JPEG files. In any case, the reader can obtain these images to verify the results for himself.

The original image files were processed "as is" but, since most web browsers do not display the TIFF format natively, they were converted losslessly to PNG format. The following thumbnails are either the reduced originals or the cropped enlargements. To view the details in full size just click on the accompanying links.

Each original image was upscaled to 4X using the three upscaling methods indicated above.

Circuit Board Image (Source)

Both interpolation methods, Mitchell and NoHalo, give good results and are quite close in quality with NoHalo being slightly superior. However, at 4X upscaling the sharpness has decreased significantly and this is to be expected as interpolation does not capture the high spatial frequencies necessary to reproduce sharp edges.

Super Resolution provides the clearest and sharpest outcome and the upscaling is truly remarkable. However, in this image at least, there is also a kind of distortion that resembles a watercolor texture and which imparts a slight unnatural look in places, although this subtlety does not detract much from the quality of the result.

Woman Image (Source)

In this case there is very little difference between the results of the three upscaling methods. With Mitchel and NoHalo interpolation, the full-size view reveals some jaggedness to the edges of the metallic arch and a roughness to the clothing. SuperRes shows a noticeably sharper face and a definite smoothness to the hair and clothing. Again, SuperRes represents the highest quality, but not to a marked degree.

Bird Image (Source)

In this example, Super Resolution produces a remarkable upscaling result that the other methods cannot match. Especially, the extreme edginess (sharpness) of the feathers is very well preserved. Other regions of the image, such as the plant stalks, are also sharply defined and free of jaggedness.

Cameraman Image (Source)

The Super Resolution method produces another superlative upscaling. Note the lack of jaggedness to the edges of the tripod legs and the clothing. The clarity of detail, especially in the texture of the coat and gloves, is quite remarkable. However, as with the example of the circuit board image, there is also a kind of watercolor effect to the camera body and lens, which, to my eyes at least, imparts a slight unnaturalness.

Einstein Image (Source)

In this final example, we see the definite benefits that Super Resolution can bring. Both the Mitchell and NoHalo methods, which are among the best interpolation methods, produce jaggedness of edges. NoHalo gives slightly sharper details but also more harsh jaggedness. In contrast, Super Resolution is both remarkably clear in detail with a total lack of unrefined or jagged edges.

---

Conclusion

As I indicted at the outset of this article, this comparison is mainly a subjective one and I will not attempt to provide a technical analysis of the above results. My desire is mainly to evaluate Super Resolution as a tool for general image processing as done by an informed amateur.

Upscaling is a very difficult problem because it requires the creation of new information based on the limited data provided by the original image. In particular, as mentioned above, interpolation methods cannot reproduce the high frequencies which are necessary to maintain sharp edges. However, as seen above, Super Resolution can indeed produce very superior outcomes when compared to traditional interpolation methods. The results, also as seen here, are not entirely consistent but this most likely is due to the particular Super Resolution model used. With more preliminary "training" of the model the outcomes likely could become uniformly improved.

The difficultly with Super Resolution, at least in the case of OpenCV superres, is that the particular model used, the EDSR_x4, requires a comparatively long time to process a given image. Again, this could be improved by using OpenCL to exploit the power of the graphics card parallel engine.

Super Resolution demands a lot of time and resources for the preliminary "training" of the model and a lot of computing power for the processing step, but, in my opinion, the results could be well worth the effort.