When technologies are used not only for entertainment purposes but also have practical applications in areas responsible for education, science, medicine, and other vital areas, their importance cannot be overestimated. On the blog, we have raised this topic quite often and talked about new developments in the field of science and medicine. Today we will tell you about our own solution, which can help surgeons and other specialists reduce the time for performing operations by increasing the clarity of the images obtained during endoscopy. And this was possible due to the OpenCV library.

Endoscopy is a modern method of instrumental diagnostics that allows you to look inside hollow organs without using a scalpel. Thanks to the development and wide implementation of endoscopic research methods in almost all areas of medicine, the diagnostic and therapeutic capabilities of specialists have significantly expanded. However, despite the constant improvement of endoscopes, their developers currently face an unsolved problem of the significant loss of image clarity during transurethral contact laser lithotripsy in the area of crushing urinary concretions. Laser lithotripsy of the kidneys is the most universal method in the treatment of urolithiasis.

However, during the laser pulse exposure on the urinary stone, the laser radiation is absorbed by the stone. This leads to the formation of a micro crater on the surface of the stone and the evaporation of a part of the substance, which is scattered in the form of an aerosol of solid and liquid particles. A micro-explosion on the surface of the stone leads to turbulent turbidity of the physiological fluid surrounding the stone. Because of this, the stone and the optical probe on the surgeon's monitor become invisible.

To avoid this negative effect, you can use various image processing methods based on noise suppression. The image correction carried out during lithotripsy makes it possible to increase the clarity of the stone and the optical probe contour and speed up the establishment by the surgeon of the contact of the distal end of the probe with the surface of the concretion. This can significantly reduce the total execution time of the operation.

Image before the laser impact and after the impact

Finding Solution

The formal goal of any method of restoring images distorted by noise is to minimize the distance (or error by some criterion) between the original image and the restored one. Various methods of denoising (restoring noisy images) have been proposed since the 80s of the XX century. Despite the fact that these methods differed greatly from each other in technical terms when using different restoring models, the common thing was that the restoring result was a function of averaging over a certain spatial neighborhood.

Today, there are many different tools that can help solve the problem of capturing and processing a video stream. The best option is OpenCV. OpenCV is a computer vision and machine learning library, which has open-source code and interfaces in various languages, including Python, Java, C++, and Matlab. This provides easy and fast portability to various platforms if necessary.

In our project, we investigated the 1CCD HD ENDOCAM Performance HD endocam. The analyzed video file has a resolution of 720 × 576(1.25:1) at a frame rate of 50, compressed by the AVC1 codec, digital format *.mp4 (*. m4v) and represented in the space YUV 4:2:0 (Y'CbCr). YCbCr can be easily converted to RGB for simple processing, as well as a reverse conversion is possible.

The essence of the developed technology for increasing the clarity of the endoscopic image is that a real-time video stream (or a video file of a set format) is processed in a loop. The first step is pre-filtering of the current frame. Next, the noise component is isolated when a special function detects a stroke. Then, in the case of a positive reaction of the noise detector, frame-by-frame processing occurs. The image of the stone is pre-stored in the buffer before the laser radiation begins to affect it. Then the original image is subtracted from the noisy frame (when it is detected from the buffer) and thus the noise component is highlighted. After that, the noise component is subtracted from the subsequent noisy frames, the relevance of which is variable. There is also the image post-filtering, due to which its clarity is significantly improved.

The Laplace operator

Let’s analyze in more detail the implementation of the noise detector of frames that appear at the time of exposure to laser pulses on the urinary stone. The essence of the method is as follows. To detect contour lines or to focus images, in many cases it is advisable to use the Laplace operator (Laplacian), which calculates the second derivative of the function by the formula:

To do this, we take one image channel (mainly grayscale) and collapse it with the next 3 x 3 kernel (Laplace kernel):

Then the variance (i.e. the standard deviation squared) of the response is calculated.

This method is relatively simple and can be implemented with a single line of Python code:

cv2.Laplacian(image, cv2.CV_64F).var ()

# Blur detection using OpenCV

If the variance falls below a predetermined threshold, then the image is considered blurred. Otherwise, the image is not blurred.

The reason why this method works effectively is in the Laplace operator itself. It is used to measure the 2nd derivative of the image. The Laplacian highlights areas of the image that contain rapid changes in intensity. Accordingly, the Laplacian is often used to detect boundaries. The assumption here is that if the image contains a high variance, then there is a widespread of results, both marginal and non-marginal, representative of a normal image in focus. But if we have a low variance, which corresponds to small response values, this indicates that there are very few boundaries in the image. As you know, the more blurry the image, the fewer borders it has.

Another challenge here is setting the right threshold, which can be quite dependent on a specific set of images. If the threshold is too low, the image is mistakenly classified as blurry, while it is not. If the threshold is too high, images that are actually blurred will not be marked as blurred.

Since the image processing algorithm based on this method works best in environments where it is possible to calculate an acceptable range of focus measurement and then detect outliers, it was decided to average the numerical value of the detector for several frames.

OpenCV in medicine

Software development

In the course of the research, we have developed a software implementation of the described algorithm for removing turbidity from the video stream in real-time. The developed method based on digital filtration of the endoscopic image will allow the surgeon conducting the procedure not to waste time waiting for the appearance of the contours of the stone and the probe in the image and continue crushing the urinary calculus. This will reduce the duration of additional manipulations during laser lithotripsy and reduce the duration of the operation. This will help reduce postoperative complications.

We formed the following functional requirements for the system:

  • it is required to display the video stream (source and processed) received from the endocamera;
  • system should provide the ability to configure the processing algorithm;
  • it should allow the user to capture video and save it to disk;
  • system should provide access to previously saved files to easily compare them with current ones;
  • information system should provide the opportunity to interact with it if there is a connected endocamera.

Implementation of the functionality

The OpenCV library provided enough functionality for the implementation of all image processing operations (individual frames of the video stream). OpenCV made it possible to create an application with a concise code and the possibility of simple modification in the future.

The correctness of the application was checked using test data (a real record of the procedure of contact laser lithotripsy of the kidneys).

As a result of testing, the following results were obtained:

  • main module copes with the task of capturing a video stream from various sources and placing frames in a buffer for processing and subsequent output, saving;
  • pre-filtering module allows increasing the clarity of the final image, eliminating it from random noise by applying a mask;
  • noise detector dynamically determines the fact of "blurring" of the current frame based on the variance values for all buffer frames;
  • algorithm (noise exclusion module) allows correctly selecting and excluding foreign objects in the frame (particles, general turbidity of the frame);
  • post-filtering module equalizes the histogram of the frame and filters high-frequency interference.
OpenCV
The original image and the image after processing

Result

Our developed project on OpenCV is aimed at improving the working conditions of specialists in the field of medicine. Thanks to the use of the OpenCV library, we have achieved our goals and clearly demonstrated the work of our product. If you have an idea for a project, write to us, and our team will help you implement it!