Mobile Application for Ulcer Detection

2.1. Thermal Image Acquisition System

The thermal image acquisition system consists of two components: Samsung S6 smart phone and FLIR ONE mobile thermal camera. The FLIR ONE camera is made of two sensors, one is a Lepton thermal sensor, and the other is a visible CMOS sensor. The visible image is overlaid on the thermal image to show the edges on the thermal image producing a final MSX blended image by the FLIR ONE IRT (infrared thermography) camera. The image depicts the hot and cold regions of the area that the camera captures [16]. The camera is based on the principle of blackbody radiation where each body emits its own thermal infrared light or thermal energy, which can be seen only when viewed through the thermal infrared region of the light spectrum [23]. Smart applications using FLIR ONE IRT Camera have been proposed for finding temperature readings in electrical based systems. The FLIR ONE IRT camera can measure high temperature values of up to 120 degrees Celsius. The camera can work continuously for up to an hour after which it needs to be recharged. Moreover, it does not drain the phone’s battery while connected to it. FLIR ONE IRT camera provides an excellent base for thermal image acquisition. The FLIR ONE IRT camera (android) is compatible with various android devices. FLIR ONE also provides a smart application that enables the capturing of IRT images, which are usually used for measuring the temperature of electrical devices. However, they can also be used for medical applications with an accuracy of up to 0.1 degree Celsius. The FLIR ONE device requires some time for charging. The device is simple to use based on plug and use ideology [16].

The used camera is based on the blackbody radiation principle, where an object in the infrared region is attributed by the energy or temperature emitted from the body [24]. However, the reading can be affected by light reflected from the object or its surroundings. Hence, it is best to take the images in a controlled environment where the temperature is set to room temperature and the light is set to a very dim environment. A cold background behind and underneath the object is mandatory. The background may be a concrete wall or a cold towel, and the object can be placed on a surface where the temperatures from the object and from the surface have minimal heat transference. If the image is to be captured by a single user, a non-reflective material such as velvet can be placed around the leg as it can act as a cold surface (background). This simple DIY (Do It Yourself) can be done at home. The distance between the object and the camera must be at least 12 to 16 cm. In addition, there should be no hot objects in the background such as an electrical switch, current carrying wire, hot food, computer, etc. Acquiring a clean image is of utmost importance.

The ulcer was simulated by heating a small metallic object (coin) and pressing it against the foot for heat transference. The heat transference was carefully monitored. The temperature difference on the foot was carefully monitored to have a region of 2.2 degree Celsius temperature difference. The thermal image is captured when the desired temperature difference is achieved.

The smartphone standalone application requires loading the thermal image from the phone’s gallery. The FLIR ONE app provides thermal readings with an accuracy around 0.1 degree Celsius.

2.2. Image Processing

A standalone smartphone system with image processing capabilities is made possible with OpenCV and FLIR ONE SDK. OpenCV is a computer vision library used for simple computer vision and image processing techniques on various platforms [25]. It is recommended to use an older version for better support and documentation. OpenCV module was imported and linked to the Android application. The Integrated Development Environment (IDE) used for Android programming was Android Studio. Although there were many other well-known IDEs, Android Studio was selected, as it is Java based and easy to use.

Fig. (1) illustrates the complete image acquisition and image processing steps in detail. The acquired thermal image consists of a normal color image as well as the radiometric data embedded within it. There are two sides of the obtained thermal image: the blended MSX image, which is an RGBA image, and the Thermal Radiometric Kelvin, which provides the required thermal radiometric data. The radiometric data is extracted from the thermal image. FLIR provides this support for developers who use FLIR ONE products. FLIR ONE SDK was used to access the radiometric data embedded within the thermal image. FLIR ONE SDK features were invoked to access the radiometric data. The path of the acquired image in the gallery was passed to the SDK and the radiometric data were obtained. The radiometric data is in the format known as Little Endian. Hence, a one-dimensional (1D) data was acquired for the thermal image. The 1D data was converted to 2D data to perform computer vision and image processing techniques to acquire accurate results. The number of values or pixels held by the radiometric data is 320 (Rows) times 240 (Columns), which is 76,800 data values. However, the data was scaled to get a matrix with values ranging from 0 to 255. These values for the radiometric data were obtained in Kelvin times one hundred. A single channel grayscale image can be constructed and shown when displayed. To understand the displayed image, the data in the grayscale image ranges in a color scale of 0-255. The zero value represents the lowest temperature from the actual radiometric data, and the 255 value represents the highest temperature. Moreover, the scaled values were used to extract portions of the image following an image segmentation procedure. Applying a binary threshold to the grayscale helps in segmenting the feet of the patient from any background structures. This is based on the idea of temperature variation between the feet segments (Hot) and the background (Cold). A threshold of 149 was applied on the grayscale image to extract the feet segments. The binary image was then eroded to remove any noise (wrongly classified background segments) that might be present. The segments were then dilated to recover the foot area that might have been lost because of the eroding process. Now feet segments were considered as the background, whereas the suspected hottest region was the foreground. The hottest region was extracted using another binary threshold of 244 for the grayscale. Although the hottest region has been identified, verification must be applied to decide whether this was an ulcerous region or not. The decision was taken based on the temperature variation between the suspected hottest region and the normal feet region. The threshold values of 149 and 244 were selected after meticulously studying various scenarios involved in early detection of DFU such as shape, size, and region of the suspected ulcerous region of the foot by using the process as described in the previous subsection. The threshold values depict the suspected region with highest variation in temperature as compared to the rest of the foot. In addition, the normal feet region was analyzed without the suspected ulcer region. The mean temperature of both regions was then calculated and prepared for further analysis. For this, the radiometric data that was previously saved was used, as the mean temperature observed was only for the pixels that depicted the required region. The difference between both extracted segments provided the needed outcomes to decide whether the case was ulcerous, which was an increase of more than 2.2 ^oC, or normal feet temperature variations.

Usually to determine whether the foot is ulcerous, the mean temperature of the feet segment, which is represented without the hottest region, is subtracted from the mean temperature of the hottest region. The Mean Temperature Difference (MTD) is then calculated. If the MTD is 2.2 ^oC less, the foot is normal, and no ulcer is detected. However, if the MTD is above 2.2 ^oC, then an ulcer is present. Moreover, false positives and false negatives results are obtained if the feet region and the hottest region are below 25 ^oC, or if the feet region and the hottest region are above 50 ^oC.

This represents the worst-case scenario with a maximum temperature that the human foot can have. If either of these conditions is triggered, then no ulcer is detected. This is under the assumption that the background is completely cold. On the other hand, the BGR image was extracted from the original thermal image. Upon reading the image, OpenCV was set to output the image in the BG R order representing the Blue, Green, and Red channels. Colors were used to signify the ulcerous region, if present. For example, blue represented the foot devoid of ulcer and red represented the ulcerous region. The background of the thermal image was removed from the BGR image to obtain only the feet segment. This was done by first converting the BGR to grayscale image. The acquired image is shown in Fig. (5). The stand-alone smartphone application automatically applies the developed algorithms and threads on the radiometric data to represent it as a 2D grayscale image. The image shown for the user is a color image of a range from 0 to 255, which represents the variations of the temperature on the acquired image between the background and the feet (foreground). Next, a threshold is applied to the greyscale image to filter out the low values and set them to zero. This works in the same way as a high pass filter. The segmentation procedure is shown in Fig. (6), where two thresholds were used to extract the feet from the background and to obtain the suspected hot region from the plantar feet. The application provides the results to the user as images including the hot, hottest, and the hot minus hottest segments. Moreover, within the application, mean temperature values for both feet, including the hot minus hottest region and the suspected hottest region segments, are represented to the user. Moreover, this helps in removing the background and acquiring a binary image, which represents only the feet segment. The result combines a BGR image of the foot without the background with the ulcer region that has been highlighted in red. The red indicates the critical region of the foot. If ulcer is not present, the result will be the BGR foot without the background and without the highlighted red region.

The application was implemented using two threads: one main thread that provides the user interface as shown in Fig. (2), and a background thread that includes the main functionality of the system. This latter thread is illustrated in Fig. (3). The background thread includes the image processing part as listed in Fig. (4). The background thread protects the main thread from being clogged up with processing techniques, which helps in providing a smooth flowing user experience.