Using Semi-auto Annotation and Optical Character Recognition for Transcription of Patient Monitor using Smartphone Camera

Abstract

Vital signs monitoring is a key function in healthcare delivery to ensure immediate and precise evaluation of a patient’s well-being. It is done by attaching monitor devices to patients which collect, store, and display values on a screen. In many low-to-medium-income countries (LMICs), hospitals still rely on manual observation and handwritten documentation of vital signs, which is susceptible to human errors, data tampering, process inefficiency, and limited opportunities for comprehensive data analysis. More advanced hospitals utilize interface engines which transmit data to electronic medical records but tend to be model-specific and are very costly. Optical character recognition (OCR) offers a cost-effective and non-invasive alternative to digitizing manual transcription of vital signs data in healthcare settings with low financial resources. An image preprocessing pipeline is proposed to perform contour-based screen extraction of the patient monitor captured by a camera, thus providing a well-defined region more suitable for subsequent tasks of object detection and data extraction. The study offers a newly accrued dataset of over 4000 images of Mindray Beneview T8 patient monitor with multi-parameter annotations. Results showed that screen extraction prior to object detection significantly improved the mean Average Precision (mAP) of the model from 68.55% to 93.65% at an IoU threshold of 0.7.