As assessments of healthcare quality become more important in medical practice [1], statistical process-control methods initially developed to monitor the quality of manufactured goods are being increasingly used to monitor clinical performance [2]. The cumulative sum (CUSUM) is one of the methods used in the medical field. The learning curve CUSUM (LC-CUSUM) test is an alternative to the CUSUM test that was developed to focus specifically on the learning period of a procedure [3,4]. Statistical process-control methods were first used to monitor surgical performance in pediatric cardiac surgery [5] and are still used to monitor cardiac surgeons' performance and their patients' outcomes [6]. Statistical process-control methods have been used in some studies to construct learning curves for anesthesiology procedures, such as peripheral vein cannulation [7], tracheal intubation and mask ventilation [7,8,9], epidural analgesia [8], transverse abdominal plane block [10], or ultrasound assessments for epidural analgesia [11,12].

The French anesthesia residency program has certain mandatory rotations, such as pediatric and obstetric anesthesia, but there is no obligation for the subspecialty rotation. Other anesthesia training programs offer sub-specialty rotations scheduled for managing specific procedures [13]. However, technical competency for specific procedures in France improves only during an elective subspecialty rotation.

Oncologic abdominal and cervicofacial surgery comprises a large part of our activities, and our department is considered a reference center for fiberoptic nasotracheal intubation (FONI) (> 250 procedures/year), thoracic epidural analgesia (TEA), and tracheal puncture (TP) for high-frequency jet ventilation (HFJV) (> 500 procedures/year) by French academics and residents. We have been searching for an objective and individual performance measurement method to assess the learning process for these procedures correctly. Thus, this study assessed the learning curve of residents performing FONI, TEA, and TP procedures during a 6-month rotation using a dedicated statistical method (LC-CUSUM test).

During three consecutive rotations, data from 18 residents (six per rotation) were collected for all performed procedures. After local ethics committee agreement to waive informed consent, the residents were recruited for the LC-CUSUM test evaluation at the beginning of their 6-month rotation in our department. Our current practices remained unchanged.

Eighteen residents performed a total of 1,049 procedures during their rotation.

The LC-CUSUM results revealed that all but four residents did not manage to demonstrate competency during their rotation and were not considered competent with the procedures after their rotation in our department. Our results are not consistent with previously published studies. For example, de Oliveira Filho and Komatsu [7,9] reported that ≥ 50% of residents demonstrate competency for lumbar epidural analgesia or orotracheal intubation procedures. Some explanations of this discrepancy should be considered. The adequate and inadequate performance rates chosen may not have been appropriate. We chose published data [14,15,16] to set performance rates and compare results from studies in other departments. de Oliveira Filho [7] determined the acceptable performance rate for learning lumbar epidural analgesia arbitrarily from samples of procedures performed by instructors. They used 20 and 40% as acceptable and unacceptable failure rates, respectively, and 45% of the residents attained an acceptable failure rate with these limits. Kestin [8] used rates for the same procedure taken from the consensus among staff anesthesiologists in their department (5 and 10% for acceptable and unacceptable failure rates, respectively). They reported that 41% of residents exceeded the chosen failure rate and concluded that the failure rate might be too stringent. They reported that 39–67 procedures were needed to achieve competency in lumbar epidural analgesia, according to the CUSUM test. No information on adequate performance rates for TP and FONI is available in the literature.

The choice of TDR and FDR plays an important part in monitoring efficiency, as our simulations showed. Test performance depends on the difference between the adequate performance level and the acceptable deviation from that level and on the number of procedures during which the trainee is observed. The larger the difference and the longer the observation period, the better the test performance. Our simulations showed that trainees needed to perform approximately 65 FONI (h = 1.67), 140 TEA (h = 1.82), and 520 TP procedures to obtain a TDR of 80% with an FDR of 5%, where the difference between performance under the null and alternative hypotheses is extremely low (1%) (h = 1.88). If a trainee is denied the appropriate number of procedures, then we are at risk of not detecting proficiency in due time by setting the limit “h” too high or we are at risk of having too many false alarms by setting the limit “h” too low. Instructors could decide not to wait for the required number of procedures when assessing competency. In practice, the test properties are mediocre due to the very stringent level of performance required for some procedures, such as TP. Therefore, a dilemma arises in which the test properties are improved, the trainee is allowed to complete a relatively large number of training procedures, or the number of procedures is limited but the properties of the test remain mediocre. Only four residents achieved adequate FONI performance. Using an efficient test to monitor learning, such as the LC-CUSUM, for a 6-month rotation in a reference center may be insufficient to consider that a resident has reached proficiency. This could be an effective approach if a regular collection is paired with an attentive survey from residency heads at the beginning of the residency. However, completing and interpreting charts could be difficult due to discontinuous activity [8,17] and the effectiveness of such an undertaking has not been evaluated. Another possibility is that the residency is organized differently, so proficiency might not be achieved but performance is acceptable. Rotations could be made to fulfill specific goals rather than fulfill a predefined time period in a department [13]. Residents would have to continue training as long as these goals were not met. An efficient statistical process-control method is mandatory for such an approach to prevent excessive training time on a specific topic [13].

Few studies have investigated teaching specific procedures for airway management. Teaching HFJV is difficult, as this procedure is not in general use, despite its validation as part of airway management algorithms [18]. Although the rate of complications is directly related to the number of procedures performed [19], minimal skill during residency has not been claimed from a simulator setting [13]. Cricothyroidotomy has only been studied on mannequins or cadavers [20,21]. Most FONI studies included either sleeping patients with no difficult airways or mannequins [22,23], and the time to complete these procedures was the main outcome studied, rather than efficacy or safety. Furthermore, such conditions for carrying out the procedures have nothing to do with patients undergoing head and neck surgery. Real-life training seems unavoidable until an acceptable level of training is reached for technical and non-technical skills during complex procedures, such as TEA or FONI, as the use of mannequins is considered inappropriate [23,24].

Using a risk adjustment for a procedure would be of great benefit. Unlike some authors [9,25], we did not use a risk-adjusted test, mainly because we were unable to find external data to quantify patient risk level, except for epidural analgesia [26,27]. This was a limitation of our study and is consistent with the requirement for specific data, as reported by Komatsu et al. [9]. We have started collecting demographic data in a new clinical database to fulfill this requirement concerning airway management.

In conclusion, the LC-CUSUM test enabled learning of unusual and specific procedures by residents to be assessed. Reference centers performing subspecialty procedures may not provide sufficient training for these procedures during a single 6-month rotation. An arbitrary number of attempts or training time may not adequately respond to individual training variations, so consideration should be given to monitoring the learning process from the beginning or redefining the residency curriculum.