INTRODUCTION

Postpartum hemorrhage is the leading cause of maternal morbidity and mortality1. Prompt intraoperative surgical management is reported to reduce acute blood loss during a cesarean section2. The B-Lynch Suture and O’Leary Stitch (bilateral uterine artery ligation) are effective methods to manage hemorrhage3. Improved education regarding these compression and ligature sutures is difficult to provide in emergent settings. Surgical models with structured assessments can provide residents opportunities for feedback.

Objective structured assessments of technical skills (OSATS) are checklists that help educators to assess technical skills of a specific procedure; it can also be known as technical skills checklist (TSC). TSC are an effective teaching tool to establish performance baseline and measure improvement through calculated numerical scores of skills completion4,5 . The first published OSATS tool was a global rating scale (GRS), designed to assess general operative performance4. Instrument handling, respect for tissue, and use of assistants are a few of the criteria used in GRS6,7.

Learners can improve skills on B-Lynch and O’Leary through TSC use in simulation. This study aimed to determine the reliability and validity of a new task-specific OSATS for B-Lynch and O’Leary in simulation.

MATERIAL AND METHODS

We created a task-specific checklist (TSC) for the B-Lynch (Appendix 1) and O’Leary techniques (Appendix 2) based on a review of textbooks and literature search of the Medline database. The checklist is scored from 0-19 for B-Lynch, and 0-17 for O’Leary; with higher scores indicating better performance. The TSC was used alongside a previously-validated OSATS Global Rating Scale (GRS) for general surgical techniques. GRS comprises five general domains and has a maximum score of 25, with higher scores indicating better surgical skill4.

Regarding model building, various models have been used, including the flank steak8,9 9, socks filled with polyester10 10, pillows, and felt hand-made models11 11. We created an easily reproducible, sustainable, cost-effective surgical model. Based on various published and non-published sources, a model using felt, yarn, intravenous tubing and pillow fluff was created and validated for this study. Twenty-five models were prepared for the study so that no model was used more than three times.

Institutional review board exemption was obtained, as this research conducted in an established educational setting involving normal educational practices. Obstetrics and gynecology residents and faculty attendees from a large urban academic medical center were recruited by researchers during resident education time. For Inclusion criterias, all obstetrical gynecology residents were asked to participate, and all full time obstetrical attendings that provided care on the labor floor from March 1 2017 to August 31, 2017. Exclusion criterias included attendings that did not have active obstetrical privileges. Because there are no other studies in the literature validating OSATS for B-Lynch or O’Leary, sample size was not able to be calculated. After informed consent, residents reported for study sessions during their didactic time. Residents completed questionnaires regarding their training and experience in B-Lynch Suture and O’Leary Stitch. Subjects had the option to wear gloves and were provided a standardized setup and were not able to “practice” prior to filming. The setup included a surgical model (Figure. 1), camera, and various distractor surgical instruments and sutures. A standard case scenario and instructions to assess and perform the B-Lynch Suture and O’Leary Stitch were read to each subject and an assistant was available for surgical assistance on request.

Each subject was randomly assigned a subject ID number (via randomization.com) at the beginning of the session and given 15 minutes to complete the task. Procedural times, suture type, needle choice and only hands using the model were recorded. Facilitators did not provide input or feedback regarding the session. Subjects were asked not to discuss their sessions with others.

After the evaluation, all videotaped recordings were muted and edited to ensure anonymity. Study packets were distributed to two faculty evaluators who were attending physicians from a tertiary institution. Each study packet contained the de-identified muted recorded sessions and evaluation tools, including the task-specific OSATS and GRS for each session. Each evaluator independently reviewed the video recordings and completed the evaluation tool for each subject. All study materials were returned to the principal investigator.

The results of the evaluations underwent statistical analysis. Statistical analyses were performed using IBM SPSS Statistics Subscription v25. Task-specific OSATS and GRS scores were assessed for reliability. Intrarater correlation was used to determine interrater reliability. We first evaluated interrater reliability by testing the hypothesis that scores by the same rater should differ when participants are stratified by experience level; new residents PGY (postgraduate year) 0 vs junior resident (PGY 1, 2) vs senior resident (PGY 3,4) vs attending physicians. Higher scores would be noted with greater years of experience. The two blinded evaluators completed both evaluation tools (TSC and GRS) for both procedures (B-Lynch and O’Leary).

We assessed construct validity or the ability of TSC to distinguish between experience levels, as a proxy for validity, or the extent to which a test measures what it is intended to measure. We tested this by stratifying overall results by experience level and by the time spent performing the procedure. One-way analysis of variance with post-hoc Tukey test was used to determine the difference between experience level and each construct validity variable. We hypothesized that greater experience would be associated with greater expertise and a shorter procedure. We determined our benchmark cut-off times at the 50th percentile duration when performed by the expert (attending physicians) participants, which was 240 and 150 seconds for the B-Lynch and O’Leary procedures, respectively. Additionally, we explored any association of education-related factors with the TSC scores. We had collected such variables, including prior exposure to task-specific literature and didactics on the initial questionnaire for all participants before the session.

RESULTS

Fifty-two of 59 residents participated in this study [12 PGY-0; 23 PGY-1,2; 17 PGY-3,4]. Twenty-five attending physicians participated in the study.

Table 1 summarizes the results of the questionnaires regarding education and experience with the B-Lynch Suture and O’Leary Stitch. In Table 1, attending physicians had more experience regarding the B-Lynch and O’Leary procedures compared to the senior and junior residents. Senior residents were more likely to have completed a simulation (different model) or had some form of previous reading. The junior residents had lower rates of previous experience, less didactic experience and lower performance on TSC and GRS for both procedures compared to the attending physicians. The results showed construct validity as senior residents and attending physicians performed better than junior residents in all assessments.

In Table 2, the two evaluators’ scores were combined to create an average TSC and GRS for B-Lynch, and separate scores for O’Leary for each group. TSC for B-Lynch ranged from 0-19.

B-Lynch TSC validity

B-Lynch construct validity was demonstrated, with significant differences in the results. New residents had mean TSC score 3.38 vs junior resident 5.63 vs senior resident 15.12 vs attending 15.04. Regarding the average GRS, new residents 6.75 vs junior resident 8.85 vs senior resident 19.35 vs attending 22.38; showing a construct validity of improved scores with increasing experience as shown in Table 2.

O’Leary TSC validity

O’Leary construct validity was demonstrated as well. New residents had mean TSC score 2.83 vs junior resident 11.54 vs senior resident 13.65 vs attending 15.20. Regarding the average GRS, new residents 6.83 vs junior resident 14.89 vs senior resident 21.32 vs attending 23.76; showing a construct validity of improved TSC and GRS scores with increasing experience shown in Table 2 .

Reliability testing for OSATS

Reliability was tested via intraclass correlation (ICC), in a two-way random effect model. The reliability of the GRS raters was ICC (2, 2)>0.99 and ICC (2,2)>0.98, for B-Lynch and O’Leary, respectively. We concluded that greater than 99% of the variances in the mean of these raters existed and less than 1% represented random variation and that the GRS score reliability was excellentTable 3 .

The reliability of the TSC raters was ICC (2, 2)=0.98 and ICC (2,2)=0.97 for B-Lynch and O’Leary, respectively. Thus, the OSAT score reliability was considered excellent. Construct validity was demonstrated because there were significant differences between the three groups for both OSATS and GRS for B-Lynch and O’Leary.

Table 1 Participants’educational experience
Educational experience/Construct validity variable Junior residentsPGY-0,1,2 (n=35) Senior residentsPGY-3,4 (n=17) Attendings(n=25)
Previously performed B-Lynch* 4 (11.4%) 6 (35.2) 15 (60%)†
Previously performed O-Leary‡ 3 (8.6%)§ 7(41.1%) 14(56%)§
Read literature regarding techniques‖ 14 (40%)¶ 16 (94.1%) 22(88%)¶
Prior didactics before this simulation# 7 (20%)** 10 (58.8%)** 11(44%)
Data are presented as n (% percent within the subgroup of junior residents, senior residents, or attending physicians). #F (2,74)=4.52, p=0.01 **Junior residents were less likely to have completed didactics than senior residents
Table 2 Average task-specific scores (TSC) and global rating scores (GRS)
Group Score Mean SD
New residents 12 TSC BLYNCH GRS BLYNCH TSC OLEARY GRS OLEARY 3.38 6.75 2.83 6.83 6.23 4.71 4.34 4.63
Junior resident 23 TSC BLYNCH GRS BLYNCH TSC OLEARY GRS OLEARY 5.63 8.85 11.54 14.89 4.34 4.00 5.50 7.07
Senior resident 17 TSC BLYNCH GRS BLYNCH TSC OLEARY GRS OLEARY 15.12 19.35 13.65 21.32 4.12 6.64 3.97 5.50
Attendings 25 TSC BLYNCH GRS BLYNCH TSC OLEARY GRS OLEARY 15.04 22.38 15.20 23.76 4.33 5.03 1.51 3.14

Table 3 Intraclass correlations (ICC) showing reliability
BLYNCH TSC 0.98
BLYNCH GRS 0.99
O’LEARY TSC 0.97
O’LEARY GRS 0.98

DISCUSSION

With simulation based education amongst residency programs becoming more prevalent, there is a growing need for assessment tools used for evaluation and procedural feedback. Surgical models with structured assessments can provide residents with initial experience, constructive feedback, and the opportunity to learn in a low-stress, high-yield teaching environment. The technical skills checklist for the B-Lynch and O’Leary, provides such opportunity. The use of GRS attempts to assess global scale domains such as “use of surgical assistants” in such a way that would otherwise not be addressed.7,8,12

The success of validating this checklist for B-Lynch and O’Leary are from identification of procedural steps and location of placement of needles on the model, allowing evaluators to score more accurately and precisely, therefore increasing the inter-rater reliability.

This study had several strengths; large study group of residents, construct validity and inter- rater reliability was shown and we validated the OSATS for B-Lynch and O’Leary using a validated surgical task-trainer. Limitations of our study included using residents from one large residency program, which could be addressed by expanding this study to other programs. A second limitation is the sample size; because there were no previous OSATS of B-Lynch or O’Leary published in the literature; determine sample size was difficult. Future studies can have better sample size and greater numbers from the basis of this paper. Another limitation to this study is recognizing that teaching and assessing trainees in a simulated environment may not translate to clinical competency to perform well during live surgery. Future study may include asses sing resident performance in the simulated environment and evaluating their performance in the operating room using B-Lynch and O’Leary OSATS and GRS in real time. Full study protocol may be accessed by contacting lead author. The advantages of using such a specific checklist allows consistency in evaluation; making this tool highly reliable and translatable with the use of GRS. When used with a validated surgical model, this task-specific tool can provide learners with initial experience, constructive feedback, and the opportunity to learn in simulation.

References

  1. Say L, Chou D, Gemmill A, Tuncap O, Moller AB, Daniels JD et al. Global causes of maternal death: A WHO systematic analysis Lancet Global Health.;2014(2):323-3.
  2. B-Lynch C, Coker A, Lawal A, Abu J, Cowen M. The B-Lynch surgical technique for the control of massive postpartum haemorrhage: an alternative to hysterectomy? Five cases reported Br J Obstet Gynecol.1997.
  3. Uterine compression sutures: surgical management of postpartum hemorrhage. Obstet Gynecol 3. Hayman R Steer P. .2002;99:502-6.
  4. Resnick RK, Teaching and testing surgical skills Am J Surg.1993.
  5. Houmard B, Mandel LS. Development of an objective structured assessment of technical skill for obstetrics and gynecology residents Goff B.A., Lentz G.M., Lee D.. .2000.
  6. Development of a bench station objective structured assessment of technical skills Goff B.A., Lentz G.M., Lee D., Fenner D., Morris J., Mandel L.. .2001.
  7. Testing surgical skills of obstetrics and gynecology residents in a bench laboratory setting: validity and reliability Lentz G.M., Mandel L.S., Lee D., Gardella C., Melville J., Goff B.A.. Am J Obstet Gynecol.;184:1462-70.
  8. Teaching surgical skills-changes in the wind Reznick R.K., MacRae H.. N Engl J Med.2006.
  9. A novel approach to teaching placement of a B-lynch suture: description of technique and validation of teaching model Vetere P.F., Wayock C.P., Muscat J., Sicuranza G.. J Grad Med Educ.;4:367-9.
  10. A tool for teaching the B-Lynch brace suture method: an inexpensive new simulation allows obstetricians to polish an essential technique 10. Pereira N Delvadia D. Am J Obstet Gynecol.;2013(209).
  11. Allam MS, B-Lynch C. The B-Lynch and other uterine compression suture techniques Int J Gynaecol Obstet.;89:236-41.
  12. Objective structured assessment of technical skills for repair of fourth-degree perineal lacerations Siddiqui M., Stepp K., Lasch S., Mangel J., Wu J.. Am J Obstet Gynecol.2008;:1-676.