Trends in orthopedic research: An analysis of podium presentations at the AAOS annual meetings from 2008- 2018

Clinicians and researchers have experienced a significant change in research being published and presented over the last decade. Largescale data has become more accessible over the years due to the rapidly expanding volume of computerized databases, with studies showing that orthopaedic researchers have increasingly utilized these databases [15]. Meta-analyses have also become popular as a statistical method for combining the data from multiple studies in order to hopefully reduce outcome bias by combining large groups of patients from multiple centers. Researchers have also witnessed the evolution of medicine where quality, outcomes, and cost-effectiveness of care is increasingly demanded, which may alter the focus of research studies in other ways [6-8]. Some of these changes have been controversial as journals and professional organizations seek to navigate these new opportunities and present high-quality research.


Introduction
Clinicians and researchers have experienced a significant change in research being published and presented over the last decade. Largescale data has become more accessible over the years due to the rapidly expanding volume of computerized databases, with studies showing that orthopaedic researchers have increasingly utilized these databases [1][2][3][4][5]. Meta-analyses have also become popular as a statistical method for combining the data from multiple studies in order to hopefully reduce outcome bias by combining large groups of patients from multiple centers. Researchers have also witnessed the evolution of medicine where quality, outcomes, and cost-effectiveness of care is increasingly demanded, which may alter the focus of research studies in other ways [6][7][8]. Some of these changes have been controversial as journals and professional organizations seek to navigate these new opportunities and present high-quality research.
Although designed primarily for tracking, insurance, coding, or billing purposes, state and national administrative databases of medical records have been made available for research purposes [1,3,5]. These databases provide substantial health care data on large populations of patients and present a relatively efficient method for researchers to obtain a large sample size and conduct studies on a larger scale. Subsequently, they allow for easy evaluation of hospital or patient outcomes and provide the benefit of increased statistical strength of conclusions [9]. However, it is important to consider the limitations both within and between databases that may bias results, including lack of standardization, varied patient data entry and collection methods, data inaccuracy, and validity [3,[10][11][12]. Consideration must be given to the utility of these databases and what role they should play in the

Abstract
The increased availability of databases for orthopaedic research purposes allows researchers to easily conduct studies and obtain large sample sizes. However, the scientific utility of these databases is in question. The purpose of this study is to determine how these databases impact the quality of podium abstracts, as evaluated by level of evidence, at the American Academy of Orthopaedic Surgeons Annual Meeting. Paper abstracts (n=8,866) from 2008 to 2018 were obtained from the Meeting Committee. Five reviewers independently determined study design, data source, and focus of each abstract. The level of evidence was assigned using the Oxford Centre for Evidence-Based Medicine scheme. body of orthopaedic research being presented to the orthopaedic community. Since studies that employ administrative databases remain controversial, it is worthwhile to analyze the extent to which these databases are emerging within the orthopaedic literature.
With the expansion of types of research including administrative database analysis and meta-analysis, as well as the evolution of healthcare which demands different types of studies such as practice management, a concern can be raised about how this will impact the quality of research. Level of evidence (LOE) designations offer a well-accepted and systematic method for clinicians and patients to evaluate research studies for quality and usefulness in order to arrive at a medical decision [13]. More rigorous orthopaedic research with a higher LOE is associated with higher rates of citations and publications [14,15]. It follows that certain studies are inherently more influential and applicable to practice. With this in mind, one would hope that annual meeting reviewers select abstracts of the highest quality to improve upon the value of the meeting. This idea is supported by recent improvement in the overall strength of evidence of studies disseminated through various orthopaedic journals and annual meetings [16][17][18][19][20][21].

Results
In total, 8,866 AAOS paper abstracts between 2008-2018 were analyzed. The total number of paper abstracts accepted each year increased significantly over the study period (Figure 1), although they did seem to level off in the final four years. This is a clearly a function of the AAOS meeting schedule which has included greater time for presentation of research projects at the podium throughout the course of the study. While the percentage of studies with Level I evidence remained stable, there was an increase in the percentages of Level II (rs=0.9, p<0.001) and Level III (rs=0.89, p<0.001) evidence with a dramatic decrease in Level IV (rs=-0.92, p<0.001) and a statistically significant but small decrease in Level V (rs=-0.62, p<0.05) evidence (Table 1 and Figure 2). Analysis of study designs revealed a significant percentage decrease in case series (rs=-0.95, p<0.001) with an increase in both case-control (rs=0.69, p<0.02) and cohort (rs=0.94, p<0.001) designs. In addition, there was a decrease in the percentage of randomized controlled trials (rs=-0.86, p<0.001) over the years, although the total number of RCT's presented did not change (n=66 in 2008; n=65 in 2018) ( Table 2).
The percentage of basic science studies significantly decreased (rs=-0.64, p<0.04) although the total number did not appreciably change, while studies involving practice management (rs=0.74, p<0.01) dramatically increased in total number and percentage (Table 3). LOE distribution of studies classified as practice management showed a significant percentage increase in Level II evidence (rs=0.77, p<0.01) ( Table 4). There were no significant changes for studies focused on either clinical practice or education.
The American Academy of Orthopaedic Surgeons (AAOS) Annual Meeting integrates innovators from around the world and serves as a leading forum for exchanging the latest in orthopaedic research. A widely attended event, the AAOS Annual Meeting provides both significant educational and clinical value. As a result, the authors hypothesize that the Annual Meeting is a reliable measure of trends within orthopaedic research over time. Although the quality and impact of a study may be subjective, the use of level of evidence (LOE) helps assess each study in a more objective and systematic fashion.
The purpose of this study is to evaluate the quality and types of studies that have been accepted to AAOS Annual Meeting. Specifically, the change in utilization of large administrative databases will be analyzed and compared with other study architectures including single and multi-center studies, along with trends in LOE as a measure of quality in orthopaedic research. The authors hypothesize that there has been an increase in large administrative databases at the AAOS Annual Meetings. However, to our knowledge, no prior study analyzes how large databases impact the scientific quality of the research presented at the AAOS Annual Meetings. Several variables in addition to data source, including study design, will be quantified to define other study features associated with LOE. This will help elucidate trends in orthopaedic research and subsequently determine if certain features are associated with changes in the clinical quality of research over time.

Methods
The study group chosen was the AAOS Annual Meeting research abstracts that were accepted for podium presentation. Podium abstracts have been shown to be of higher quality and are more likely to be subsequently published in refereed journals [22][23][24]. Thus, poster presentations were excluded from this study in order to provide a more uniform selection of higher quality studies that are likely to represent the state of research being performed annually and reported in the orthopaedic literature. Electronic versions of the AAOS Proceedings were obtained courtesy of the AAOS from 2008-2018 for a total of 11 consecutive years. Each paper was independently assessed by five reviewers, and the results were tabulated. None of the reviewers had formal epidemiological training, but each reviewer was familiar with the general principles of research methods and designs. In order to systemize the approach to assigning the LOE and to ensure consistency between reviewers, the study utilized the 2011 LOE scheme designed by the Oxford Centre for Evidence-Based Medicine (OCEBM) [25]. Many LOE systems focus on treatment effects and harms. However, the design of the OCEBM levels system is based on the entire progression of a clinical encounter (i.e. diagnosis, prognosis, treatment, benefits, and harms), which allows for more comprehensive assessments of each individual study [13]. The benefit of the OCEBM system is that the study can be evaluated and applied to more specific aspects of patient care.
The following variables were recorded: (1) study design (case series/reports, case-control, cross-section, retrospective, prospective, randomized control, systematic review, meta-analysis, other); (2) the number of institutions involved (single-center, multi-center, state/ national database); (3) study focus (basic science, clinical practice, practice management, educational); and (4) level of evidence (I-V). Reviewers also had the opportunity to review the authors' published manuscript if further explanation of the method or design was necessary.
The data was extracted from each study and recorded in Microsoft Excel. Nonparametric Spearman rank correlation was used to assess   There was a dramatic increase in the percentage (rs=0.9, p<0.001) and absolute number (n=31 in 2008; n=157 in 2018) of studies utilizing national/state databases while the percentage of studies using singlecenter data decreased (rs=-0.92, p<0.001). There was no change in the percentage of studies conducted with multi-center data ( Table 5).
Further analysis of the LOE distribution based on data source revealed that studies using national and state databases had significant percentage increase in Level II evidence (rs=0.85, p<0.001) ( Table 6). There were, in addition, significant percentage increases in both Level II (rs=0.78, p<0.01) and Level III evidence (rs=0.78, p<0.01) as well a significant decrease in Level IV evidence (rs=-0.90, p<0.001) in studies using single-center data ( Table 7). There were no significant changes in any LOE of studies using multi-center data.

Discussion
The fundamental finding of this study is that the quality of research measured at the AAOS meeting is not only being maintained but is actually improving as measured by the LOE. The greatest impact appears to be in the Level IV studies which have dramatically declined as a percentage over the 11-year period measured to the benefit primarily of Level II and Level III studies. During this period, the total number of abstracts accepted for presentation has increased by over 40% reflecting schedule changes on the part of the AAOS program committee. It would be logical to raise the concern that the increase in presentation opportunities could lead to dilution of the quality of the studies; however, the opposite appears to be the case. The authors did not evaluate information about number of abstracts submitted for evaluation, but this data would suggest that the number of quality abstracts submitted is keeping pace and even exceeding the acceptance opportunities.
The finding that research based upon administrative databases has risen to over 17% of podium presentations is consistent with previous studies and clearly indicates the prominence of these studies in use today as well as some measure of their utility and general acceptance [1][2][3]. The ease with which researchers can access large sample sizes makes these databases appealing for researchers who wish to conduct studies in a timely manner while strengthening conclusions. This study demonstrates a concomitant increase in the use of large databases and a decrease in studies performed at a single institution. It is likely that the overall validity of these databases will remain controversial for some time. Many authors have cautioned about the generalizability of this type of data and the potential for significant bias [1][2][3][4][5]. The fact that these types of studies seem to be replacing single center studies is difficult to evaluate, but in the most recent year, there are still over 4 times as many single center studies as database studies reflecting a continued preference on the part of researchers or reviewers for the latter.
It is important to note that the percentage of randomized control trials (RCTs) performed has decreased over the last decade, but the absolute number has not changed very much. Although no study is completely free of confounding or bias, well-conducted RCTs are the hallmark of evidence-based medicine. However, RCTs tend to be complex, burdened by regulation, time-consuming, and often require extensive funding and outside resources, which may explain this trend [3,26,27]. Despite this decrease in randomized designs, the rate of level I designated studies accepted by the Annual Meeting committee remained consistent over the study period. Based on the findings, the absence of a significant change in the percentage of level I studies was due to the implementation of high-quality cohort, systematic review, and meta-analysis studies. Historically, the consensus has been that study designs in clinical research are hierarchical, with RCTs remaining the most prominent tool. Although RCTs are likely to provide the best possible evidence, well-designed cohort studies have demonstrated remarkably similar summary results when compared to RCTs [28]. The stability of studies with level I evidence reassuringly demonstrates that the gradual change in study methods by researchers over the years has not compromised the reliability of the highest-quality evidence presented at the Annual Meetings as measured by LOE. It will be worth evaluating in the future the role of the meta-analysis as a relatively recent addition to the armamentarium of researchers that seems to be gaining in prominence although the total numbers remain small. It is also impossible to know whether this reflects the preference of authors or the bias of reviewers.
The increase in the rate of acceptance of Level II studies is encouraging since it reflects the improved quality of the studies being accepted by the committee. Based on the grading criteria, level II studies in particular are represented by high quality diagnostic and prognostic cohort studies in addition to systematic reviews [25]. This is reflected by the significant increase in cohort studies throughout the study period. The only other variable measured in this study with significant increases were the use of large databases. In fact, when further analyzing the LOE distribution based on each data source, results showed that studies specifically utilizing large administrative databases were associated with a statistically significant increase in Level II studies. This may be attributed to the large effect sizes that can be obtained from these databases. Although the overall percentage of studies utilizing singlecenter data decreased over time, LOE distribution analysis of singlecenter studies also revealed an increase in the percentage of both Level II and Level III evidence. The fact that single-center studies are associated with improvements in scientific research despite a relative decrease in percentage of this data source highlights the continued importance of these small, but high-quality studies in research.
Likewise, the rate of acceptance of level III studies by the committee increased. Similar to level II studies, level III studies are largely represented by non-randomized controlled cohort studies [25]. However, level III studies typically have smaller sample and effect sizes than level II studies and do carry a greater risk of confounding. Based on the study's findings and the OCEBM's classification system, the increase in level III studies is driven by these cohort studies and smaller single-center studies. Though the evidence may not be as strong as a level I or II study, the level III studies still present sufficiently strong data to come to a decision and are occurring at the expense of less well controlled studies (Level IV and V).
The increase in level II and III evidence demonstrates the improved quality of the abstracts accepted by the committee. Nevertheless, it is equally important to address the decrease in level IV studies. Level IV studies are largely represented by case-series or otherwise poorly controlled cohort studies. Generally speaking, case-series are associated with weaker forms of evidence. As mentioned previously, an increase in large cohort studies was likely the driving force behind the increase in level II and III studies. Similarly, the yearly decrease in the rate of case series designs is mirrored by a decrease in level IV studies. LOE distribution analysis also showed that there was a significant percentage decrease in Level IV evidence in studies using single-center data. There appears to be a shift in focus away from low-quality study designs at these smaller centers. Another low-quality form of evidence according to OCEBM's system, level V studies are largely observations utilizing mechanism-based reasoning. The absolute number did not change over time, but the percentage of Level V observation studies, certainly the weakest of all study designs, decreased significantly. The relative decrease in level IV and V studies again supports how the Annual Meeting continues to improve the overall quality of studies from top to bottom.
It is not particularly surprising that research is, in part, affected by greater societal concerns and an increase in studies looking at practice management was demonstrated. In the context of alternative health care models including prospective payments and bundled payments, many societies have focused their offerings of seminars and presentations on the orthopaedic surgeons' role in these new systems. Clearly, a significant reason for the increase in focus on practice management may be attributed to the relative lack of literature on this topic and a decision on the part of the clinicians and society to increase the opportunities for presentation [7,8]. One can reasonably assume that this emphasis is commensurate with an interest on the part of researchers or meeting attendees, or both, to be educated in this fashion. The distribution of LOE for practice management studies showed an increase in highquality Level II evidence. As information becomes computerized, outcomes, quality of care, and value-based data can be gathered rather efficiently at the single-center level or with the use of large databases, both of which were also associated with increases in Level II evidence. Regardless of the reason for the increased popularity of these types of studies, the overall quality of research being presented as measured by the present analysis has not declined.
Despite this advancement of LOE, there still remains room for improvement. It is reassuring to observe that the rate of level I studies has not decreased over time. However, because level I studies are the most rigorously conducted and thus the most reliable, it would benefit orthopaedics as a specialty to strive to increase the number of such studies. Based on this study's findings and the OCEBM levels, improving the rate of level I studies does not solely rely upon RCTs. Large cohort studies, systematic reviews, and meta-analyses that are well-conducted utilizing already available databases can help increase the rate of level I studies.
This study has several limitations. A potential confounder is that the authors reviewed abstracts accepted rather than abstracts submitted. This creates potential that there is bias among the reviewers for certain types of papers. However, given the large number of reviewers and the large number of paper presentations that are ultimately published in journals, this seems reasonable for evaluation. In order to create a more powerful analysis, the authors decided to review paper abstracts for every Annual Meeting from 2008-2018. While LOE is an excellent method of systematically evaluating the quality of research articles, the sheer number of abstracts required several reviewers. Having several reviewers reduces the risk of grader bias, but evaluating LOE can still be a subjective matter, which could result in minor inconsistencies between observers. In theory, using OCEBM's classification system reduces the risk of inter-grader bias. Still, it is important to consider that focusing on specific criteria for each level entails a risk that graders assign a level without critically analyzing the study as a whole [29]. The increased focus on research practices and evidence-based medicine today warrants critical analysis of medical literature. The explosion of research evidence, partly due to these large databases, has led to a multitude of innovative diagnostic or treatment methods, many of which may provide marginal benefit at best even though the presented data portends otherwise [30]. With the development of many alternative clinical choices, physicians must be able to incorporate both literature and patient preferences when making medical decisions or recommendations. It is important for modern-day clinicians to utilize their own judgment and evaluate studies within the context of good patient care independent of an assigned LOE [30,31]. Ultimately, the OCEBM levels do not provide a recommendation even if the study is supported by strong evidence. The responsibility lies with the clinician to be aware of different study designs and to determine how to best apply the findings [13,28].
Further, grading of each study was largely based on the information provided within the limitations of each abstract. If there was uncertainty regarding a particular feature of the study design, the reviewers evaluated the published manuscript if it was available. Despite these limitations, the sheer number of abstracts reviewed and the subsequent statistically significant trends found in this study confirms the continuous improvement of the abstracts accepted and distributed at the AAOS Annual Meetings.
In conclusion, the quality of research in orthopaedics as measured by the LOE of paper presentations at the AAOS annual meeting has improved over an 11-year period, in spite of a considerable increase in the number of presentations, which alleviates the concern that more papers could be associated with lower quality. During this same time period, the type of research has evolved including an increase in studies using large databases and meta-analyses, as well as a significant increase in practice management research. This has all occurred without negatively affecting quality. Large databases provide researchers with a powerful tool that enables the utilization of large study populations to facilitate the study of clinical questions. However, there is limited research regarding the impact of these databases on the scientific quality of abstracts. The findings in this study validate the hypothesis and show that the growing use of large-scale databases by researchers has been associated with increased volume and quality of studies accepted to the AAOS Annual Meeting over the last decade, although this does not prove cause and effect. In spite of this favorable finding, the authors caution that conclusions drawn from database studies must continue to be carefully scrutinized on an individual basis as the database, methodology, accuracy, and scope of study can impact not only study results, but also patient care.