Article Text
PDF
Abstract
Background Management concepts cycle through healthcare in trends lasting 3–5 years. This may hinder policy-makers, healthcare managers, researchers and clinicians from grasping the intricacies of a management concept and prevent organisations from realising the potential of these concepts. We, therefore, sought to characterise how the newest management concept, value-based healthcare (VBHC), is used and understood in the scientific literature.
Methods We developed a novel five-step approach: (1) identification of a trend-starting article, (2) identification of key conceptual aspects in the trend-starting article, (3) collection of citing articles and identification of citing text, (4) categorisation of citing text to evaluate which aspects were used and (5) categorisation of citing text according to the structure of observed learning outcomes (SOLO) taxonomy to evaluate understanding.
Results We identified four aspects in the trend-starting article, ‘What is value in healthcare’, of which value and outcomes were the most cited. More than one-quarter of the citing texts demonstrated no understanding of the aspect referred to; most demonstrated a superficial understanding. Level of understanding was inversely related to journal impact factor (IF), and did not change significantly over time. A deeper understanding was demonstrated in those articles that repeatedly cited the trend-starting article.
Conclusions None of the four aspects were understood at a level required to develop the management concept of VBHC. VBHC may be undergoing a process of dilution rather than diffusion. To break the cycle of management trends, we encourage a deeper reflective process about the translation of management concepts in healthcare.
- Healthcare quality improvement
- Health services research
- Management
- Evaluation methodology
Statistics from Altmetric.com
Introduction
The list of management concepts that have made their way into healthcare is long: business process re-engineering, total quality management, continuous quality improvement and lean, to name a few. However, as soon as we start to grasp one, a new concept seems to come along. This pattern of trends has been dubbed ‘pseudo-innovation’—the repetition and reinvention of essentially similar ideas and methods using different terminologies every 3–5 years.1 The consequence of pseudo-innovation is that quality improvement initiatives have difficulties to scale up or spread to new contexts,1 perhaps because the cultural and organisational infrastructure needed to impact patient results do not have enough time to develop and mature.2
Our current understanding about the diffusion of innovative management concepts in service organisations is that it is dependent on how the innovation is perceived, the characteristics of the involved individuals and contextual and managerial factors.3 Greenhalgh et al4 found that the diffusion or dissemination of an innovation is dependent on the resource system, knowledge purveyors, change agency and the user system. Taylor et al5 found that of 73 applications of the plan-do-study-act (PDSA) method, a cornerstone in quality improvement, only two demonstrated full compliance with the principles. A realist review of lean found compelling evidence of superficial applications.6 Why these superficial or incomplete applications repeatedly occur is unclear. Walshe1 postulates that pseudo-innovation persists due to the interests of knowledge purveyors (to consult, write books and research articles, or present at conferences) and users looking for a quick fix.
The latest management concept to garner attention is value-based healthcare (VBHC). It has been touted as ‘the strategy that will fix healthcare’.7 Porter and Teisberg8 propose that we redefine healthcare in terms of value, explicitly defined as outcomes divided by costs. So have knowledge purveyors and users (policy-makers, healthcare managers, researchers and clinicians) grasped the intricacies of this management concept or is VBHC in danger of becoming the next example of pseudo-innovation?
Aim
The aim was to characterise how a nascent management concept (VBHC) is diffused and understood in the healthcare literature. Two questions were addressed: (1) which aspects are used? and (2) how well have these aspects been understood?
Methods
We developed a novel approach to analyse the diffusion and understanding of conceptual ideas in the literature. It consisted of five steps: (1) identification of a trend-starting article, (2) identification of key conceptual aspects in the trend-starting article, (3) collection of citing articles and identification of citing text, (4) categorisation of citing text to evaluate which aspects were used and (5) categorisation of citing text to evaluate how the aspects were understood. This approach enabled a statistical analysis.
Identification of a trend-starting article
We chose to search the Web of Science database because of its citation index function and ability to link to impact factor (IF). We searched for articles, through 2 May 2014, using the broad string (TOPIC (value healthcare) AND AUTHOR (Porter ME)). Since we were looking for a trend-starting article, we chose the most cited article.
Identification of key conceptual aspects
We read through the trend-starting article and used an inductive approach to identify meaning units on a semantic (explicit) level, which were condensed into codes (n=52). We grouped the codes into themes that represented the key conceptual aspects of the article.9
Collection of citing articles and identification of citing text
We collected all articles, through 2 May 2014, that cited the trend-starting article. In these articles, we identified all citations, extracted the surrounding text and gave each ‘citing text’ a unique identifier. We also extracted data on article title, journal, setting, year of publication and 1-year IF from publication year. IF for 2014 was not available, and therefore, we used the IF for 2013. We categorised the articles into theoretical, empirical and reviews.
Categorisation of citing text to evaluate which aspects were used
The first authors independently used a deductive theory-driven thematic analysis approach to code the main theme of each citing text according to the aspects identified in Step 2.9 During analysis, the title, journal, author, IF and year were hidden to minimise bias. We compared the codes, and discrepancies were identified and discussed until consensus was reached.
Categorisation of citing text to evaluate how the aspects were understood
In systematic literature reviews, one considers the quality of an article mainly as a function of study design. In this case, we were interested in quality as an expression of understanding. We, therefore, applied the structure of observed learning outcomes (SOLO) taxonomy to determine the level of understanding expressed by the authors in each citing text.
The SOLO taxonomy is a way to characterise students’ learning quality by determining the structural complexity of an outcome of a learning process, such as a text.10 The taxonomy consists of five levels that we adapted to authors instead of students. The lower levels of the taxonomy represent surface approaches to learning; the top two levels describe deep approaches to learning.11 The first, level 0: prestructural, corresponds to a state of incompetence or non-learning; of missing the point or where nothing is known. At level 1: unistructural, the author is able to identify one relevant aspect. At level 2: multistructural, the author is able to describe several relevant, but independent, aspects. At level 3: relational, the author is able to integrate different aspects into a structure, to compare. At the top, level 4: extended abstract, the author demonstrates the ability to generalise to a new context, domain or application, and/or develop the concept further.12 Thus, a ‘deep’ approach to understanding is requisite to apply, explain or analyse (level 3) or reflect, theorise or generalise (level 4).12
We assigned each citing text a SOLO level based on the structural complexity of how the identified aspect was used. As in Step 4, the initial coding was done by the first two authors independently with article data hidden. Discrepancies were then discussed and resolved in consensus. All empirical articles with SOLO level 3 citations were full-text analysed to determine if they empirically tested the aspect(s).
Statistical analysis
We analysed the data generated in the previous steps descriptively (IBM SPSS Statistics V.22) to understand which aspects were used (frequency counts) and how they were understood (median, IQR). We used χ2 test for trend (MedCalc V.14.10.2) to assess if the proportion of the aspects changed over time. We used Spearman's Rho (IBM SPSS Statistics V.22) to assess correlations between the level of understanding (SOLO) of the citing texts with IF and year of publication, respectively. The number of repeated citations per article was analysed for correlations with SOLO level at the article level, since it is a property of the article, not the citing texts. The SOLO level of the article was defined as the highest attained SOLO level in the article.
Results
Our literature search for a trend-starting article (Step 1) yielded 12 peer-reviewed articles. For every year, the number of articles that cited these 12 increased (figure 1). ‘What is value in healthcare’13 was by far the most cited article. We, therefore, defined it as our trend-starting article.
Overview of citation statistics for articles on value in healthcare.
Four identified aspects of the trend-starting article (Steps 2–4)
In the trend-starting article,13 we identified four key conceptual aspects: value, outcomes, cost/revenue (including reimbursement, costing, resource use and spending reallocation) and system (including organisation and stakeholders).
We found 199 articles (original research papers, proceeding papers, commentaries and reviews) that cited the trend-starting article. We excluded non-English language articles (n=12), and one article was not retrievable. Of the remaining 186 citing articles, 63% were specific to US healthcare settings, and 84% were published in clinically oriented journals. We identified 255 citations of the trend-starting article. Of these citing texts, 51.4% referred to value (table 1). The proportion of aspects used did not change significantly over time.
The number of citations and level of understanding (SOLO level) for each aspect in the citing texts
Variation in understanding (Step 5)
We found examples of all SOLO levels except for extended abstract (see table 1 and online supplementary appendix table A1). Twenty-six per cent of the citing texts missed the point and demonstrated a prestructural understanding (figure 2). An example of this:
The distribution of the level of understanding (SOLO level) among the citing texts. SOLO, structure of observed learning outcomes.
Such patient-centeredness [defined earlier in the article as the patient's experience of transparency, individualization, recognition, respect, dignity, and choice] is reflected in the heightened awareness that the quality and value of healthcare services are most appropriately determined from the perspective of the individual patient.14
At the citing-text level, the value and outcomes aspects had the highest median SOLO levels. Based on IQR, outcomes seemed to be the most well-understood. We did not find a significant change in terms of increased or decreased understanding of the aspects over time. We found a weak significant negative correlation between the level of understanding and IF such that the higher the IF, the lower the demonstrated understanding of the cited aspects of the value concept (table 2).
Spearman's ρ-correlations between level of understanding (SOLO level) and IF
At the article level, we found that as the number of repeated citations within the same article increased, the number of articles that attained SOLO level 3 increased. This correlation was statistically significant. In analysing the data from the perspective of the highest attained SOLO level per article, we also found a weak significant negative correlation with IF (table 2).
None of the 54 empirical articles tested the aspects. Of those that had SOLO level-3 citing texts, five15–19 applied one or more of the aspects empirically.
Discussion
Our literature search revealed a growing interest in the concept of value in healthcare in the peer-reviewed medical literature. Authors addressed all four aspects of the trend-starting article with a clear focus on value. However, for many of the citing texts, the understanding of the aspects appeared to exist primarily at a surface level. This level of understanding was weakly inversely related to journal IF, and did not change significantly over time.
Through the five-step approach, we may have identified a plausible and contributing explanation for pseudo-innovation. However, we recognise the existence of limitations to our research approach. Including all of Porter et al's articles or an earlier article might have improved the probability of finding citing texts at SOLO level 4. Full-text analysis of all the articles, not just the empirical ones, might have captured examples of higher levels of the SOLO taxonomy if the authors developed their understanding of VBHC over the course of the article or developed ideas based on value without formally citing it, so-called ‘implicit citations’.20 For example, Porter et al extended the three-tiered model to chronic care in a subsequent article included in our analysis.21 This could be interpreted as an extended abstract level; however, it was presented not as a development, but as a completely new model. We also recognise that ‘value’ is a concept in its own right, with different meanings (eg, economic values, moral values, scientific values) inherent to different disciplines22 and a component of other management strategies (eg, it is integral to lean). Economists, clinicians and patients can all have different perspectives of value. However, we would argue that ‘value’ per se cannot be regarded as a new management concept that is gaining attention the same way that VBHC is. Moreover, the majority of articles were published in clinically oriented journals and many of the ‘academics’ appear to be ‘practitioners’, suggesting that the concept is being spread by practitioners themselves.
Keep the concept simple and the title catchy
That value was the most frequently referred to aspect suggests that the value equation (value=outcomes/cost) is central to the value in healthcare concept. The focus on outcomes is understandable given that a large part of the trend-starting article centres on the three-tiered hierarchy for identifying outcomes. However, value is cited twice as often as outcomes. So, while the focus of the article is on outcomes, it is cited as a reference for value. This lends credence to the idea that the title of the article has a larger impact than the content of the article.23 ,24 It may also be due to an increased interest in value as a concept applicable to healthcare, independent of Porter et al's specific definition. This may have implications for how authors choose titles when introducing new management concepts. It could also reflect the fact that the value and outcomes aspects are more developed in the trend-starting article than the cost/revenue and system aspects, and why the outcomes aspect was the best understood.
Authors miss the point
More than a quarter of the citing texts missed the point (SOLO level 0) of what they referred to. A simple explanation for this is that authors follow ‘least-effort behaviour’25 or are poor, careless or even flippant in their referencing.24 ,26 Another could be that authors’ reference management concepts on a surface level were without a deep understanding of the consequences and implications of the concept. An additional explanation was the pattern of ‘close but no cigar’: the citing of similar themes that were incorrectly related to the four aspects. For example, subjective patient-reported experience measures were linked to the outcomes framework,27–29 when Porter explicitly argues for the use of non-experience-based outcome measures.13 Others referenced Porter when discussing the importance of patient-centredness14 ,19 ,30 instead of understanding the provider-centredness of the value in healthcare concept.13 And some articles confused the concept of value or outcomes with the concept of quality, 31–33 despite Porter's explicit clarification of the difference.13 All three examples are suggestive of a ‘buzzword’ approach to citing research where authors relate or link to value regardless of their understanding of the VBHC concept. They illustrate how the dilution of a management concept may contribute to pseudo-innovation.
Dilution is not only a function of SOLO level 0. SOLO level 1 and 2 may also contribute as illustrated by the ‘value’ aspect. SOLO level 1 encompasses those citing texts that address the existence of ‘value’. SOLO level 2 is a repetition of the parts of the equation. This does not imply that a low SOLO level is incorrect; it could be adequate for the purpose of the citation. However, given the challenges quality improvement has had with a lack of uniformity in nomenclature or content34 and that ‘any activity could be renamed a quality project’,35 references that merely link to the value aspect (SOLO level 1–2) may contribute to dilution.
We might not actually stand on the shoulders of giants
One could presume that the higher the IF, the higher the level of understanding due to a stricter peer-review process or a higher level of writing quality. However, this relationship was not found. In fact, an increase in IF by five corresponded to a decrease in SOLO level by one. Could this be because high-impact journals are cautious about publishing articles related to waxing trends? Those articles that repeatedly cited the trend-starting article demonstrated a better understanding. It could be that when an article is referenced more often, authors have or develop a better understanding of it.36 However, we also found articles37 ,38 in which there was a consistent repetition of the same mistake of misinterpretation.
For policy-makers and managers, it may be of greater importance to maintain legitimacy by embracing the latest trend rather than to do what is best for the organisation.39–41 Perhaps, this is the case for researchers as well? These explanations, while worrisome for the credibility of healthcare management research (and research in general), provide a plausible, albeit probably partial, explanation for why management concepts have short life spans. A poor level of understanding could also explain some of the difficulties described by Walshe1 in repeating quality improvement successes when scaling up or translating to new contexts.
Only five articles applied the theory empirically (SOLO level 3). The lack of extended abstract (SOLO level 4) could be a reflection of a nascent understanding inherent to a new field. Since the VBHC concept is not based on empirical data, research that is conducted without a relational or extended abstract understanding could be a sign that healthcare is adopting the concept without the rigour, scepticism and evidence base that is otherwise expected in medicine.42 If we have learned from the experiences of lean and PDSA,5 ,6 we should consider a scenario where the level of understanding will not change as more articles are published or that applications will not adhere to the principles of the concept. This may provide a clue as to why management concepts in healthcare follow 3–5-year cycles. If policy-makers, healthcare managers, researchers and clinicians have not invested in developing a deep understanding of the concept, it can be easier to look for a new one when the novelty has worn off. The level of understanding may, therefore, play an important role in the diffusion, or dilution, of management concepts.
The low SOLO levels suggest there is a need to learn about the management concept instead of just implementing the tools. Maybe, we now have an additional explanation for why years after quality initiatives, basic principles may have yet to reach the depths of the organisation.43 While training has been identified as a success factor,34 ,44 Thor et al45 describe how experiential learning (which includes reflection) lessened management scepticism. The higher SOLO levels often require reflection, and maybe, therein lies a possibility to avoid turning a ‘new’ management concept into the next management trend by enriching training with sessions that encourage learning and reflection.
Conclusions
Knowledge purveyors have a role to deepen and diffuse, not dilute, the concepts that policy-makers, healthcare managers and clinicians apply. Our results suggest that the level of understanding achieved by knowledge purveyors may influence the diffusion of innovations, particularly since they are often practitioners themselves. The application of the five-step approach presented in this study on other management concepts in healthcare would help to validate further this conclusion and the method itself. In the short time since we completed the analysis, more than 141 new articles have been published that cite ‘What is value in health?’13 Will the fate of VBHC be any different from its predecessors? Based on our review, little indicates that it will be otherwise.
Acknowledgments
We would like to acknowledge the enthusiastic support provided by Mesfin K Tessma in reviewing our statistical analyses and the Clinical Management Research Group for their feedback on the manuscript. The authors declare that they have no conflicts of interest whether related to financial interests, activities, relationships and/or affiliations.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
- Data supplement 1 - Online supplement
Footnotes
Twitter Follow Carl Savage at @SavageCarl
Contributors CS presented an idea that the authors together developed into the current study. JJF and DE contributed equally to the data collection and analysis and share first authorship. CS reviewed the analyses and arbitrated discrepancies. All authors drafted and revised the paper together.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement The excel file, in which the citing text articles have been analysed and categorised, is available upon request.