Care Opinion Canada receives and publishes stories about a wide range of health and care services. Every story is moderated before publication.
On publication, we send email alerts to many different kinds of organisation, including providers, commissioners, regulators, universities and patient groups, to facilitate responding, learning, and change.
When we moderate each story, we add a “criticality score”.
What is the criticality score?
The criticality score is provided as a service to staff using our system, so that:
- Staff can quickly identify stories which might need urgent response, urgent action, or escalation
- Staff can set alerts filtered by criticality, so that relevant people can be notified of serious issues, while not being alerted for stories which raise no concerns
- Reports can be created which focus on issues of concern which may need further review and action
The criticality score acts as a signal to help prioritise attention. The purpose is to enable staff using Care Opinion Canada to filter and focus feedback to significant issues for action. Reflecting this, the score is not public, and is available only to logged-in staff users.
How is the criticality score assigned?
The criticality score is the moderator’s assessment of the most negative statements contained within the story. This assessment is made by reference to a set of criterion-based criticality definitions.
Note that the criticality score:
- Is not the story author’s own “rating” of a service
- Is not Care Opinion Canada’s assessment of service quality
- Is not the “average” sentiment of a story based on all the statements it contains
How can staff know the “overall view” of authors towards their services?
The criticality score does not reflect the overall view of a story author, because as noted above:
- It is assigned by the moderator, not by the author
- It is focused on the most negative statement, not on the story “as a whole”
So the criticality score is not a good measure of author’s “overall view”. Instead, this can be assessed using the author’s own ratings of the service, if the author has provided those during the story entry workflow, or via the provider’s own survey. These ratings will reflect the authors own views of each service.
Should Care Opinion Canada provide a “positivity score”?
Instead of the criticality score, should Care Opinion Canada provide a positivity score? Such a score would have to be based on the most positive statement contained within a story.
We think this would provide little or no working value to staff. For example, a story containing the phrase “After everything that happened, Mary stood out as a shining beacon of hope” would be rated highly on such a score.
Yet the story might contain a range of serious criticisms of the process and outcomes of care which would merit immediate attention and the involvement of senior staff. A positivity score would give no indication of this. Nor would it provide some other important benefit to compensate for this loss of value.
Should Care Opinion Canada rate the “overall” sentiment of a story?
Should Care Opinion Canada rate the story as a whole to reflect the balance of the positive and negative statements contained within it?
We have noted that this would reflect a moderator’s view of the experience as a whole, rather than the story author’s own assessment of the experience as a whole, which we would prefer.
Additionally, it is unclear that this would be either practical or indeed meaningful. For example, consider story A: “My mother was treated appallingly and rudely by almost all staff. Only Jackie, who was wonderful, saved the day.”
What is the “overall sentiment” here? Is the rudeness of most staff counterbalanced by the helpfulness of one other? If so, perhaps we should rate it in the middle of the scale.
Now consider story B: “Everyone who cared for my mother was fine, but nobody stood out.” Again, what is the overall sentiment? It seems to be towards to the middle of the scale again.
Attempting to judge the overall sentiment has resulted in the same rating being given to two stories which are clearly very different. Indeed, story A presents obvious opportunities for learning and reflection, yet an overall rating fails to identify this at all.
Should Care Opinion Canada provide a criticality score for each service linked to a story?
Some care providers understandably feel that for many stories with a high criticality score, linked to multiple providers, the score “doesn’t apply to us”. They would like a separate score for each provider.
The main issue here is cost and feasibility. In many stories, it is difficult or impossible to assign different parts of the story to different providers. Attempting this would inevitably require more moderator time, training and audit, and so would increase subscription fees for everyone.
In addition, stories linked to multiple providers often reflect a pathway of care in which each provider “did their bit” but the pathway overall was not joined up. Attempting to assign a score per provider might easily suggest the pathway was satisfactory, when in fact it was not: and this should be of concern to providers, even if they believe their own care was good.