The Smarter Guidance team has been working to make content smarter -- make it more readable and clearer for users.
But how do we measure if we've been successful? I've devised one way and while it'll take more than a blog to go through the overall figures, let's look at one example - bluetongue.
Bluetongue is a notifiable disease and its guidance was re-written by the Smarter Guidance team last year. The goals of Smarter Guidance are simple - to make it clear to the reader:
- who the content's for
- what they're going to find
- what they have to do
Ways we do this include:
- better titles and summaries to make it clear if the content is right for the reader
- shorter, plain English content with a focus on what must be done to follow the law
- addressing the user and making them the focus of the content, not Defra
Bluetongue: content compared
| Test | Old version | Smarter Guidance version | Result of changes |
| Titles | 1 word:“Bluetongue” (although it was in the A to Z of notifiable diseases, which all have brief titles) | 8 words:“Bluetongue: how to spot and report the disease” | More descriptive title with a focus on an action. Makes it easier to distinguish with other pages, eg, “Bluetongue in Greece and the European situation”. |
| Content length | 2,874 words | 304 words | Almost a tenth of the original length. Along with a clearer title, this makes it more focused. |
| Readability | Readability scores (the lower the number the better):Fog index - 18.5 Flesch-Kincaid - 14.6------ Words per sentence:20.1 ------ Syllables per word (longer means more complex words): 1.9 Non-dictionary words (potential jargon): 32% Percent complex words: 26% |
Readability:Fog index - 12.0 Flesch-Kincaid - 8.4------ Words per sentence (includes bullet points):9.9 Syllables per word: 1.7 ------ Non-dictionary words: 21% Percent complex words: 21% |
Readability has improved. Shorter sentences with fewer complex words. Also has fewer words that readers may not understand.Main need clearer - to get users to report suspected bluetongue outbreaks. Includes advice on signs to look for.Old version was a mish-mash of news, legislation and history of the disease, along with how to report it. |
| Focus of the article (word count per 1,000 words) | 'You'/'your': 2.8'Defra': 1.4
'Must': 1.4 |
'You'/'your': 19.3'Defra': 3.2
'Must': 3.2 |
Counting the number of certain words and how frequently they appear. This serves as a way of getting an idea of who the content is aimed at. References to the audience as 'you' have gone up. |
We look for 'must' as this is the word we tend to use when telling the user what they have to do to follow the law. 'Defra' appears to have gone up (not good - as this suggests we're writing for Defra, not the user) but we're looking at frequency per 1,000 words and the article was only 304 words.
The other thing to be aware of is that this analysis is condensed and the programs that run the analysis are just that -- programs, so it's still down to our judgment for what the output means. Also with the Smarter Guidance article only being 304 words it means that references per 1,000 words is skewed.
Other ways we could measure improvement include:
- contact rates to contact centres -- this is the first link in the Smarter Guidance version, has this made a difference to when and how users get in touch?
- instances of bluetongue reporting and outbreak - however, this would only apply if there is another outbreak (and naturally we hope there isn't one)
- Google traffic analysis - the GOV.UK page is second after Wikipedia in Google search, but this is good as we only want relevant users to come here (those who want to spot the disease, not find out general information)
Guidance compared
So is guidance smarter? Yes, measurably so. This is just one example but is indicative of the general changes Smarter Guidance has made and the expected benefits.
To give you an idea of these changes, here is a comparison of the most common words on the old Defra pages and the GOV.UK Smarter Guidance ones. Spot the difference in the intended audience.
Blog written by Jonathan Richardson.
2 comments
Comment by Joshua Mouldey posted on
This is a really useful example of measuring 'technical' improvements to content (before the 'testing with users' part).
Would you mind saying a little more about the tools you prefer to use to get those metrics? Particularly which tools give the:
- '% of foreign words'
- '% of complex words' (does this mean 3+ syllables?)
- focus/frequency
Comment by Jonathan Richardson posted on
Hi Joshua, I used a few programs. The main ones were the Perl Lingua modules (including Fathom). There is also Python Textstat that I've not used but has been recommended. For frequency I used AntConc to get the wordlist and some manual calculations to get the frequency.