An overview of our rater guidelines for Search
At Google, we like to say that Search is not a solved problem: We’re constantly making improvements (more than 4,800 last year alone). These changes can be big launches or small tune-ups, but they’re all designed to make Search work better for you, and to make sure you can find relevant, high quality information when you need it.
One of the key ways we determine if an improvement to Search works well is through the help of search quality raters. This group of more than 10,000 people all over the world work from a common set of search quality rater guidelines used to evaluate the quality of search results — which are publicly available. Today, we wanted to give you an idea of how these guidelines work, and how — just like Search itself — they improve over time.
What are the search quality rater guidelines?
The quality rater guidelines are more than 170 pages long, but if we have to boil it down to one phrase, we’d say they help make sure Search is returning relevant results from the most reliable sources available.
Information quality is at the heart of Search, and our systems fundamentally work to surface high-quality information. The rater guidelines help raters determine if a planned improvement is meeting that goal by providing a clear, uniform definition that all raters use to assess the results they see.
More specifically, high-quality information is content which demonstrates expertise, authoritativeness and trustworthiness on a topic, or E-A-T for short. For example, a health site with content from doctors and produced by a medical institution would have a high level of what many would consider to be expertise, authoritativeness and trustworthiness. The rater guidelines also define low-quality content on the web, such as content that spreads hate or seeks to deceive users.
Who uses the guidelines?
As we noted, the changes we make to Search are rigorously tested and evaluated by real people. Our search quality raters provide us with insights and evaluate pages against our guidelines to help make sure our systems — and proposed improvements — are working as intended.
What that looks like in practice is often a “side-by-side” test where a rater will look at two sets of Search results, one from the current version of Google and the other from an improvement we’re testing. Raters will review the pages in each set of results, and evaluate if the pages are a helpful match for the query based on our rater guidelines.
The ratings they provide don’t directly impact how a page or site appears in Search. Instead, they help us measure how well our systems are working to deliver great content.
How often are the rater guidelines updated?
Just like we make improvements to Search, we update the rater quality guidelines from time to time to make sure they’re working as intended.
Some changes are meant to tackle issues we’ve identified in Search, and may include expanded sections and new examples to help guide raters. For example, in 2017, we updated our guidelines to provide more detailed examples of low-quality web pages that included misleading information, unexpected offensive results, hoaxes or other content.
Sometimes, we identify concepts that are especially challenging for raters. We then make changes to the guidelines to improve ratings. In 2020, for instance, we provided new guidance on how to tell if a result from a dictionary or encyclopedia would be useful for a certain query.
Other changes focus on things like refreshing the language for clarity and updating organization. That’s what made up most of our October 2021 update, which included clarifications of what constitutes lowest quality content, and refreshed and modernized guidance on researching the reputation of websites.
We rigorously review, test and evaluate all changes to ensure they’re helpful and having the intended effect. And we have a publicly available log at the end of our guidelines describing in detail any changes we make. Ultimately, these updates are designed to make Search work better for you.
