Is your drop real, or did the mix change?

What it measures and what it does not

The cohort is the fifteen queries with the most impressions in the BEFORE period. The test looks those same ones up in the AFTER period. If they are still where they were, your average got worse because new queries arrived at low positions, and that is not a loss. If they fell too, the loss is real.

Positions are weighted by the impressions each query used to bring. Unweighted, query fifteen counts as much as query one, and that is not how traffic is lost.

Churn is reported separately from the verdict, and that separation matters. A query going quiet means opposite things depending on whether the whole set is growing: on a growing site it is the weak queries evaporating, which is normal; on a flat one, it is loss.

Three things to respect or the result is worthless

• Close both periods on complete months. With the current month, positions come from queries with one or two impressions and swing on their own. On one of my own sites the verdict moved from real drop to mild deterioration purely by closing on eleven days instead of the previous month.
• The BEFORE period needs volume. A cohort of fifteen queries with twenty impressions each measures noise.
• This does not rule out people searching less. Search Console carries no search volume, so that third cause gets checked in Google Trends, not here.

The case where this test was actually used, including the three bugs the first version of this calculation had, is in separating a real drop from a mix change. That article also shows where these reports come from and how to close the two periods.