3. Data Source, Tools & Methodology

The Majestic Million

Majestic surveys and maps the internet and has created the largest link intelligence database in the world, from which we referred to The Majestic Million, a free to use list of the million domains with the most referring subnets. We filtered the list by UK TLD and exported the first 1,000 results to inform our data set in this research.


“Top” is a subjective notion, and may be subject to differences of opinion as to how to define. The Majestic Million uses link data and therefore may not correspond to data sources that rank sites by traffic. In addition, we found that our list of 1,000 UK domains contained some sites that gamed their way into the data set using very manipulative link practises. Many of these were retail ‘fake’ sites and many were returning a 500 status and were no longer available in the time between cutting the data and getting to work. We removed these URLs from the data-set meaning the final list that went to audit contained 739 URLs.  There may still be some poor quality websites that remain within our source list however we had to balance the interest of data hygiene with having a robust sample that would largely match with what most web consumers would consider to be top websites in the UK.

Finally, to filter The Majestic Million to the UK region, we used the TLD filter with UK value. What this means is that we are missing .com domains that serve UK audiences. Notable exceptions are social media giants like Facebook and Twitter.

The data was extracted and cleaned from The Majestic Million at the end of October 2017 and audited by November 10th 2017.


Source: Audited Sites from The Majestic Million


The Lighthouse Audit Tool

Lighthouse is an open source audit tool and Chrome developer’s project, used to evaluate many performance criteria across four areas which are: Progressive Web App, Performance, Accessibility and Best Practises. You can either install the Lighthouse browser extension or use as the default audit tool in the Audit tab of Developer Tools in the Chrome browser.


At Erudite we believe Chrome developers are leading the way in educating the industry about Progressive Web Apps, and with the natural link to Google Search this is the best resource to help evaluate performance criteria that may set a website ahead of the competition.

We manually ran Lighthouse on each of the sites in our data set and recorded a checklist Y/N if the site passed the audit criteria. To make the work succinct and comparable we restricted our audit to these five Progressive Web App characteristics, in which we posed these questions:

  1. HTTPS – Is the site served over HTTPS?
  2. Service Worker – Does the site register a Service Worker?
  3. Speed/Time to First Interactivity – Is the site interactive in less than 10 seconds on a 3G connection? (<10 Seconds TTFI)
  4. Add to Home Screen – Does the site prompt the user to Add to Home Screen?
  5. Content Sized for Mobile Viewport – Do they serve content that fits right to the device viewport, either by serving responsively or adaptively?


Source: Lighthouse – Tools for Web Developers


“Our industry has seen some significant changes in the 12 years Majestic has been crawling the web, but none as significant as the pivot to mobile web use. We’re really excited to see how the data looks when search engines increasingly prioritise algorithms according to mobile device performance.”

Dixon Jones, Global Brand Ambassador – Majestic

Download the PDF Tweet