Behind the scenes, we’re constantly improving the search engine that powers the website.
At the beginning of December, we made a major upgrade to our search technology to make your searches faster. Our lead Search Engineer, Dr Boon Low, explains what we did and what it means for you.
As The British Newspaper Archive’s lead search engineer, my job is to deliver the best search experience we can for our customers. It’s a complex job – every time a search is performed, we look through billions of words spread across hundreds of millions of articles and millions of pages, and try to bring back the most relevant matches for what you’re looking for.
The team’s work revolves around three main areas: improving the search interface, so it’s easy to search and your results are presented in an easy-to-understand way; interpreting what you’re searching for and bringing back the most relevant results; and finally, doing this quickly and efficiently, so your results are delivered in a flash.
Over the course of the summer, we started looking at ways to increase the speed of searches. The first thing that we looked at was increasing the speed of the software that processes your searches and returns the results (turbo-charging the brains of the search, if you like). We also looked at hardware improvements we could make to give the software the best possible chance of working at full capacity (beefing up the body that supports the brains). We then started building new search machines and testing how they worked with the huge volume of data that sits behind The BNA until we had found a build that we were happy with.
By the end of November, we were ready to go, and now came the tricky part: putting it all live. This involved bringing the new machine into service on the live site and making sure that it worked well with the older machines. We then took each older machine out of service, upgraded it and put it back into service in turn until the whole farm of machines was upgraded (yes, we really do call it a farm!).
It’s harder than it sounds, because we made the changes directly to the live system, rather than take the search down while we made the changes. We’re happy to say that we completed the work successfully over 3 days, and with no downtime at all – in fact, you shouldn’t even have noticed that we were doing anything behind the scenes.
So what’s the end result? Well, it’s fair to say that we are all delighted with the improvement in speed, which was better than we had anticipated from our early tests.
Before the upgrade, we were taking 1.6 seconds on average to return results for search queries. Since the upgrade, average search time is down to 0.6 seconds – we have managed to shave an entire second off.
To put it another way, the vast majority of searches on The BNA are now completing in under a second – around the same speed as searches on Google – and the average search is now over 250% faster.
We hope you notice the difference!