An introduction to search engines
Search engines have become a prominent feature of modern life. Most of us use them to navigate not only the internet, but the world around us too. If we're searching for an answer or information, it's unlikely we'll reach for a book or an encyclopedia – rather, we'll use the internet to conveniently answer our questions.
Through decades of web browsing, search engines have become recognized as the go-to way of sourcing information quickly online. In fact, the term 'googling' was officially added to the dictionary in 2006, making online search engines the preferred way of finding what we want, instantaneously.
For the vast majority of web users, search is also an integral aspect of online navigation. When visitors arrive on a website, they probably have a destination – or at least an intention – in mind. Search enables this by concisely converting visitor queries into relevant results that lead straight to great content or products.
Complex navigation, unclear categorization, poor user interface, and substandard search design can lead to major barriers in user experience (UX) and hinder user journeys through your website.
To counter this, an effective and robust custom search engine empowers you to:
- Streamline user experience (UX)
- Improve product/page findability
- Target user interface shortfalls
- Optimize paths to conversion
- Track and understand your customers
- Embrace new search technology
- Boost brand and website recognition
What's more, custom search engines can gather a wealth of metrics that tell you more about customer behavior – opening doors for website optimization opportunities and better understanding of visitors.
What is site search?
When you browse a website – searching for the product or page you have in mind – it's likely your cursor will go one of two ways: either to the navigation menu or to the search bar.
On-site search is essentially a search engine bespoke for a website, focused exclusively on the site's contents. It lets visitors search the full scope of a website without needing to trawl through the jungle of (sometimes) poorly-mapped navigation or numerous pages.
For a visitor, an effective site search can be a saving grace – particularly if the site is content-heavy or has a large selection of products. For a website administrator, it can be a powerful tool that provides deep insight into the way their visitors behave.
Why is site search important?
The necessity of on-site search intensifies as a website grows. For modern businesses, a compelling online presence is essential, so websites should be designed with user-friendliness and engagement in mind.
Site search enables this by providing better on-site functionality, improved product findability, and a shorter path to products or pages – all key aspects of UX design. This reduces pain points for your visitors and breaks down barriers that may otherwise make them bounce off your site.
Navigation vs. search
Navigation and search are critical components of UX design, yet they can compete for viewers' attention.
In one of the studies half of the respondents looking for products online said they prefer to use on-page navigation rather than a search bar. Don't be alarmed – this doesn't mean your site search plans are doomed or wasted on visitors who'd rather navigate by click (and bear in mind the other 50% of people still go straight to search!). It simply demonstrates that the navigation bar can be the more obvious starting point.
When it comes to navigation vs. site search, it's important to consider user intent and the individual visitor's stage in the buying cycle. People who use site search are typically found to be at the late buying stage (where a purchase is more likely), and conversion rates through site search can be up to 50% higher on average.
Moreover, regardless of how well your site hierarchy and navigation is designed, there will always be visitors who don't know how to use it – or don't want to. Therefore, it's wise to cater to those who prefer to find their way around via an on-site search function. Why? These visitors are more likely to convert compared to regular browsers, and they want/expect a shorter journey to find what they need.
Navigation acts as a map for browsing visitors, while search enables users to take control and make informed queries. This doesn't mean you should neglect either – both tools are reliant on each other to create a streamlined and well-rounded user experience. On average, 74% of businesses find that consistently improving or catering to user experience will boost conversion rates, while simultaneously building brand loyalty and user engagement. Therefore, it's equally important to focus on search.
A few factors to consider:
- Search empowers website administrators to collect and utilize a wealth of data.
- The combination of both search and navigation strengthens your website UX and provides users with more pathways to find what they need.
- Site search analytics enable you to gain insights into what's being searched on your website and how often these queries are made.
- When a certain search term is consistently entered at a high frequency, it can be an indication that this is an item you ought to feature in your navigation menu. In this way, search and navigation are complementary to each other.
- When you are able to effectively analyze the results, site search acts as a form of user testing, which helps remove the guesswork from the design/optimization process (according to 66% of businesses).
How are people using search?
Unlike standardized site search tools, custom search engines offer in-depth analytics that can reveal the intricate search patterns of your website's typical visitor.
These insights and metrics can include complex details such as what your users are looking for, what time of day they're searching, keywords, click-through rates of search results, evolving trends, and much more.
84% of businesses don't actively optimize and measure their site search – so custom search engines can provide a competitive edge over rivals. Deeper insights allow businesses to use data-driven decision-making to adjust their methods, enabling them to achieve important commercial outcomes such as increasing conversion rate or revenue.
For example, take the 10 million search queries we've logged from Site Search 360 users. These metrics provide an insight into just how a custom search engine can help businesses improve their site search.
Here are our top tips:
- The average query length is 1.6 words or 11.34 characters. To cater to this, your search box should be wide enough to contain 26 characters, covering 95% of queries.
- Most users expect the search bar to be found in the top right-hand corner of the page. The left-hand corner is the second most expected placement.
- Only 20.2% of queries are searched more than once, meaning 80% of entries are typically unique (only seen once). Your site search engine should be able to cope with unexpected queries.
- Your chosen search engine should account for advanced syntax, such as the inclusion of characters or numbers. 23% of queries will contain apostrophes, and over 15% include one or more numerical digits.
- 19.6% of queries will return no results. You need to ensure your 0-result rate is as low as possible, so choose a site search option that allows you to analyze search data and alter your content or product descriptions accordingly.
Now imagine this level of insight specific to your website queries. Your business could instantly identify trending search queries, product indexing/labelling issues, or any informal terminology that visitors frequently use.
Moreover, custom search engines enable you to view these details as frequently as you like, ensuring you can adjust and optimize your website in response to your user's search experience in real time.
When a user begins to type a search query, auto-complete can instantly suggest relevant results.
Source: sitesearch360.com after analysing 10+ million search queries
Why are custom search engines the perfect choice?
Not all search engines are created equal. If you've ever been hit with a 'no results found' page, then the frustration will be all too familiar.
An effective site search tool is essential if you want to ensure your visitors will never be left scratching their heads over where to go next. But what are the benefits of opting for a custom search solution?
Semantic search
'Semantic search' refers to the process of discovering not only keywords but also the intent and contextual meaning of a search query. Good custom search engines will factor in the intent of a query, the visitor's previous searches, contextual relationships with other keywords, and the relative success of similar queries to infer the best results to display.
When a search is not semantic, it only focuses on keywords – thus the results are dependent on whether A) the user searched an exact keyword or B) the keyword is featured in the desired content/products. If neither A nor B are achieved (which is very often the case) users struggle to get results. This is a common frustration for web administrators, as many baked-in CMS search tools don't have semantic search capabilities, which can lead to irrelevant or 0-results.
Consequently, semantic search is essential for improving product findability and significantly improves the user's search journey as well as on-site UX. It's a key component of turning your search from a trial-and-error function into your most powerful site-navigation tool.
Natural language search
The way we communicate with a search engine is largely dependent on our intent. If a user searches 'how do I __?' or 'where to find a ___,' they expect to find helpful and relevant results in the blink of an eye. Natural language processing is the key method of achieving this and works by translating complex human phrases or words into queries that an AI algorithm can easily digest.
Human language is riddled with colloquial and informal phrases, and many of your website visitors won't be inclined to put on their 'robot hat' and input rigid search queries formed only of exact keywords. Advanced custom search engines recognize and understand informal language, synonyms, questions, and statements that relate to your keywords by using natural language processing – so that even complex search queries can be understood. This reduces the likelihood of irrelevant or 0-results, and instead quickly provides appropriate results.
Actionable analytics
If you identify with the all-too-common group of webmasters who aren't sure how often your search is used, how well it works, or what people are searching – then custom search engines can fill that important gap.
When a user is on your website, it's important to know whether their journey is heading in the right direction. Custom search engine dashboards can grant you all the metrics your heart desires, with a wealth of analytics and data on user behavior. For example, custom search engines deliver deep insights about each individual search query, such as:
- Search loading time
- Amount of results for a search term
- What search result was clicked
- The position of the clicked results (first, second etc.)
- Bounce rate (how many times users don't select any search result)
- Frequently used search terms
- Amount of search queries over time
- Traffic flow (where your visitors are going)
- Popular content/products
- Trending search terms
- Total unique searches
- Sessions with search
- Search refinements
These metrics provide a much deeper understanding of visitor behavior, allowing your business to adjust and optimize your website products, pages, and design accordingly. If you're able to fine tune your site to improve the user search journey, then visitors will be more likely to return.
Uncluttered design
Part of creating the perfect on-site search experience for your visitors is ensuring their journey is streamlined. Advertisements on your search engine results page (SERP) can distract your visitors and leave them looking outwards rather than focusing on their on-site journey. Furthermore, such clutter can be annoying and burdensome when scattered throughout your SERP – further fragmenting the customer's journey.
Search should be consistent with the branding of your website, so the perfect custom search engine should give you the ability to go ad-free with minimal external branding. It's worth noting that even Google custom search has no option to remove Google branding and leaves businesses vying for attention against their competitors' paid-for advertising on their own website – so it's important to pick the right solution.
Smarter search
Custom search engines give you the power to do more. It's not just a search function for your customers, but an essential tool for your business to get better control over your website. There are so many small, yet powerful, details that can be optimized by using custom search engines.
For example, you can add search tools to your 404 pages, adjust how many or how few automatic crawlers you want to index your site, include search on your FAQ, and support different language queries.
Additionally, you can cross-domain your search tool and index content from different sites onto one index – so visitors can use your search across all your domains from a single search bar. If you're a business with content published across multiple places, this can help users source results easily and increase page views across your sites.
Platform-friendly
The future of site search is no longer just about 'type and click.' Mobile, voice, and visual search are becoming increasingly popular – and custom search engines are better equipped to prepare for these growing trends. The rise of mobile voice assistants such as Amazon's Alexa, Google's Assistant, or Apple's Siri means that up to 34% of people are now searching and purchasing products using voice search, so voice search capabilities are essential for businesses who want to stay ahead.
It may not seem urgent now, but customer expectations are slowly leaning towards more innovative search capabilities. Custom search engines enable you to stay on top of new search processes and adjust your search tool for emerging trends.
Buy or build
Custom search engines remove the hassle of building a tool internally. When businesses choose to make their own search solution, it can be a drain on internal development resources, especially if there isn't a developer dedicated to the project.
For businesses with the resources, architecting your own custom search engine will give you full control over every aspect of your search tool, but bear in mind your project time constraints, finances, and skill level.
The value of customization
By adding a custom search engine, your business can better tailor its UX to your website users. It's likely that an out-of-the-box, basic site search may be enough, but by adjusting certain aspects you can create a unique search function bespoke to your website.
Style and corporate identity
When you provide your visitors with a search solution that has been optimized for your website, they will find what they're looking for more quickly and experience a streamlined search experience. For example, custom search engines can be configured using a search designer to create your desired theme, color scheme, tracking tags, style, accessibility options, suggested queries, result settings, layout, filters, and navigation – giving your business full control over each important aspect of on-site search.
When your on-site search software is seamlessly integrated with your website both functionally and visually, it is seen as a built-in, not bolt-on, tool.
Visually, this allows you to create a search that is consistent with your brand image and corporate identity. Functionally, it prevents your visitor's journey from being disrupted by third party ads, or external websites.
SERP page
One way custom site search allows you to customize and boost your pages is by using data points based on certain criteria to create unique search results. Data points are facets that extract specific information from pages and feature them directly in the snippet of the search result. For example, a food retailer that has a range of content could choose to display “calories” for recipe search results, or “price” for product search results. This provides more control over how your search results are displayed, which is especially important for websites that have different types of content.
Additionally, custom site search allows you to create content groups to categorize your search results. For example, search queries can be sorted into categories such as news, images, or books. This provides better navigation on your search results page and improves the findability of the content on your website to ensure your visitors can find exactly what they need.
Similarly, you can optimize your SERP by reordering results, redirecting specific queries to specific URLs, or by rewriting queries to direct users to keywords. Rewriting or redirecting queries enables your search function to instantly send visitors towards more relevant results, and reordering results can enable you to place the most relevant ones at the top. This is useful for businesses who have certain campaigns, objectives, or goals for the month, as they can promote certain products or pages. Most importantly, this shortens the user journey and helps visitors find what they're looking for quickly.
You can customize your SERP with content groups and use data points based on certain criteria to create unique search results.
Custom filters
By creating custom search filters bespoke to your website and your visitor's search query, you empower users to search more precisely by enabling them to easily drill down into large sets of results. For example, in a product search, filters could be price, color, or size. If your user has searched a term with a high number of results, custom filters provide them with contextually accurate filters for their search term, enabling them to get to what they want, fast.
Your visitors' search journey - the key elements
Powerful on-site search is a valuable tool that provides greater levels of customization and streamlines UX. With this in mind, what are the key aspects of each individual user's search experience – and how can your business use custom site search engines to improve UX?
What are the key elements of an effective search?
- Presentation – how does it look?
- Processing – how does it work?
- Results – how does it deliver?
Presentation — how does it look?
You don't need flashing buttons or funky fonts, but the presentation of your on-site search function is still important.
Identity and brand
On-site search engines should be customized to match the image and identity of your brand. This ensures consistent visuals for your users' search experience and emphasizes brand identity throughout each visitor touchpoint.
Contrasting visuals/color schemes or logos from third party ads or websites can be off-putting for users searching on a website, so search presentation should express your brand individuality and be free from clutter.
Placement
When it comes to search box placement, consistency is key. A study by A. Dawn Shaikh and Keisi Lenz found that the ideal placement for a search box is the right-hand corner of the page. It's the most familiar placement of search, and so visitors who are new to your website can instantly deduce where to go.
Additionally, the search tool should be available on every page. This ensures that a search can begin no matter where a visitor is on their journey through a website.
Stylization
Site search engines should be easy to identify and visable at first glance for new visitors, with clear stylization that labels the purpose of a search bar. At a minimum, your search bar should have placeholder text that invites your users to type in their query, and a maginifying glass icon - a well recognized symbol to denote the search function at a glance.
Icon vs. button with text
Because today's web users can easily recognize search functionality labeled by the magnifying glass symbol, spelling out the word 'search' is generally unnecessary. This is great news for international businesses, as images are universal in a way that words are not – the familiarity of the magnifying glass means there's no need to translate 'search' for each region. (However, for accessibility you should include the word 'search' or a translation as alt text for graphics-only buttons.)
However, if you can spare the space on your site, a traditional search button, labeled 'search' next to the input field will maximize its findability – just make sure you still include the icon too!
Icon-only search box?
Icon-only search – i.e. a clickable magnifying glass symbol alone – can work well for the mobile version of your website, where visitors are used to seeing less text and a different style of navigation.
However, we don't recommend this style for your desktop site, as it can make the search function a lot harder to find. And once visitors do locate it, they'll need to click and wait for the input box to appear, work out where to type, and sometimes click yet again to focus on the input field. This creates friction in the user experience, whereas an open-entry text field, ready for typing, lets the user simply click once or hit Enter to begin the search journey.
Styling voice search
It's not necessarily clear how best to signpost your voice search. Combining a text-labelled button, search icon, and a voice-specific icon takes up space and looks messy. Clear iconography is the way to go here – for example, Google uses a magnifying glass and microphone:
Do remember that not all browsers have adopted speech recognition API. For example, voice search functionality may work in Chrome or Opera but not be available for Safari users.
Processing — how does it work?
Auto-complete
The behind-the-scenes processing of site search engines is integral to the quality of your users' search journey. We've found that 9.2% of users will click on recommended search results powered by auto-complete, so an effective search engine should include suggestions that are contextually fitting to the visitor's original search query. This is an effective way to guide users towards relevant results – enabling them to find what they're looking for more quickly.
Lightning-quick search
Search engines should offer a lightning-quick search. The average service response time of a robust custom search engine is between 10 and 100ms (20ms on average) per request. This can be achieved by automatically routing users to the closest server, guaranteeing them results, fast.
Additionally, it's useful to cache recent queries so the suggestions and results are pulled even faster for future searches. Considering 40% of people will abandon a website if it takes more than three seconds to load, this aspect of search engine processing is central to success
Platforms
Businesses should consider that not every user will access a website through a desktop. In recent years, more and more people are whipping out their mobile devices to browse, making mobile the most popular platform. On-site search engines should be platform agnostic, and – combined with website design – should automatically detect device-type and adjust accordingly.
Source: www.statista.com/statistics/241462/global-mobile-phone-website-traffic-share
Accessibility
Everyone should be able to easily navigate your website, and search engines should serve the increasingly accessibility-conscious world. Functions such as voice-command search or supported languages can not only deliver a better user experience for everyone but also help businesses remain cutting-edge and on top of trending new ways to search.
Thanks to sophisticated speech recognition, voice search enables users to make queries by speaking into their microphone – eliminating the need for a type-interface that many people may find difficult to use.
Additionally, custom search tools can be altered to meet accessibility guidelines such as WCAG 2.0. These guidelines are designed to direct businesses towards user-friendly interface components that are easily perceived by all users. For example, the perceivable, operable, and understandable components of on-site search such as text alternatives, screen contrast ratio, text resizing, or page timing. This is especially useful for government institutions or schools that want to make their website easy to use.
Lastly, custom search tools can support assistive technology such as screen readers. For people who are visually impaired, illiterate, or with learning disabilities, screen readers are essential for conveying displays into non-visual means. For example, text-to-speech, sound, or braille.
Results — how does it deliver?
The search engine results page (SERP) is the highlight of the visitor's search journey. After firing off a search term, your users should be greeted by a shiny page of relevant and appealing results.
There are a variety of ways the SERP can be displayed. Starting with the most basic, a result list contains the title, link, and search result snippet, and is easily understood by the user:
If your business wants to display visual results, a grid layout SERP can be best-practice. Displaying images can be useful if your users will likely be searching for pages or products for which visual references are useful e.g. clothing or artwork:
Your SERP could contain tabs that let your user choose which category they want to see results from. For example, Google provides not only website results, but also categories such as images, news, videos and shopping:
The ability to choose which option is best for your business is what places custom search engines as the front-runner of UX. When your SERP is tailored to the users of your website, they'll find what they need quicker, and enjoy a smoother search experience.
Analytics
The last key element of the visitor journey is how user search queries/results can provide your business with a wealth of analytics and data about how users interact with their given search results. This data can allow you to optimize your website, content, and products further – by identifying aspects of the search process that fall short.
For example, you can infer whether you should create synonyms for new frequently searched terms, or if you've indexed content/pages correctly. If visitors are searching a term often, but they're not clicking on the given results – the analytics of your custom site search should help you spot this, so you can make alterations, add new search terms, and remove search term dead-ends.
Implementation challenges
Installing custom search engines into your website should be simple, but it varies from solution to solution, and what works beautifully with one CMS might suddenly be a tricky beast when paired with another. Indexing can cause a headache, and complex queries (and where to put them!) can create confusion.
We believe software should play nice with a wide range of platforms. Using our own custom search tool, Site Search 360, as an example, here's how it can reduce your pain points for each respective CMS:
JavaScript
The majority of CMS platforms such as Wix require JavaScript code to install custom search engines. Site search does not come as standard for many CMSs – or is extremely limited in its capabilities.
To install a search tool through JavaScript, users simply copy the script into their site before their </body> tag and use configuration parameters to customize the way search results look and behave. This provides webmasters with all the great in-built benefits of a custom site search engine, without the hassle of building a bespoke solution for themselves.
Users with limited technical knowledge can use a search designer tool to remove the headache of sourcing external development for their website, which can be a costly procedure. Additionally, custom site search solutions are constantly kept up-to-date, are backed by developer support, and are security-compliant.
Security
The most robust site search platforms are compliant with GDPR requirements that ensure the protection of personal data. For example, Site Search 360 does not sell, rent, trade, or otherwise transfer any personally identifiable data without user consent. As an independent data controller who processes visitor data on your business's behalf, Site Search 360 includes pre-signed data processing addendum (DPA).
All sensitive information that is submitted via the site search platform is encrypted via a secure socket layer (SSL) technology, and the communication between your site and the Site Search 360 server is secured via HTTPS request.
WordPress
Although WordPress features a basic site search tool as standard, its reach is limited. The built-in tool does not feature faceted search or semantic search, and has minimal customization options. Most significantly, it doesn't allow you to search through additional custom modules such as banners or sidebars, so you can't determine what content should be indexed, featured in the search results, or skipped (headers, footers etc.)
Custom site search can tackle these issues by providing contextual and customizable search for your WordPress site. You can determine what content should be indexed and, more importantly, visitors can retrieve relevant results, no matter how vague or colloquial their search query is, and filter their results instantly. Additionally, custom search is available as a WordPress plugin, so is easy to install and deploy.
Cloudflare & Lightspeed
Similar to most CMS platforms, Cloudflare does not come with a built-in site search tool. Websites built using this platform leave visitors at the mercy of poor site search, leading to worsened product/page findability, user frustration, and reliance on the navigation.
Our custom site search solution identifies the seven key parameters that have the highest impact on a Cloudflare website's look and feel, and configures visually to your website. To install, simply download the Cloudflare application and enable it on your account.
We also offer integrations with Lightspeed and Drupal, and the list keeps growing.
The baked-in search function of most CMSs fails to match the level of control and insight of custom search engines, so we made it easy to implement an effective and insightful custom search tool on these sites. In turn, this gives businesses the power to improve user search experience, on-site UX design, and website optimization processes.
Keeping your data up-to-date
Custom search engines automatically update your data on a regular basis – as no one wants to manually re-index their site after every single update, article edit, or page deletion. However, there are also manual ways to edit and update your site index. We've compiled a knowledge base that details the manual ways you can keep your search index fresh, but here's an overview of possible indexing strategies:
Automatic crawlers
Put simply, a 'crawler' refers to an internet bot that systematically browses the internet for the purpose of web indexing. Web indexing is how the content of your website will be understood by search engines and makes your pages eligible to show up in search results.
Crawlers work by visiting your “root URLs” (i.e. your homepage) and following the links that point to pages within your site. These can be identified by creating a sitemap that contains the URLs from all sites that you want to index – the crawlers then automatically add it to your global index. You can also configure the crawler to follow links to any subdomains of your page. This can easily be optimized in your crawler settings found on your custom search engine dashboard.
APIs
You can simply add pages from any of your sites via the API using your API key. You can either index by URL or send a JSON object with the indexable contents. By using the API, you gain full control over the content in your search engine and choose how you index your data.
Zapier
Zapier is a tool that allows your business to connect the applications you use (i.e. email, instant messaging tools) to automate tasks involved in using Site Search 360 and save time. For example, you can set up a 'zap' via Zapier to send you an email once your site indexing is complete, keeping you up-to-date about the behind-the-scenes of your site search engine. This allows web applications to share data, pass information between your programs, and automate processes so you can achieve more.
Make
Make allows you to create customized scenarios for the repetitive tasks involved in using Site Search 360. Not only can it connect apps, but it can also transfer and transform data – enabling you to automate your workflow by setting up triggers on specific custom search actions (e.g. running a site re-index).
In conclusion
When businesses utilize a custom site search tool, they enhance their users' search experience and streamline the customer journey through their website.
Catering to visitor behavior through website optimization empowers your users with a clearer search pathway – not only helping them find the items/content that they need, but keeping them coming back, again and again.
By embracing custom search engines, businesses can stay a step ahead by curating search results and providing a more personalized search experience to each unique user. Additionally, the rich metrics provided by custom search engines deliver deeper insights into user search behavior – enabling businesses to optimize their website through data-based decision-making.
Lack of, or poor, site search functionality will cause friction to user journeys, so it's important to cater to a streamlined on-site experience – whether your wheel spins on products, content, sources, or services.
Happy Searching!
