With the vast number of active websites that are available at the fingertips of the modern-day internet user, it is vital that a search engine is able to display only the most relevant and important information related to a query.
So how can search engines possibly hope to display relevant information to their users when there are so many web pages?
The answer involves the processes of crawling and indexing, a fundamental part of how search engines work, and equally important when considering search engine optimisation (SEO). So, what exactly is crawling and indexing?
Crawling and indexing in SEO describes the process, carried out by search engines, of finding and storing information held on websites. Search engines use software called a ‘web crawler’ to find web pages via links. This information is then stored in a database, or ‘index’. When a user performs a search, the search engine reads from the index to provide the user with web pages relevant to the search query.
Using software known as a ‘web crawler’ (sometimes referred to as a web spider!), the entire world wide web may be searched in a complex and automated process. This method of data gathering allows search engines to cover vast quantities of web pages one link at a time (this is why internal linking is so important in SEO!).
The web crawler finds an initial link contained within a page, and then seeks to identify all the links that are accessible by that initial link. This process repeats, and results in the collection of many billions of web pages. These pages are stored in a huge database, or ‘index’.
When you search the world wide web using a search engine like Google, you are actually searching the index of sites that a web crawler has collected. Google estimates that each search performed using their engine involves the use of around 500 servers to ensure that a search is carried out efficiently, and that only the most relevant and important information is shown to a user.
Important: If your web pages aren’t indexed, they can’t be found on a search engine!
Crawling and indexing becomes a key part of how search engines work, and how they are able to consistently show the most relevant information when users search their index. To ensure that your website appears as the most relevant result for a user’s query, your website (and all relevant pages) must be crawlable and indexable.
If you do not want your website to be visible to a web crawler, you can add a ‘noindex tag’, which is an on-page directive that instructs search engines not to add your website to their index. Likewise, if you can’t work out why a page isn’t ranking, it could be a good idea to check that it doesn’t have a ‘noindex’ tag.
When it comes to SEO, crawling and indexing is a vital consideration for web developers wishing to perform well in search result rankings.
Websites that are not optimised for search engines to crawl and index their information will achieve lower search rankings than those that are, which will result in lower numbers of people visiting a website. This is because search engine crawlers literally can’t find pages to rank, which is why internal linking is so important.
In short: yes, you can! It is important that a web crawler is able to crawl through your site, and not just to it. You should therefore consider the tips below to ensure that your site is optimised for a search engine’s web crawler software. It is important that a web crawler is able to crawl your entire site, and not just part of it.
Our expert team is on hand to help with your unique SEO requirements! Take a look at our full range of SEO services, and be sure to fill out a contact form on our website and we will get back to you as soon as possible. Alternatively, why not check out our Knowledge Base hub pages for more answers to your SEO questions?
SEO Executive
Founder
Our founder, Will Hitchmough, worked at a number of high profile Sheffield Digital Agencies before founding Wildcat Digital in 2018. He brings an extensive knowledge of all things related to SEO, PPC and Paid Social, as well as an expert knowledge of digital strategy.
Digital Marketing can be a minefield for many businesses, with many agencies ready to take your money without knowing how to deliver results. I founded Wildcat Digital to deliver digital success to businesses with smaller budgets in a transparent way.
Content Strategist Team Lead
With a degree in Marketing and a background in more traditional, offline marketing, Chloe joined Wildcat in 2021 after deciding to move into the digital marketing industry. She joined us as a Content Specialist and quickly moved up the ranks, becoming a Content Strategist and later an SEO Team Leader.
Outside of work, Chloe is an avid creative. If she’s not knitting, you’ll likely find her behind a sewing machine or in the kitchen trying (and often failing!) to make sourdough.
SEO Account Director
Paul has a strong background in SEO, having previously founded and ran a successful eCommerce business, as well as running a personal blog that achieves an average of 17K users per month. Paul’s knowledge of SEO is extensive, with a strong emphasis on client handling and technical SEO.
Outside of work, Paul enjoys spending time with his family and staying active with weight lifting and combat sports.
Head of Operations
Nadea joined Wildcat in 2021 and has since overhauled the way we work. She has a background in various administrative, operations and HR roles, which gave her the experience and skills needed to lead and organise a growing business.
Outside of work, Nadea loves anything that keeps her active, but she especially enjoys hiking, camping and mountain biking.
SEO Account Manager
With a background in sales, Molly is a natural Account Manager, brilliantly handling any issues that come her way. Having joined us as a Digital Marketing Executive, and working part-time through her final year of University, Molly is a shining example of how hard work pays off. She is now an SEO Account Manager with a particular interest in Content and Client Management.
In her spare time, Molly loves to get out in nature, hiking and exploring the Peak District. She also loves cooking and likes to unwind with a bit of yoga.
Senior PPC Account Manager
Libby joined Wildcat in 2021 as our first PPC hire. With a degree in Digital Media Production, a Master’s in Digital Media Management and previous experience in Social Media Management, Libby hit the ground running and has since climbed the ranks to Senior PPC Account Manager and has a particular interest in the eCommerce sector.
Outside of work, Libby likes gaming, and cooking and likes to keep active by lifting weights.
SEO Team Leader
Jasmine joined Wildcat in 2022 with a strong background in SEO and Account Management. At the time, she was finishing up a Level 4 Apprenticeship in Digital Marketing from the Chartered Institute of Marketing, and has since worked her way up to SEO Team Leader. Jasmine excels at content writing and promotion, and particularly enjoys finding creative ways to join the dots on multi-channel campaigns.
In her spare time, Jasmine volunteers at a charity, helping combat loneliness & social isolation experienced by older neighbours. Outside of Wildcat, she owns a catering company, Savery Grazing, creating delicious grazing tables & platters for a range of events. She also loves skiing and exploring the Peak District.
Senior PPC Executive
Prior to joining Wildcat, Braden founded and ran his own Social Media Management agency where he gained valuable skills in growing brands’ online presence, which has since proven invaluable in his role at Wildcat.
Outside of work, Braden is a big football fan, regularly travelling to Pride Park to support Derby County. He also enjoys watching and playing snooker, hiking in the Peak District, and attending air and car shows.
Senior SEO Executive
After spending ten years managing businesses, restaurants, cafes and event spaces across Sheffield, Jon decided to change careers and joined Wildcat as an SEO Executive in 2022. He especially enjoys the client management side of the job, helping them to understand digital marketing and ways in which they can build their business’s presence online.
Outside of work, Jon likes to keep fit with running, badminton and football, and also loves music.
SEO Executive
Rachel joined us as a graduate, having recently graduated from Sheffield University with a degree in English Literature. Since joining, Rachel has proven herself to be a Content Queen, and particularly enjoys getting stuck into projects that allow her to explore her creative side.
Outside of work, Rachel enjoys a wide range of hobbies, including rugby, singing, reading, and spending time with her family and three labradors.
PPC Team Leader
Carl joined Wildcat in 2023 with a wealth of experience in PPC. He has experience working with a wide range of clients, industries and budgets, from small local businesses all the way to international e-commerce businesses. Carl has a particular interest in strategic planning and forecasting. He is passionate about data analysis and creating sustainable long-term cross-channel strategies.
In his spare time, Carl enjoys keeping fit by lifting weights, bouldering and running. He’s also a keen musician, having played the double bass and bass guitar for 20 years.
Team Lead & Technical SEO Account Manager
With a degree in Computer Science and SEO experience dating back to 2017, Dariusz has a wide range of SEO skills and knowledge. His specialist knowledge of Technical SEO has firmly landed him the title of Wildcat’s Technical Wizard, and he has recently taken on the responsibility of Team Leader for the Panthers Team.
In his spare time, Dariusz loves hiking, experimenting and trying new coffees and loves learning new things. He is currently learning more about CRO and AI and how this could benefit our clients.
SEO Account Manager
With a degree in Film and TV production, and a varied career history, Jamie made the move to marketing with a Masters degree in Digital Media Management. He has since worked in SEO at Agencies across Sheffield, before joining Wildcat and working his way up to SEO Account Manager. Jamie has a particular interest in backlinks and Digital PR and has recently gained a client a valuable backlink from Forbes!
In his spare time, Jamie is an avid foodie and loves trying new restaurants and cuisines. He also loves to travel and spent a year travelling to Australia after university.
SEO Executive
Andy joined Wildcat in 2023 after starting his digital marketing career in-house for a local Sheffield company. Since joining, he has developed a strong interest in Technical SEO and has strong skills in Account Management.
Outside of work, Andy loves music and plays in a couple of bands. He also enjoys rock climbing, cycling, photography and good food.
Client Success Coordinator
Siena joined us in 2023 with a background in sales and digital marketing. She leads on client relationships across the company, ensuring that our customers are happy throughout their journey with us, from their initial consultation through to onboarding and beyond.
Outside of work, Siena enjoys travelling and getting stuck into the local culture. She likes to make the most of her experiences and particularly enjoys watching sunrises and sunsets from beautiful locations around the world.
Senior SEO Executive
Thea has a wealth of experience in SEO, having previously worked for other Digital Marketing Agencies in Sheffield. She has a particular interest and skills in Technical SEO, but is more than willing to get stuck in and give anything a go.
Outside of work, Thea spends most of her time with her children, but also loves reading, photography and gardening.
PPC Executive
Before joining Wildcat, Tom worked across different industries, building skills in sales and customer service. He later developed a passion for digital marketing whilst working on personal marketing projects and freelance ventures, and gained numerous certifications in PPC and Social Media.
Outside of work, Tom enjoys staying active by going to the gym and hiking. He also loves travelling and motorbiking.
Head of Growth
Rich joined us in May 2024 to head up our growth team. With years of experience helping other agencies to grow, Rich joins us at an exciting time as Wildcat is working on a five-year plan to become one of the biggest agencies in the UK.
Outside of work, Rich is a father to three children, which keeps him very busy! He’s also recently started running again to keep fit and loves a bit of DIY.
July 23, 2024
Search intent (also known as user intent) is the goal of the user when they search a query on Google.…
Chloe Robinson
July 22, 2024
Redirects in general can be a confusing topic, knowing when to use one and why is key. There are a…
Thea Chapman
July 12, 2024
Welcome to our July Marketing Roundup! As we reach the height of summer, the marketing world is heating up with…
Jasmine Savery