Here we will discuss some of the core concepts useful for any SEO implementation.
The discussion will consist of the following topics:
- Where does SEO fit in your digital strategy? You may use paid ads, emails, or social media. We need to ask ourselves where does SEO fit, how do we use it, and where it is relevant.
- Search Intent and SERPs: Well, the next thing that we will be discussing here is search intent and search engine result pages (SERPs). We will learn about what we can see in search engine results, and it will be quite an interesting discussion. Hopefully, you will find it valuable!
- Indexing and Crawling: Next, we will go in a little bit more depth with technical things like crawling and indexing where we’ll discuss how it is that items show up in the search results and what we can do to diagnose if we are not showing up in search results and very fundamental concepts there.
- Algorithms Overview: The next important thing in this discussion will be the algorithms. What are the main algorithms, what’s the evolution been in algorithms, and where are we now? That can help us understand some of the most current optimization efforts that are up there!
Where does SEO fit in your digital strategy?
There are two main parts of this concept, which we will thoroughly discuss here in this section.
- SEO Methodology: It is going to be an overview of generally how to organize search engine optimization projects.
- Mapping SEO to the Sales Funnel: Here, we’ll map the Search Engine Optimization to the sales funnel.
SEO methodology is a process that we use when implementing SEO generically for any type of website in any niche. It consists of the following five steps generally:
Usually, you are going, to begin with, the research phase, where if you are working with a client or internally, you are going to define goals.
That is what we are going to accomplish, what pages do we want to see more traffic in, what kind of conversions we want to have.
We also investigate different keywords, based upon your personas or the various portions of the sales funnels that you might target.
There is often some competitive analysis that takes place here. And because SEO, like most digital marketing activities, is a process where you define some of the tests that you want to run.
Let’s say, for example, if we change something on our website, does it improve against the goal that we have defined. So for this, you’ll have to define those goals and then define the tests that are going to prove those goals.
2. Site Audit
The next phase is to do a comprehensive site audit. So in this case, you will be looking at inhibitions to whether or not:
- Is your site indexed?
- Is your site accessible by search engines?
- What is your content like?
- Analysis of the authoritativeness of your website links?
- Your link building process
So, research first and then do the site audit. Oftentimes, these two steps could be done at the same time, but there is a first step and the second step.
Later, once you are done with the research and have audited your website for the different types of issues that it may have.
The next thing is Optimization. Now oftentimes, with people doing search engine optimization, they will start at step three.
They will say, let’s start changing page titles, let’s begin changing meta description, let’s get more links.
And they haven’t done the necessary research and audit steps to make the most out of these optimization phases. So this step includes things like:
- Site optimization
- Content optimization
- Technical optimization
- Local optimization
We do all of these types of optimizations, and some of the tests that you want to run related to these optimizations are all performed in this step.
Once you have done the optimization of your pages, then you can move into the amplification phase. That is about getting backlinks to these pages that we have already optimized. So, this amplification phase consists of:
- Doing backlink analysis
- Identifying different sources for links
- Using social media to get the word out about different pages that you have optimized.
Once you are done with the above steps, then you want to, very importantly, use this fifth step. Which is, measure what you have done, against the test(s) that you have defined and report on it. Like:
Hey! How have we done it? Are the things that are doing are measurable? Have they had the desired impact? So, based on the report, you can define the goals and tests again, and then repeat the process, if required!
Another way to look at this SEO methodology is to say:
- Step one: We are really defining the competitive landscape. Like we are talking to our internal stakeholders and our clients and defining the landscape.
- Step two: We went to do a site audit and identified what our strengths and weaknesses within that competitive landscape are. Where are we strong, where we are weak, what are the things we can change.
- Step three: The last thing is the number of implementation phases. So again, often, people will directly start search engine optimization activity at step three (“Optimize”) when they haven’t really done the appropriate research and proper site audit phases. Therefore, in this case, when you are doing SEO, you realize that you are probably going to have to move things left a little bit and make sure that you are defining goals and doing a lot of research before changing/optimizing stuff on the site. It’s really helpful!
Mapping SEO to the Sales Funnel
Let’s look at the sales funnel and how it relates to the search queries. It is very helpful for search engine optimization because you can learn something about where someone is in the sales funnel, by the types of queries they make.
What is the Sales Funnel?
Let’s begin, however, with a brief introduction to the sales funnel itself. The sales funnel often consist of three phases called:
- Top of the funnel
- Middle of the funnel
- Bottom of the funnel
After going through these three phases, you get a sale! The sale can be anything like buying at your website or your store or whatever and however you are selling, it will be marked as a sale, once it is purchased, after going through these three phases of the sales funnel.
Top of the Funnel
When we think of people on the top of the funnel, the top of funnel relates to people defining their challenge. So, they have a problem, or they have some kind of intent, or they are just looking to define it.
So, for example, you imagine that you are going to a wedding and you are trying to figure out what kind of clothing to wear, based upon the weather. You know, you are looking for ideas, you are doing question-type queries.
Middle of the Funnel
Middle of the funnel is where someone has transitioned into whether comparing different options. So, they have defined their problem.
Now they are looking at different options and refining those options based upon whatever criteria are relevant to them.
It might be price it might be proximity or availability of different types of products.
Bottom of the Funnel
When people transition into the bottom of the funnel. This is where they are looking at different solutions specifically.
So, they have decided, what their decision criteria are, and now they are maybe looking at different brands, different options, or like, do I build that myself or do I have to hire someone to build that for me, so I try to fix my bathroom or do I hire someone to fix it for me, etc.
Those examples are at the very bottom of the funnel kinds of behaviors.
So that is the sales funnel, and this is how sales funnel works generally!
Keywords and the Sales Funnel
So the interesting thing is that we can tell based upon search behaviors, kind of where people are within the sales funnels.
So, if we look at different queries, that is, we look at different search engine result pages, we can actually see maybe where people are!
At the top of the funnel, people are often querying things that either question formats, they tend to have very low technical language; they are not using the correct terms.
There are kinds of queries that people are using where they are trying to learn about the problem and are not going to use the correct technical language.
There tends to be lower competition at the top of the funnel queries. You are not going to see a lot of ads because it is harder to target those people. Because it is such a broad type of query.
In the middle of the funnel, types of queries that occur, become more like comparisons.
So, this option vs. that option or compare these two options, that’s very middle of the funnel type queries.
If you search for some of the queries like that, you will see the middle of the funnel kind of results.
You’ll start noticing that “Ads” are showing up in the search engine result pages, in the middle of the funnel queries.
In the bottom of the funnel search queries tend to be very branded, there tends to be a lot of ads showing up and much higher competition apparently, because it’s so close to the purchase/sales.
So, when things are close to the purchase, it tends to have more competition, as more people are going to buy based on those bottom of the funnel search queries.
Let’s illustrate it with an example of widening the doorway: Let’s say we have a house, and we want to widen our driveway, and before we begin widening our driveway, we were not sure that we need any special permits for that, so just imagine this use case.
Now, here I’m going down the sales funnel for this, “I need to build a driveway.”
At the top of the funnel, my questions are like; I don’t know what my problem is? What if my driveway crosses someone’s land, is that a problem? Do I need any special permits for that? What should I do about that?
In the middle of the funnel, I start using the technical language; I start using the correct terminology like oh, you know I either need the right of way from my neighbor or maybe to need to talk about getting an easement.
You are not going to see such terminologies at the top of the funnel because it’s considered a more technical language, alright!
But in the middle of the funnel, now I know the terminologies, and how it works, therefore, this is where I am going to decide that either I’ll create this myself or I should hire someone to do it for me.
For this, when I searched it in search engines (while I’m in the middle of the funnel phase), I saw an ad or a search result saying something like “Asphalt gravel and driveway contact” (it’s just an example).
At the top of the funnel, now I can see some of the brands appearing in search results and the top ads.
I am going to search specifically for (e.g., Asphalt gravel) (a branded search query) to know more about them and take my final decision about purchasing there (or anyone else’s) service!
How is the Sales Funnel relevant to SEO?
We might ask ourselves, how is this relevant to search engine optimization? Why are we talking about the sales funnel?
Well, the reason is that almost everyone starts their purchase process using search queries. 89% of purchases start with a search query.
That is, they are Googling or Binging stuff and trying to learn about the different solutions that they are seeking, prior to ever making contact with the brand.
So, by the time someone actually reaches out to a brand or has a brand experience, they have already traveled down that sales funnel quite a ways, we need to take that into account!
SEO and Sales Funnel explained with an Example
I will explain this based on the example that I have explained in an example of widening the doorway.
So, we are thinking about where SEO works?
Well, SEO plays a decent role in placing your brand higher up the funnel, so we could say that we are going to attract people to our website with these question-based and comparison-based queries.
Maybe, we can give them compelling offers, or you can get their email address and then move them or nurture them down to the bottom of the funnel, as now they have brand awareness.
Therefore, SEO plays an essential role in attracting and developing brand awareness higher up the funnel (top & middle of the funnel) and then bringing them down to the bottom of the funnel, where ads become more competitive, and purchase happens!
Search Intent and SERPs
In this section, we are going to talk about Search Intent and Search Engine Result Pages. Here is the outline of this section’s topics:
- What is a SERP? You are going to see this term commonly being used in blogs and in discussions about SEO.
- What is Search Intent? We will talk about what is the meaning of search intent and how it works.
- Three types of Search Intent: Search intent is an exciting way of assessing what we see within the search engine result pages, and we’ll discuss the top three types of search intent.
- Branded search intent: We’ll discuss briefly branded search intent. It’s not the typical three types of search intent, as mentioned above.
What is a SERP?
What is the meaning of SERP? In SEO, this term is used quite a bit; SERP refers to Search Engine Results Page.
When I run a query in a search engine, the page that shows up with the results is called SERP.
Here is the example of SERP for the query “SEO”: I’ve queried the word “SEO” in Google, and it gave me this result page, the SERP!
The design of these SERPs varies from search engine to search engine. Just like Google has different designs, Bing has different, and so on.
What is Search Intent?
What is meant by search intent? Search intent is the way or the intent that a person has when he runs the query.
Like what is that they want to learn or see or do by completing that query.
Just like when a person searches for “learn SEO,” and the other person searches for “hire SEO,” both have different search intent as one is looking to learn about how SEO works.
Whereas, the other person is willing to hire an SEO guy to work for him.
Three Types of Search Intent
Let’s talk about the three types of search intent and how you can drive these by looking at a search engine result page or SERP.
There are just three typical flavors of search intent, though sometimes if you read an article, that breaks that into five or six types of search intent.
But, oftentimes, there are three main search intent types considered in SEO. Which are discussed as below:
1. Informational Search Intent
The first one and the most common one is called Informational Search Intent.
Let’s look at a search engine result page that shows informational search intent.
Here in the above example (see picture), we have run a query about corgi breeds. It is a very excellent example of informational search intent.
In an informational search intent query; what we see is just information about a particular topic.
As we have searched about corgi breeds, the SERP is showing us the informational results about the facts, pictures, and a Wikipedia article about this breed.
There is no link that says about buying the Corgi or any comparisons, in this SERP, it’s all just the information about Corgi at this result page.
Generally, 50% to 80% of search queries are informational.
You must be thinking that where informational search intent queries show up in the sales funnel.
They are primarily associated with the top of the funnel search queries.
A lot of people are conducting informational type search queries, and it’s the most common type of search. It tends to be related to the beginning of the journey that people make towards the purchases.
2. Transactional Search Intent
The next type of search intent or the second type is called Transactional Search Intent. Let’s take a look at transactional search intent with an example.
Here in this example (see picture), I have queried the word “dishwasher,” not saying that I want to buy a “dishwasher,” not saying what the prices of “dishwasher are.”
Just the word dishwasher, and we can say something about the results here. Compare this to the informational search intent query results.
Here we see shopping results (on the right side, in the picture). We can see very heavy ads placement at the top of the results.
And again, if we cover up the SERP features placements, all the results are branded results and not organic results in this transactional search intent query result page.
(Note: In the informational search intent query result page, all SERP features placements were showing only the organic results)
So, this is the transactional search query result page, with lots of ads on it. This search intent type is generally related to the middle of the funnel and the bottom of the funnel.
As it can be like, I’m comparing dishwashers, I’m looking at two different types, or maybe I’m ready to make that purchase.
So, transactional search intent starts moving us down towards the purchase.
The last most common search intent, the third type is Navigational Search Intent. Let’s have a look at search engine results page with the help of an example of navigational search intent:
Here, I have searched for “climbing gym,” and as you can see in the picture, the resulting page is completely different from the other two types of search intent.
At the top of SERP, I am seeing the local pack, that is the “map results.” It shows me a map and three top results related to my navigational search intent query.
The SERP features placement is covering up with all non-organic results, that is the map and all. As the search engine thinks (based on my query) that I may want to visit a physical location “gym,” a particular business that has a physical location; therefore, the search engine displays these map results.
Now, because navigational search intent has a high correlation with visiting a physical location after it has been completed, it is considered the very bottom of the funnel. Imagine that you are searching for a sandwich shop near you or looking for coffee, the likelihood is very much that the search engine is going to interpret that you want a sandwich or you want a coffee right now.
So, that means it’s very close to the purchase, and therefore, it falls at the bottom of the funnel.
Branded Search Intent
When developing an SEO strategy, it’s important to understand the concept of branded search intent.
This type of search intent often comes into play when you see a business doing keyword research or analyzing their keyword rank positions.
Because often, businesses will be tracking keywords that have their name included in it.
So, if you say banana republic and you track the keyword, let’s say, women’s sundresses banana republic or men’s dress shirt banana republic, and those are the keywords you are tracking.
It’s important to make the distinction that those are branded search queries. Let’s take a look at what a branded search query or branded search intent looks like.
Here we have an example of this (see the above picture). So, let’s run the query for “climbing harness.”
If you are a rock climber, you have a climbing harness, and here is the query that shows that.
We can tell by looking at the search results, without knowing anything else that what we have here is transactional search intent.
As we see, lots of ads and lots of shopping results in SERP feature placement areas because the search engine is interpreting as we want to purchase something.
We also see category search results in organic results as well.
However, if we add in this branded “Petzl” term in this query (see the above picture) [Petzl is a manufacturer of climbing harnesses], look at what happens to those results.
We still have transactional search intent taking place here, as we see a lot of ads and shopping results.
But look at those organic results, the top organic results are now Petzl.
So, when you include the name of the company in the search query, you are going to see that business showing up in search results, as you are telling the search engine that you want search results something narrow to that brand.
So, when you are tracking some keywords overtime for a client and when you talk to that business owner, the owner says that we rank very high for XYZ search queries.
Well, then if you look in their account information or their tracking results in whatever tool they are using to monitor their keywords, you may see that they are ranking for those terms + their brand name.
That is a branded search intent, and in this type of search intent, you’ll find that the business rank typically very high, which is different then if you remove the brand name from the search query and see for how they rank for the non-branded term.
It’s a fundamental distinction, and it’s a widespread mistake that businesses make!
Common SERP attributes
A good SEO is intellectually curious about what shows up in search engine result pages beyond just what the results are but also what are the additional features and attributes that show up in a SERP.
Let’s take a look at an example, and we’ll walk through some of the common elements that we see in search engine result pages as well as what they mean.
So here we have a search engine result page (see above picture) for a query that says, “how to remove a sticker from a car window.”
Let’s go and walk through the things that we see on this results page. So, first, at the top of every search engine result page is a count of pages that meet the criteria that we have used.
This number (count) is beneficial when you use search modifiers.
Therefore, you can adjust the search query to say, “only show me results on a particular site,” and it will show you roughly the count of pages that it has within the site that you have designated.
Let’s go and jump on the next thing on the page; this is called “Featured Snippets” (see picture). Featured snippets come in three different forms:
- 2-3 sentences
Featured snippets generate from among the first ten organic results.
Therefore, the search engine empowers to answer the questions in particular with featured snippets, and it will go to any of the first ten organic results and find what it thinks is the most accurate answer for that query.
Featured snippets do not always show up, they can change in a short time, so if you search query now, and then you search in 5 minutes again, you may find different content in featured snippets.
Down below that, often associated with featured snippets, are these things called, “People also ask” boxes.
The ‘People also ask’ box is itself made of other featured snippets.
You can see in the above picture; there are other associated questions with that query in the “People also ask” box.
These are all other ways of saying the same question, or they are related questions.
So, if you are creating content, it could be helpful to look at the “people also ask” box and get a sense of maybe other ways of people saying it or their ideas that are similar.
If you open any of the “people also ask” box, by clicking on the down arrow at the left end, you will find that these are all featured snippets themselves.
So, the search engine is showing you one featured snippet at the top, and then it’s showing you other featured snippets that it feels confident that it has an answer in it.
The next thing in this example is “Organic Results” (see above picture). Everything above this portion of the page could be considered SERP Features.
And here we have the organic results which have been ranked by the prevailing search engine algorithm.
“The primary goal of the search engine is to deliver value to the user, based on a search query.”
Therefore, you are going to see a lot of different changes in the search engine result page as the search engine tries to give you the most value.
Let’s take a look at this example here (see above picture). We can assume that what the search engine is implying is that it says like:
Look when you ask ‘how to remove a sticker from a car window’’ people probably click on videos a lot, therefore, after enough people click on videos, I’m just going to show you videos in these search results.
That is the reason the top organic result in this example is a YouTube video result.
Here is a different pattern (see above picture). Moving away from car stickers, we are looking at “flower decoration ideas.”
What’s the very first thing that shows up on this page is an image pack. There are a bunch of different images in it.
Again, we can assume here that the search engine is saying in this way, “I guess that people want to see images of this query; therefore, I show images.”
Search engines are trying to deliver the most value possible within the few clicks by the user.
Now the exciting part! Let’s again take a look at this result page and also at the previous examples result page.
When an image pack shows up higher in the results page, note that the images on the navigation tab show up first.
Just like that, see the previous example where the videos pack is showing up, you will notice that the first thing in the navigation tab that shows up is videos.
So, you can see how much comfort these search engines provide to their users.
A good SEO always pays attention to minor changes in the SERP and asks:
Good SEO will always think about every minor change in the search engine results page and ask himself why is that happening? How is it being formed? Where is the information coming from?
Because this helps in understanding how you can reach the SERP features placements.
Search Engine Indexing and Crawling
In this section, we are going to look at this essential concept of crawling and indexing. This section consists of a few different parts. The main parts are:
- How do pages show up in search results? We will look at different pages and check about what the search engine result page looks like. How do pages show up in search results? Can all pages show up or there have to be some attributes for showing them in specific parts of search engine results? What are the different types of search intent? Answering those kinds of questions.
- What is a cached page? We’ll dig into what are cached pages and how to deal with them. Therefore, for those pages, which are part of the search engine index, what can we know about them? We will take a look at the concept of cached pages.
- How to control the crawlers and Robots.txt? We’ll see how we control what the search engine crawlers see on our website. Are there any relevant things that we can do to either block the crawlers or there might be inadvertently blocking the crawler, for pages that are important to us. Therefore, we can say that we’ll discuss here how we can control the crawlers, by looking into this concept called Robots.txt.
How do pages show up in search results?
Let’s answer this question: How do pages show up in search results? That is, why do some pages show up in search results and others don’t.
It is good to start with this concept when discussing this topic of crawling and indexing. Look, the search engine’s job is to organize the world’s information and make it universally accessible and useful.
Now, the current way that people access the search engine databases is through typing words or saying words (voice search).
Therefore, What the search engine is trying to do is trying to make information accessible through the words that people use to describe their intent.
So, if we think about the process, by which they do that, we could say that the search engines have these kinds of two functions:
- Crawling and Indexing: They are trying to find the value on web pages, oftentimes in content, videos, images, or text on webpages. They are trying to find the relevant information and then they need to deliver it to people in some way that matches their search intent, with the words there have been used. Therefore, we can break this into a little bit more proper search engine optimization terminology we would say, the finding part is called Crawling and Indexing.
- Ranking: Search engines are going out and trying to find content and then the delivery of it is then ranking it and displaying it in a search engine result page (SERP).
Let’s focus our efforts here on the crawling and indexing process.
The often-used analogy of crawling and indexing is looking at it like a subway system or roadmap.
In the above picture, we can imagine that as a subway-system or train-network. Consider it like all of the stations in the network represent pages (webpages). And between these different pages are links. So, we consider the train tracks as links in that analogy.
Do not worry, if you are still confused about this example, just continue reading and you’ll love the way it is being explained!
The terms crawling and spidering are used synonymously for most of this discussion.
Well, what the search engine does (the way that it crawls sites), is that the search engine starts to crawl some known valuable site, Wikipedia for example, which has all the links to other sites.
The search engine will read all the Wikipedia pages and wherever it links out to other pages, it will then go through those links. It will crawl through those links to reach those other-pages and read them. And search engines won’t stop there, but continue to even read those-pages to find any further links, then crawl to the more others, etc so on and so forth.
This is the process of crawling with the search engines, looking to find all of the valuable pages across the internet and save them somewhere.
What these search engine crawlers or spiders are doing is they are storing them in their database in an easy to retrieve format.
So, they want to be able to say that when I search for corgi puppies, then the search engine is going to be able to correctly identify relevant pages that are about corgis and the corgi puppies. And then depending upon what search intent I display, search engines rapidly find the pages that meet my given criteria.
The way these search engines store this data in their data storage centers, is what we would call The Index. So, it has crawled the pages and then indexes them in the storage center.
The important thing to think about here is that if your pages have not been crawled and have not been indexed by the search engine, then they are not going to show up in the SERPs.
This can be one of the major issues facing a site. Why am I not showing up in ranks? Well, the first question here should be, are my pages even in the index? Are they even able to be displayed? (Regardless of the quality of the content on that page)
Therefore, the crawling and things which are associated with crawling and indexing can be a major problem for a site, if it is not properly indexed!
What is a cached page?
Let’s talk about cached pages and how they relate to indexation. As well as some of the things we can drive from search operators.
So, when a page or site is crawled by a search engine, the page attributes are recorded in the index.
Here we have an example of a page (see picture) when the search engine crawls this page, it is going to record things like:
- Page title
- Follow through internal and external links
- H tags (headings)
- Load speed
But it is also going to look at a ton of other attributes associated with that page including things like:
- How fast it loads
- Some sense of whether or not it has security certification
- Does it have any social buttons [social signals]
- How is it to read the page
- Text size
- Inbound links
This is often said to be 200 or more ranking signals that are collected by the search engines for indexing and ranking a page. All of that collected data is stored in a storage center, which we call The Search Engine Index.
Basic Google Search operators
Now, how and what can we know about the indexed pages that the search engine has about our or customer’s site?
We can get a sense of what pages are in the index for a site by using the search operators.
How to check if a website is indexed in a search engine using a search operator?
Site:yourdomain.com (make sure not to add space(s) before or after the ‘:’ colon)
What you get is you get all of the results that the search engine has for that site (see picture) or at least close to its directional nature, it’s not going to be 100% accurate but it’s going to be near to that number.
So, here we can see we went to the google.com site by typing site:google.com in search engine. We have now instructed the search engine to just show us pages that are on the given site.
You can see “About 538,000,000 results” or something nearly like that (based on your geo-location) that represents how many pages are approximately indexed in the search engine which meets the given search criteria.
These indexed numbers of pages can be more or less than the actual number of pages published on a given website and those would be the things that we want to understand, why is that the case?
How to check the cached version of a page using a search operator?
These pages that search engines have in the index are referred to as Cached Pages.
Let’s look at an example of what is a cache page? We can see any given page for its cached version. imagine that you have a blog or a page on a website, in this case, it’s a Wikipedia page (see picture).
Consider that now the page has been changed, for example, you may have updated the picture or you have updated the text on the given page (or any page of your own website/blog/etc).
Just because you update the page on your site doesn’t mean that the search engine has also updated that same page in its index.
If the SE has not crawled the site recently, it will not display the information associated with that newly updated page. Therefore, we want to see what is the current cached version of my page that SE has in its index versus the one that is maybe live on your website/blog, etc.
How do I see the cached version of my page in a search engine?
Well, we can use this example here with the Wikipedia page. Let’s go and take the URL of this Wikipedia page and then add the search operator “cache:” with the link:
When I type the above URL with the cache search operator in the search box, what that delivers me is this version (see picture), you can see there is a little text-area across the top of the page now, that says, “This is Google’s cache of…”
You can also see these three items underneath it:
- Full version
- Text-only version
- View source
So in this case, full-version is bolder which means we are viewing the full version of this page, although we can look at the text-only version or view source by clicking on the respective link text.
The important thing to think about with these three options is that when a search engine crawls your page it does not render it in a browser, not on a mobile (on a mobile this might look different) however on a desktop it is not rendering it in a browser but the search engine crawler just reads the source code of the page. So, as we have to see what is the version of the page that the search engine sees when it crawls the page.
We should click at this “view source” text (see picture) and what this is going to show us is the code of the page (also known as source code of the page).
This is going to show us the code of the page as you can see in the picture. It is showing us how the search engine will look and index our page in their database.
How to control the crawlers and Robots.txt?
It is important to understand how robots.txt works when trying to diagnose any crawl issues and also when using it to control how crawlers should see your sites.
Let’s ask ourselves a question: when a site is indexed by a search engine is there any way that we can provide instructions to the SE about which pages to crawl and what not to crawl.
Let’s say there are sections of sites that don’t have a lot of value or we have the alternative pages that we obviously want to show up in the search engine results. How can we do that?
Controlling web crawlers with Robots.txt
We can control that with robots.txt. Robots.txt file contains instructions for web crawlers. Because, when a web crawler hits a site the first thing they are going to read is the robots.txt file and the sitemaps simultaneously.
Robots.txt file example – Basic syntax:
Sitemap:yourdomain.com/sitemap.xml or whatever path your website has, for the sitemap.
This is the very basic syntax/text for a robots.txt file. You can simply copy/paste this code in your robots.txt file (for any new website) while making sure that you are going to change/update the correct sitemap.xml path in this code and then upload it to your website’s root location.
(If you are confused about what is your website’s sitemap path, you can ask for help in the comments section of this page.)
For advanced instructions for robots.txt file, continue reading.
Here is how you fill out robots.txt with advanced level control. Here you are instructing a specific crawler that is named as Googlebot-image which crawls the images on any website.
The first line defines that these instructions are for which type of crawler, that is:
Then the next line instructs the crawler/user-agent not to crawl/index anything that is in the “/checkout” folder. The ‘ * ‘ wild card means that this instruction is applicable for anything before (prefix) and after (postfix) the text “/checkout” in the URL. That means, no matter what is written before or after ‘/checkout’ in the URL, the crawl is not allowed to crawl such links.
In the next line we are seeing the same instructions but for another section of the website, which contains the text “/p/” in its URL.
Let’s look at the Nike robots.txt file at (you may observe different settings later in this file if they have done some modification over time). They kind of have to instruct things if you scroll way down on Nike’s robots.txt file.
It is a good example, as we can see quite a few instructions for different web crawlers/user-agents. We are seeing a few permissions and some disallow files/folders. We can also see very specific instructions for Googlebot, Googlebot-image, Baiduspider, etc.
As you can see the Baiduspider user-agent has many of the languages disallowed, other than the Chinese language, which makes sense as Baiduspider is basically for the Chinese search engine.
I hope these examples have given you a better understanding of what robots.txt is and how it works. But if you still have any questions, feel free to ask in the comments section of this page.
Make sure that you have configured your robots.txt file properly. Because, if, by mistake, you disallow any of your main pages/sections of your website in the robots.txt file, you’ll never find it in search engine results and those pages/sections will never rank. That can become a great loss for you, later on.
Search Engine Algorithms Overview
In this section, we will talk about algorithms. I will give you an understanding of some of the common algorithms as well as a brief history of algorithms.
- History of Google search algorithms: How those algorithms came into being. We will discuss some terms you may or may not be familiar with.
- Recent trends in SEO: How have the recent algorithm updates changed the way that search engine results are being displayed.
So let’s go and dig into that.
History of Google search algorithms
Let’s talk about the search engine algorithms that you most likely encounter and have to deal with when updated. There are always adjustments going on behind the scenes with improvements and feature updates with the search engine results displayed, including major algorithm updates.
Here in the picture, you can see the updates till 2013, that shows all the renovations that took place with Google algorithm updates. Let’s discuss the main algorithm updates that you should be familiar with.
Google Caffeine Update 2010
In 2010, there was a major improvement and I would say that a major rewrite in Google search engine algorithm update called Google’s Caffeine Update.
This update aka “Google Caffeine update” was largely based on the freshness of articles. You can imagine news sites that had breaking news and they had new pages on their sites.
How does the newness or the freshness of those articles contend against articles that had been around a long time, they might have a lot more links.
You can imagine that the importance of the value of that page might not surface if you are working on only inbound links. So, it was a big problem and Google caffeine update was developed to solve that.
However, because of this radical reshuffling of these results, Google caffeine update required a series of patches.
Let’s look at some of those patches and updates that came out.
Google Panda Update 2011
In 2011, there was an algorithm update called Google’s Panda Update. Panda update was about identifying the quality of pages.
That includes, for example, the importance of inbound links, and trying to discourage thin content.
How do we identify thin content? How do we make sure that it doesn’t make it into the index? Or it better doesn’t show in the result page, that was the very Google Panda update 2011.
Google Penguin Update 2016
In April 2012 an update came with the name of Google Penguin Update. And this Penguin update is still being updated along with other upcoming Google algorithm updates.
This update was mostly about identifying spam and penalizing individual pages rather than whole websites.
Google Penguin update made sure to identify and reduce spam in search result pages. This update also did something interesting where it could identify spam on individual pages rather than penalize the entire site that made it a little bit more targeted in terms of spam identification and reduce the incidents of spam in search engine result pages.
Google Hummingbird Update
In 2013 Google presented the next major rewrite of the Google algorithm and this was called Google Hummingbird Update. We live now in the world of hummingbird update.
The most important aspect of Google’s hummingbird update is that it is based on understanding entities of queries and the relationship between words. Therefore, it isn’t just about typing in words into search engines and having the search engines try to find pages that have those words on them.
Google Hummingbird update was a big breakthrough as it changed the concept, the way that search engines understand the relationship between those words by, for example, reading pages and seeing how often word-pair shows up or how often words show up in the same kind of order, within the pages.
In this update, when the search engine is displaying the results, it tries to understand the results more like a human mind. It was a very interesting update, happened in 2013!
Google Possum Update 2016
Another pretty significant update was in 2016 and that one was called Google Possum Update.
Possum update was specifically related to local search results, the search results that display local map-packs. What it does is it helps understand and bring in to the ranking factors, the location of the searcher. That confirms, where you are when you are searching.
This radically changed the way that the search results for the local search are displayed.
For example, if I am looking for a sandwich and I say a sandwich shop, and I’m on my mobile phone, and I’m on one side of the town vs another side of the town. The resulting shops that are near me are going to be different from the results which I’ll get while searching from the other side of the town.
Therefore, in that sense, the competition increased among the different businesses within the same city.
For instance, before Google Possum update, I am the best sandwich seller in Chicago and my shop is far from your location from where you run the query, I’ll show up at the 1st position in your search results.
Whereas, after Google’s Possum update, maybe I am still considered the best sandwich seller in the city, but Google will not show my shop on top of the search results if you are not located near my shop. You will see some other shops at the top of your search results page in local map-packs that are well-known in your nearby area.
Therefore, the Possum update has significantly changed the game when it comes to local search results.
So, these were some of the Google algorithm updates that are salient.
Recent trends in SEO
Let’s now discuss recent trends in search engine optimization based upon what we know about the latest algorithms as well as search intents, and how the algorithm is trying to satisfy it.
As we know that in 2013 the algorithm Hummingbird was launched by Google which looks beyond just individual words and rather looks at entities associated with those words.
It looks for the relationship between the concepts and tries to understand the words as the way that of a human’s mind.
Therefore, it is not just identifying the incidence of the words you are using on a given page, but it’s trying to understand what it is actually that you are looking for and provide the result that matches that value as closely as possible.
Hummingbird update’s impact on SEO
In the word, where we are querying a search engine which has such capabilities, we have to ask ourselves what are some of the impacts, and what are some of the behaviors and features that we see that are either enabled or developed on account of this.
Query length has gotten longer
One of the interesting things is that as the search engines have become smarter the way that we interact with them has changed.
One of those ways is that the query length has gotten a lot longer so we start seeing instead of word jumbles or short words.
For example, men’s shoes. We see something more like, what kind of shoes should I wear to a catchup party, or what kinds of shoes are comfortable and formal and waterproof.
In the real world, those are the kinds of things you are going to ask another human rather than necessarily to a search engine. But now, you can ask the search engine and you will get a proper answer.
People are asking more questions from search engines
We also see that it is getting longer in a specific way. As a lot more questions are being asked and you can see this graph (see picture) represents a year over year of growth these trends have persisted for the last few years.
When people are querying things, you see a lot of growth in the ‘how does this work’ or ‘who is this person’ or ‘where can I go for some bit of information’.
People ask questions and they expect answers from search engines in a similar way that we talk to other humans.
The interesting thing is that starting around the same time as hummingbird update, we began seeing the incidence of these featured snippets.
Featured snippets in search results
Featured snippets are hummingbird’s effort to answer those questions.
For example, see in this search result (see picture), that when I type ‘how to open a checking account’, the search engine says there are 10 million+ results. However, we also see there is a featured snippet.
That means, that the search engine feels confident, that among those 10 million+ results, it has one result or one statement from among those search results that it thinks, that it can answer that question correctly.
You can also try this with things like, ‘how tall is the statue of liberty’ or ‘how far away is the moon’.
Things like that have a known value, so when the search engine feels confident that it has a known value or a known answer, it will tempt to answer it within a featured snippet.
People also ask box in search results
We also know that the “People also ask” box represents other featured snippets.
Here, hummingbird update is not just guessing what the correct answer to your question is, but it’s also saying that there are other things that I understand to be related to the question that you have asked.
In the example, the entity of ‘checking account’ exists within a relationship map to things like ‘free checking accounts’, ‘how to open a checking account’, and ‘how do you start your own bank’ well maybe that is a little far off [ 😄 ] but either way, there are some other questions that perceive as being highly related to them.
As we know, featured snippets show up in three different types:
- Short answer (2-3 sentences)
It is really interesting to see the proliferation of featured snippets and what it implies about the search engine’s willingness to answer questions about the things it knows.
Featured snippets are not generated only by the first result listed in SERP.
Some people will look at them and say, “Oh that is just a summary of the first result”. Well, that is not true.
Search engines can take featured snippets from any of the top ten results that appear on the first page of search results.
These results are highly weighted to the first, second, and third top results, but it can be technically taken from any portion of the top ten results.
It isn’t the first paragraph, isn’t the last paragraph, it could be somewhere in the middle of any piece of content on that first page.
SERP Features have a dramatic impact on CTR
Featured snippets have had a pretty significant impact on the click-through rates (CTR) of results.
As you can imagine in a world where we ask questions a lot more, and the search engine is just providing us the answers. Then what compelling reasons do we have to go and visit the page? No reason mostly!
So we can see some of these results here:
- On the left-hand side are mobile device results
- On the right-hand side are desktop devices
- The grey area represents where there is no click through to a page, most probably the visitor has got his answer within the featured snippet, that is why the visitor has not clicked on any results.
- The orange represents the paid clicks on ads – As we can see paid ctr is quite small but that generates billions of dollars for the search engines.
- The blue represents the free clicks on organic results.
Here is why you have to pay attention to featured snippets and what they represent as an opportunity.
As mobile search becomes more and more prominent as it outpaces desktop searches, you can imagine those feature snippets take up more of that screen realistic.
So, if you have a lot of users that are on mobile devices and you have been ranking pretty well, or guess what, if featured snippets start showing up for the keywords you are ranking for.
It’s possible that you won’t even be visible on the screen anymore because the featured snippets have taken a lot of that screen space.
Featured snippets are currently being delivered to many voice queries!
That means, when you ask your google home or your Alexa or personal assistant, ‘how far is it to the moon’ or ‘how tall is the Eiffel tower’ or anything like that, the device is just reading featured snippets and answering your voice query.
So, if you run the same query on desktop or mobile, you’ll probably see the same result show up in a featured snippet as what the personal assistant has said.
Currently, most of these personal assistants are reading these featured snippets when they are providing voice search results.
And you know it makes sense of why they do that because:
- You are asking questions.
- You are looking for a short answer.
- You want some authoritative answer.
- You want answers that are algorithmically generated, to make sure that the answer is correct.
That’s it for the SEO Fundamentals guide. If you have any questions, please feel free to ask in the comments section.