Keyword Research and Competitive Analysis
Keywords are the foundation for any PPC, SEO or Outreach campaign. The keyword research scene has changed drastically in the last few years as Google’s algorithms change to favor authoritative brands and become infinitely more sophisticated with their capacity to understand context. When working on SEO keyword research, looking at keyword volume and long tail opportunities is by no means enough in the current landscape. Understanding how to perform competitive analysis is a fundamental aspect of keyword research. What’s the point of choosing keywords with high volume, unless your site has the right metrics to compete?
In this article, we’ll walk you through the process of keyword research from the point of view of SEO. We take into consideration not just the process of finding keywords, but also qualifying them based on the relevance and competitiveness. We’ll teach you the different metrics to consider when analyzing keywords, including the best tools to use and a step by step process.
Here are the tools you’re going to need:
- Google Keyword Planner
- Swiss Made Marketing
- Link Research Tools
- Google Drive (Specifically the sheets application) or Excel
When brainstorming for keywords, where should you start? For most industries, unless you already have a list of tried-and-tested keywords from Adwords, your first step is Persona research. You need to understand your industry, and audience, incredibly well and figure out what they might be typing into the search engines to find relevant products or services. These terms will give you your jumping board – a place to start the process of keyword research.
Before you go much further, you’ll need to understand the theoretical construct around keyword research. You can’t simply choose a couple of very broad, untargeted terms, nor can your keyword research just be a list of words pertaining to the EXACT meaning of what you offer. Think in keywords in terms of BUCKETS. Each bucket represents a concept, and within each bucket you can hold a number of specific terms related to that bucket. To illustrate the concept, let’s use the term Weight Training as our Bucket term.
Now we’re going to think of Head Terms that are related to our Bucket Term. Head terms are short, 2-3 word phrases that are contextually relevant to your main term. What’s important to do here is to think laterally instead of vertically. We are not so much as trying to get more in depth with our Head Terms but instead expanding the reach of the Bucket Term.
As you can see, Head Terms aren’t variations on our Head Term, but topics closely related to our Bucket Term.
Next come Long Tail Keywords and Body Keywords. Body Keywords are 3-4 word terms that have high volume and are closely related to your “Head” or “Bucket” terms. Long Tail terms are 4+ word terms that have low volume and are semantically related. The key here is that you may only have a small number of body terms that comprise a large volume of searches each, whereas you may have thousands of long tail terms, each with only 1-2 searches / month, that together create massive volume. Long Tail terms is where you achieve DEPTH. These groupings need to be narrowly focused and as closely related to their head term as can be.
For example a selection of Long Tail and Body Keywords for the Head Term “Calorie Intake” would be:
The difference between Body Terms and Long Tail Terms is how narrow their targeting is. So a term like “calorie calculator” could be classified as a Body Term because there are variations on it that could narrow its focus, like “medical calorie calculator” or “youth calorie calculator”. However, “calorie intake for muscle building” is much narrower in its targeting. These terms are usually comprised of more words and there are very few areas, if any, where the term could become more specific.
What Body Terms lack in focus they make up for in reach and vice versa for Long Tail, so they both should be included in your marketing and SEO efforts.
It is okay for a head term to be a keyword in its own group. For PPC marketers, you can think of the Bucket > Head > Body > Tail structure as the Account > Campaign > Ad Group > Keyword structure.
This concept of broad to narrow groupings and aggregating our keywords by how they are related is crucial to understand before getting on with the practicalities and the tools for keyword research and competitive analysis.
Creating a Comprehensive Keyword List
The first stop that we’re going to make is with Google’s Keyword Planner Tool. This is where we are going to begin the process of compiling our list of potential keywords so we can begin conceptualizing and brainstorming.
We are going to select Search for new keyword and ad group ideas.
Google will let you enter your landing page, a product or service (bucket or head term), or product category. You can input a landing page or product category if you would like, but we’re going to focus on the the keywords. So we’re going to input our bucket term which will be “google penalty removal”, a service provided by Gryffin.
When we click on Get Ideas we get a graph that shows the average monthly searches for the keyword. As well as “Ad group” and “Keyword” suggestions and their competition and search volume.
When judging relevancy you want to make sure that someone who is interested or could be potentially interested in your service might use that term in Google. You should also look at the Avg. monthly searches and competition metrics. When looking at competition data, remember that this is specific to ADWORDS, not SEO. So a keyword may have a very high bid price, but have low competition in Google organic.
We’re going to now start creating our plan.
Under the Ad group ideas you want to find ad groups -head terms- that are related to your service.
Click through to the relevant ad groups. If you see keywords that are relevant and you’d like to add, then click on the arrow next to the relevant keywords.
When you’re done searching for keywords from 1 ad group, then move on to the next ad group. However, instead of adding the Ad groups to your plan, look to the right under your plan:
and add your own ad group with the icon. Then type in the name of the ad group. If you don’t add your own ad group, Google will automatically choose the first ad group that you selected, and you’ll lose the closely related targeting available. As you evaluate keywords, add all of the keyword that seem relevant. Don’t worry about quantity yet, quality is what you’re aiming for when adding keywords to your plan.
Once you have gone through the ad groups and added your keywords. Your plan should look something like this:
Now depending on how broad your service or products is you can have fewer or more ad groups with fewer or more keywords. But the ad groups should all be narrow and specific. Don’t be afraid to add more ad groups, the more specific the better.
The “Ranking Dropped” ad group is a good example of thinking laterally. When someone is hit with a Google penalty, what are they going to experience? A drop in their page’s rankings, so the audience represented by these keywords are looking for google penalty removal services before they even know that they need Google penalty removal services. You’re reaching them at the top of the funnel based on their knowledge and ability to search about your products or services.
After you have finished your plan you are going to export your keywords and ad groups by clicking on the icon.
Here you will be able to export your plan to an CSV file. If you are working with AdWords, you can include a bid in your plan which will give you traffic estimates, and export to an AdWords Editor CSV which will allow for import into AdWords. Here we’re just going to download an excel CSV.
Now if your prefer to work in Excel that is okay. I will be using Google Sheets throughout this piece.
So your data will look like this:
You can remove the Currency, Segmentation, Suggested bid, and Impr. Share columns, they will not be necessary.
And now we have our initial keyword list:
Next we will begin the process of researching these keywords in more depth.
Using SEMrush to Research and Improve Keywords
SEMrush is a very useful tool for finding keywords and examining competition, which is what we’re going to do now. So in your spreadsheet we’re going to add another sheet titled Competitors.
Now we’re going to take our most relevant keywords from our keyword list and put them into Google. In this case I’ll use “google penalty service”
Now I’m going to take the URL’s in green from the high ranking pages and paste them into my competitors list. So it will look something like this:
Now I’m going to take one of my competitors from this list and put their URL into SEMrush.
SEMrush will now give us valuable information about our competitors including traffic, if they have ads running, which countries they’re ranking for, and -what we are looking for- the keywords they are ranking for. For some sites that only rank for a few keywords, SEMrush won’t give that much info because there isn’t that much to report.
This will give us all of the keywords that this website is ranking for according to SEMrush. This keyword list is extremely lucrative as you’re gaining direct insights into the keywords that competitors are targeting.
It is important to look through this list and see if there are keywords that relate to your head terms. You can do this by looking through the entire list, or by adding a filter. If we click on we’ll get this:
So now I’m going to filter the keyword results so I only get keywords that contain “penalty” in them.
So now we see that this site is ranking on three of the keywords that we are going after. So I’m going to hit then choose CSV.
Just like we did with the last report. I’m going to copy and paste the information into Google Sheets.
All of this information is going into a new sheet labeled SEMrush. It’s okay to only have the keywords in the sheet, but if you think the other metrics will be useful then you can go ahead and leave them.
SEMrush also gives you keyword suggestions based on keywords as opposed to a website. So if in the same search bar we input a keyword instead of a website, in this case “google penalty” we’ll get:
Now SEMrush will give us a Phrase match report and a Related keywords report. They will also give us search volume for the keywords. We can export either report if they have a lot of related keywords.
SEMrush will also give us domains that are ranking organically with the keyword and domains that are advertising on with the keyword.
So it might be useful to repeat the process we did with our competitors with the domains in these lists to get more keyword ideas and information about the competition.
Now that we have gotten a few more keyword ideas from SEMrush, my list looks like this now.
Now we’re going to expand and improve this list with a new, particularly powerful, tool.
Swiss Made Marketing
Swiss Made Marketing aka SECockpit is a uniquely powerful tool for not only getting ideas for new keywords, but also information about keywords already in our list.
After logging into the tool, we’ll be greeted by this page:
The process of using this tool will be similar to SEMrush. What we’re going to do is click on this icon to create a new keyword search. I’m going to use “google penalty removal” for this keyword search:
After we hit the tool will start generating a keywords report, this usually takes 1-3 minutes. Once the report is finished, it will look like this.
Before we start gathering the keywords, we’re going to want to make a container to keep them all in. We can do that by adding a new folder by clicking this icon. This will create a folder where we can drop all of our keywords in. I’m going to name my folder after the keyword that I used to generate the list, “google penalty removal”
You can choose a unique icon for the folder if you want. Once you’ve named your folder you can hit to create your folder. Now you should see your folder in the left column of the page.
Now that we have our container for the keywords, we can start dropping some in there. If we take a look back at our generated keyword list, we should arrange by monthly searches descending. This ensures that we get the most related and the most searched for terms first.
Now you can go through the list and simply drag and drop your potential keywords into your newly created folder.
Once you have completed your dragging and dropping, your folder should now be filled with new keywords.
You can repeat this process with new keyword searches, and you can add new folders if need be. Once you are done adding keywords to your folder, we’re going to export our list by clicking and choosing your prefered format. Once again, this information is going to go into its own sheet labeled “Swiss Made Marketing”, or “SMM” for short. You can remove any columns that won’t assist you.
The next tool we are going to use is SpyFu.
SpyFu works really well at giving you competitive keywords and allows you to search for both keywords based on a head term or website. We’re first going to click on .
This allows us to search for keywords based on one keyword (or several) and give us more information about traffic and cost if we’re going to be advertising on these keywords. I used “google penalty removal” for my smart search and found this:
Now we’re going to want to sort by Global Daily Searches by clicking on . Make sure that the numbers are descending. Now we can look through this list and check mark the boxes next to new potential keywords.
Once you have selected all your keywords, you can click on to export your keyword list to excel. Make sure to choose Export Selected to CSV so you don’t export the entire list SpyFu gave us, only the ones we selected.
Create a new sheet labeled “SpyFu in your spreadsheet application and place your keywords inside.
You can also take a name from your competitor’s list and search for keywords on SpyFu that way. Simply take your competitor’s site and put it into the search bar.
As you can see, we get a similar list of keywords and we can follow the same process of selecting and exporting to add these keywords to our research.
You can do this with as many keywords and competitors you like, it will be easy enough to remove duplicates later.
Creating an “All Terms List”
Now that we have sheets of possible keywords from all of our different tools, we can consolidate them into one master list of keywords. We’re going to create a new sheet labeled “All Terms”
Next we’re going to into our other keyword lists and add a new column labeled “Status” here we’re going to plug in a simple “Yes” or “No” as we go through our keyword lists one last time and exclude any that are not related, or have too poor of metrics for your objective.
Don’t worry about catching duplicates here just yet. We’ll take care of this later. Once you have filled out the “Status” column you are going to copy the “Yes” keywords and move them into your “All Terms” sheet; just the keyword will suffice.
Tip: To make your copy and pasting simpler, you can right click on the “Status” column once its filled and choose “Sort Sheet A → Z” Make sure that the heading row is frozen before doing this though.
So your keyword lists should look like this after you have judged their statuses:
and your “All Terms” sheet should look like this:
Now we can take care of duplicates by right clicking on the first column and sorting by “A → Z”. This way any duplicates will be shown right next to each other and we can easily remove one of them, or more if there are multiple copies of the same keyword. Google Drive has a free add-on tool called “Remove Duplicates” that you can use.
Once we have removed duplicates, we are left with a clean list of potential keywords that we can do competitive research on.
Now that we have our keyword list, it’s time to research how much competition these keywords have. We’re going to do this with Swiss Made Marketing and Link Research Tools.
Before we cover how to use these tools, let’s take a look at the important metrics to consider when looking at competitors’ metrics:
- Power Trust – LRT’s version of Pagerank
- Page Rank – Google’s value on link power. This is a deprecated metric and will soon become obsolete.
- MozRank – Moz’s version of page rank
- Page Authority – Moz’s version of the authority of the page.
- Domain Authority – Moz’s version of the authority of the domain.
- Juice Backlinks – The number of unique links going to the page
- Linked Domains – The number of links to the domain
- Total Backlinks – The total number of backlinks to the page
- Internal Site Links – The number links in the domain that point to pages in the same domain
- BLdom: Backlinks to the domain
- DomPop: Referring domains from unique C classes
- Link Velocity (LV) – How quickly the site is building links (Data is aggregated by past months)
- LVT: Link Velocity Trends
- SWR: Sitewide Ratio for links
- DLR (Deeplinks) – Ratio of how many links go to internal pages as opposed to the homepage.
- DomCreated – When the domain was created
- ACrank – Majestic rank (similar to MozRank)
- Glidx – Number of pages indexed in Google
- TitleRank-home – the position of the linking homepage’s title in Google’s search results
- Alexa – Amount of traffic to the site.
- DomPopEdu: Links from .edu domains
- DomPopGov: Links from .gov domains
These metrics are important in establishing whether or not you will be able to compete with other domains for the keywords you’re targeting.
Using Swiss Made Marketing For Competitive Analysis
Swiss Made Marketing has some great tools for competitive analysis. Now we’re going to click on this icon to generate information about our keywords. When prompted with this:
Just simply copy and paste our keyword masterlist inside and click on ,
It will take the tool a few moments to get the information, but once it does we’ll see this:
Now we can export this with all the metrics we need, but before we do that let’s click on one of the keywords:
Here we’re going to get a handy analysis summary of our keyword. This displays the top ten ranking pages in Google for the keyword. We can observe their mozRank, Page Authority, Domain Authority, Juice Backlinks, Linked Domains etc. for the pages.
Red means that that page has a high ranking for the metric, and therefore will be extremely difficult to compete with.
Orange means that it’s possible to overtake the page, but it will be difficult.
Green means that overtaking the page in this area is possible.
You’re going to want to observe this page to get a better understanding of your competition for your keywords.
If we go back to our keyword list, we can export it just like we did our keywords originally.
You can navigate between windows via the tabs at the top of the page.
Now that we are back to looking at our keywords
Here we’re going to examine the metrics above for our keywords and determine which ones have available “spots.” Spots meaning pages in Google’s top ranks where you have a shot of overtaking due to your better metrics.
We’re going to now create a new sheet labeled “Target terms” and in there we are going to put the keywords with potential spots for our page.
I also included an “Observations” column that has information on how to rank in these positions.
For reference, here are all the metrics that are provided by Swiss Made Marketing and what they refer to.
- Words: Number of words in your keyword
- Monthly Searches: Number of times the keyword is searched for in a month.
- Rank 1-10: How difficult it will be to rank 1-10
- Visitors 1-3: The number of visitors you can expect if you rank 1-3
- Rank 1-3: How difficult it will be to rank in the 1-3 positions
- Visitors 4-7: How many visitors you can expect in the 4-7 positions
- Rank 8-10: How difficult it will be to rank in the 8-10 positions
- Visitors 8-10: How many visitors you can expect in the 8-10 positions.
- MR Avg: The average Moz rank for the pages ranking for the keyword.
- MR Max: The page Moz rank of the page with the highest page rank.
- PA Avg: The average page authority of the pages ranking for the keyword
- PA Max: The page authority of the page with the highest page authority
- DA Avg: The average domain authority of the pages ranking for the keyword
- DA Max: The domain authority of the page with highest domain authority.
- JL Avg: The average number of links from unique referring domains on the pages ranking for the key word.
- JL Max: The number of links from unique referring domains from the page with the most juice links.
- Domains: available domains that can be used for the exact match for the keyword. If domain is (n-e-t for the keyword Gryffin Media, then the available domain is gryffin-media.net)
- CPC: Estimated Cost Per Click
- Competition: The level of competition for the keyword
Using Link Research Tools for Competitive Analysis
Link Research Tools is a very useful website for competitive analysis with keywords. Once you Sign in you’re going to want to click on Competitive Keyword Analyzer.
Just like in Swiss Made Marketing, we’re going to paste our keywords from our master list in here. The only problem is that Link Research Tools only allows you to paste 10 keywords at a time. Which only means you’re going to have to run multiple reports if your keyword list exceeds 10.
In our report, we’ll be able to see a heatmap based on our keywords.
But for more specific information, you’re going to want to scroll down under Details.
Here we will find the top ten ranking pages for Google for the keyword. We’re going to export this list by clicking on . Once again, you’re going to take the information, copy and paste into our own spreadsheet under a new sheet labeled “LRT Top Keywords”
Now we’re going to follow the same process that we did with the Swiss Made Marketing Sheet and pick out pages that we can do better than and put their keyword into our “Target Terms” sheet.
The Nuts and Bolts of Competitor Analysis
All the work you’ve done to this point is in preparation of having the necessary data in front of you for analysis.
Now that you have a list of keywords with the top sites ranking, you need to delve into all of the statistics to see which keyword has what we call “Open Spots.” Generally, every keyword will have at least 3 “immovable” spots – these are sites like wikipedia and news publications that no amount of optimization will dislodge.
Your work will entail how many other “Open Spots” each keyword has, and whether the metrics of the pages ranking for those spots are attainable. Of course you’ll need to have run all of the metrics for your own site and pages so you can use those as a point of comparison.
Then you can look at as much data as you can to make your choices.
To start, you can immediately exclude pages with brand names or big authority, like Wikipedia, Best Buy, Nokia, etc.. These pages aren’t going anywhere so long as the brand is popular.
Next look at DomPop – the number of unique links to that domain. If your site has 10 DomPop and the average of the competitor sites is 1000, it will take you a LONG time to have enough authority to compete for this term.
Next, look at other link metrics like PageAuthority, DomainAuthority, and LVT. You want to look for “Outliers” – pages that have significantly smaller metrics than others. For example, if the average Domain Authority is 50 but one of the pages ranking in the top 10 has a Domain Authority of 20, then this presents an opportunity. If you then find that the LVT for that site is a very high, positive number you’ll know the reason they’re ranking well with fewer links is because they’re rapidly gaining links.
Don’t forget to look at other factors such as links from .gov and .edu domains, as sometimes these can help give a site higher Trust. In addition, the average age of the domains can help you understand whether your brand new domain can break in.
Other factors such as pages indexed in Google and social shares can also help you understand the whole story of each page ranking in the SERP’s for the keyword you’re analyzing.
You can also go to Swiss Made Marketing’s and Link Research Tools’ heat map summaries as you go through these lists to help you quickly/easily pick out opportunities.
Once this process is finished, your “Target Terms” sheet should look like this:
Now here is our final keyword list. Now we need to simply organize the list appropriately.
Competitive Analysis For AdWords
This step is only required if your keywords are going to be used for an AdWords campaign. Here we’re going to go back into Google Keyword Research Planner, but this time we’re going to click on this:
Then we’re going to paste our keywords into the box and hit
Here you’ll be able to set up the bid you are planning to use with these keywords. Then you will get a detailed estimate of how much traffic you are going to get and the Avg Pos. of your ads. The Avg Pos. metric will be an indicator of how competitive the keyword is. The lower the position, the more competition there will be based on your bid.
This information is useful and can save you time and money when setting up your AdWords campaigns.
Creating a Mind Map
The best way to organize these keywords is via mind map or word cloud. We will be using Mindmeister.
We start out our map with the brand or company that’s providing the service. Now we’re going to start thinking about the concept again, Head > Body > Long Tail, structure. So the branch we just created is for “Google Penalty.”
Now I’m going to go into my keyword list and pull out all the different topics we find in our key word. The Body Terms or Ad Groups. After going through my keyword list my map looks like this:
These are broad topics but specific topics. This grouping might change as you expand your keywords. You might find that some groups will need to be split, and that’s normal. Now that we have our groups. We’re going to go into our keyword list and put our keywords into these groups accordingly.
As you can see, all the keywords are included based on whether they share words with the group. Some keywords match the group name identically but that’s completely acceptable. With a map like this it will be very easy to conceptualize and edit your keyword lists.
Keywords are the foundation for any SEO improvement and PPC campaign, unfortunately that knowledge of that fact isn’t a secret so it is important to pair your keyword research with competitive analysis.
As with most things, the best way to learn is to get your hands in and do it yourself, so give it a try and let us know if you need help!