Tracking Your Link Prospecting Using Lists in Link Explorer

Posted by Dr-Pete

I'm a lazy marketer some days — I'll admit it. I don't do a lot of manual link prospecting, because it's a ton of work, outreach, and follow-up. There are plenty of times, though, where I've got a good piece of content (well, at least I hope it's good) and I want to know if it's getting attention from specific sites, whether they're in the search industry or the broader marketing or PR world. Luckily, we've made that question a lot easier to answer in Link Explorer, so today's post is for all of you curious but occasionally lazy marketers. Hop into the tool if you want to follow along:

Open Link Explorer

(1) Track your content the lazy way

When you first visit Link Explorer, you'll see that it defaults to "root domain":

Some days, you don't want to wade through your entire domain, but just want to target a single piece of content. Just enter or paste that URL, and select "exact page" (once you start typing a full path, we'll even auto-select that option for you):

Now I can see just the link data for that page (note: screenshots have been edited for size):

Good news — my Whiteboard Friday already has a decent link profile. That's already a fair amount to sort through, and as the link profile grows, it's only going to get tougher. So, how can I pinpoint just the sites I'm interested in and track those sites over time?

(2) Make a list of link prospects

This is the one part we can't automate for you. Make a list of prospects in whatever tool you please. Here's an imaginary list I created in Excel:

Obviously, this list is on the short side, but let's say I decide to pull a few of the usual suspects from the search marketing world, plus one from the broader marketing world, and a couple of aspirational sites (I'm probably not going to get that New York Times link, but let's dream big).

(3) Create a tracking list in Link Explorer

Obviously, I could individually search for these domains in my full list of inbound links, but even with six prospects, that's going to take some time. So, let's do this the lazy way. Back in Link Explorer, look at the very bottom of the left-hand navigation and you'll see "Link Targeting Lists":

Keep scrolling — I promise it's down there. Click on it, and you'll see something like this:

On the far-right, under the main header, click on "[+] Create new list." You'll get an overlay with a three-step form like the one below. Just give your list a name, provide a target URL (the page you want to track links to), and copy-and-paste in your list of prospects. Here's an example:

Click "Save," and you should immediately get back some data.

Alas, no link from the New York Times. The blue icons show me that the prospects are currently linking to Moz.com, but not to my target page. The green icon shows me that I've already got a head-start — Search Engine Land is apparently linking to this post (thanks, Barry!).

Click on any arrow in the "Notes" column, and you can add a note to that entry, like so:

Don't forget to hit "Save." Congratulations, you've created your first list! Well, I've created your first list for you. Geez, you really are lazy.

(4) Check in to track your progress

Of course, the real magic is that the list just keeps working for you. At any time, you can return to "Link Tracking Lists" on the Link Explorer menu, and now you'll see a master list of all your lists:

Just click on the list name you're interested in, and you can see your latest-and-greatest data. We can't build the links for you, but we can at least make keeping track of them a lot easier.

Bonus video: Now in electrifying Link-o-Vision!

Ok, it's just a regular video, although it does require electricity. If you're too lazy to read (in which case, let's be honest, you probably didn't get this far), I've put this whole workflow into an enchanting collection of words and sounds for you:

I hope you'll put your newfound powers to good. Let us know how you're using Tracking Lists (or how you plan to use them) in the comments, and where you'd like to see us take them next!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

How Much Data Is Missing from Analytics? And Other Analytics Black Holes

Posted by Tom.Capper

If you’ve ever compared two analytics implementations on the same site, or compared your analytics with what your business is reporting in sales, you’ve probably noticed that things don’t always match up. In this post, I’ll explain why data is missing from your web analytics platforms and how large the impact could be. Some of the issues I cover are actually quite easily addressed, and have a decent impact on traffic — there’s never been an easier way to hit your quarterly targets. ;)

I’m going to focus on GA (Google Analytics), as it's the most commonly used provider, but most on-page analytics platforms have the same issues. Platforms that rely on server logs do avoid some issues but are fairly rare, so I won’t cover them in any depth.

Side note: Our test setup (multiple trackers & customized GA)

On Distilled.net, we have a standard Google Analytics property running from an HTML tag in GTM (Google Tag Manager). In addition, for the last two years, I’ve been running three extra concurrent Google Analytics implementations, designed to measure discrepancies between different configurations.

(If you’re just interested in my findings, you can skip this section, but if you want to hear more about the methodology, continue reading. Similarly, don’t worry if you don’t understand some of the detail here — the results are easier to follow.)

Two of these extra implementations — one in Google Tag Manager and one on page — run locally hosted, renamed copies of the Google Analytics JavaScript file (e.g. www.distilled.net/static/js/au3.js, instead of www.google-analytics.com/analytics.js) to make them harder to spot for ad blockers. I also used renamed JavaScript functions (“tcap” and “Buffoon,” rather than the standard “ga”) and renamed trackers (“FredTheUnblockable” and “AlbertTheImmutable”) to avoid having duplicate trackers (which can often cause issues).

This was originally inspired by 2016-era best practice on how to get your Google Analytics setup past ad blockers. I can’t find the original article now, but you can see a very similar one from 2017 here.

Lastly, we have (“DianaTheIndefatigable”), which just has a renamed tracker, but uses the standard code otherwise and is implemented on-page. This is to complete the set of all combinations of modified and unmodified GTM and on-page trackers.

Two of Distilled’s modified on-page trackers, as seen on https://www.distilled.net/

Overall, this table summarizes our setups:

Tracker	Renamed function?	GTM or on-page?	Locally hosted JavaScript file?
Default	No	GTM HTML tag	No
FredTheUnblockable	Yes - “tcap”	GTM HTML tag	Yes
AlbertTheImmutable	Yes - “buffoon”	On page	Yes
DianaTheIndefatigable	No	On page	No

I tested their functionality in various browser/ad-block environments by watching for the pageviews appearing in browser developer tools:

Reason 1: Ad Blockers

Ad blockers, primarily as browser extensions, have been growing in popularity for some time now. Primarily this has been to do with users looking for better performance and UX on ad-laden sites, but in recent years an increased emphasis on privacy has also crept in, hence the possibility of analytics blocking.

Effect of ad blockers

Some ad blockers block web analytics platforms by default, others can be configured to do so. I tested Distilled’s site with Adblock Plus and uBlock Origin, two of the most popular ad-blocking desktop browser addons, but it’s worth noting that ad blockers are increasingly prevalent on smartphones, too.

Here’s how Distilled’s setups fared:

(All numbers shown are from April 2018)

Setup	Vs. Adblock	Vs. Adblock with “EasyPrivacy” enabled	Vs. uBlock Origin
GTM	Pass	Fail	Fail
On page	Pass	Fail	Fail
GTM + renamed script & function	Pass	Fail	Fail
On page + renamed script & function	Pass	Fail	Fail

Seems like those tweaked setups didn’t do much!

Lost data due to ad blockers: ~10%

Ad blocker usage can be in the 15–25% range depending on region, but many of these installs will be default setups of AdBlock Plus, which as we’ve seen above, does not block tracking. Estimates of AdBlock Plus’s market share among ad blockers vary from 50–70%, with more recent reports tending more towards the former. So, if we assume that at most 50% of installed ad blockers block analytics, that leaves your exposure at around 10%.

Reason 2: Browser “do not track”

This is another privacy motivated feature, this time of browsers themselves. You can enable it in the settings of most current browsers. It’s not compulsory for sites or platforms to obey the “do not track” request, but Firefox offers a stronger feature under the same set of options, which I decided to test as well.

Effect of “do not track”

Most browsers now offer the option to send a “Do not track” message. I tested the latest releases of Firefox & Chrome for Windows 10.

Setup	Chrome “do not track”	Firefox “do not track”	Firefox “tracking protection”
GTM	Pass	Pass	Fail
On page	Pass	Pass	Fail
GTM + renamed script & function	Pass	Pass	Fail
On page + renamed script & function	Pass	Pass	Fail

Again, it doesn’t seem that the tweaked setups are doing much work for us here.

Lost data due to “do not track”: <1%

Only Firefox Quantum’s “Tracking Protection,” introduced in February, had any effect on our trackers. Firefox has a 5% market share, but Tracking Protection is not enabled by default. The launch of this feature had no effect on the trend for Firefox traffic on Distilled.net.

Reason 3: Filters

It’s a bit of an obvious one, but filters you’ve set up in your analytics might intentionally or unintentionally reduce your reported traffic levels.

For example, a filter excluding certain niche screen resolutions that you believe to be mostly bots, or internal traffic, will obviously cause your setup to underreport slightly.

Lost data due to filters: ???

Impact is hard to estimate, as setup will obviously vary on a site-by site-basis. I do recommend having a duplicate, unfiltered “master” view in case you realize too late you’ve lost something you didn’t intend to.

Reason 4: GTM vs. on-page vs. misplaced on-page

Google Tag Manager has become an increasingly popular way of implementing analytics in recent years, due to its increased flexibility and the ease of making changes. However, I’ve long noticed that it can tend to underreport vs. on-page setups.

I was also curious about what would happen if you didn’t follow Google’s guidelines in setting up on-page code.

By combining my numbers with numbers from my colleague Dom Woodman’s site (you’re welcome for the link, Dom), which happens to use a Drupal analytics add-on as well as GTM, I was able to see the difference between Google Tag Manager and misplaced on-page code (right at the bottom of the <body> tag) I then weighted this against my own Google Tag Manager data to get an overall picture of all 5 setups.

Effect of GTM and misplaced on-page code

Traffic as a percentage of baseline (standard Google Tag Manager implementation):

	Google Tag Manager	Modified & Google Tag Manager	On-Page Code In <head>	Modified & On-Page Code In <head>	On-Page Code Misplaced In <Body>
Chrome	100.00%	98.75%	100.77%	99.80%	94.75%
Safari	100.00%	99.42%	100.55%	102.08%	82.69%
Firefox	100.00%	99.71%	101.16%	101.45%	90.68%
Internet Explorer	100.00%	80.06%	112.31%	113.37%	77.18%

There are a few main takeaways here:

• On-page code generally reports more traffic than GTM
• Modified code is generally within a margin of error, apart from modified GTM code on Internet Explorer (see note below)
• Misplaced analytics code will cost you up to a third of your traffic vs. properly implemented on-page code, depending on browser (!)
• The customized setups, which are designed to get more traffic by evading ad blockers, are doing nothing of the sort.

It’s worth noting also that the customized implementations actually got less traffic than the standard ones. For the on-page code, this is within the margin of error, but for Google Tag Manager, there’s another reason — because I used unfiltered profiles for the comparison, there’s a lot of bot spam in the main profile, which primarily masquerades as Internet Explorer. Our main profile is by far the most spammed, and also acting as the baseline here, so the difference between on-page code and Google Tag Manager is probably somewhat larger than what I’m reporting.

I also split the data by mobile, out of curiosity:

Traffic as a percentage of baseline (standard Google Tag Manager implementation):

	Google Tag Manager	Modified & Google Tag Manager	On-Page Code In <head>	Modified & On-Page Code In <head>	On-Page Code Misplaced In <Body>
Desktop	100.00%	98.31%	100.97%	100.89%	93.47%
Mobile	100.00%	97.00%	103.78%	100.42%	89.87%
Tablet	100.00%	97.68%	104.20%	102.43%	88.13%

The further takeaway here seems to be that mobile browsers, like Internet Explorer, can struggle with Google Tag Manager.

Lost data due to GTM: 1–5%

Google Tag Manager seems to cost you a varying amount depending on what make-up of browsers and devices use your site. On Distilled.net, the difference is around 1.7%; however, we have an unusually desktop-heavy and tech-savvy audience (not much Internet Explorer!). Depending on vertical, this could easily swell to the 5% range.

Lost data due to misplaced on-page code: ~10%

On Teflsearch.com, the impact of misplaced on-page code was around 7.5%, vs Google Tag Manager. Keeping in mind that Google Tag Manager itself underreports, the total loss could easily be in the 10% range.

Bonus round: Missing data from channels

I’ve focused above on areas where you might be missing data altogether. However, there are also lots of ways in which data can be misrepresented, or detail can be missing. I’ll cover these more briefly, but the main issues are dark traffic and attribution.

Dark traffic

Dark traffic is direct traffic that didn’t really come via direct — which is generally becoming more and more common. Typical causes are:

• Untagged campaigns in email
• Untagged campaigns in apps (especially Facebook, Twitter, etc.)
• Misrepresented organic
• Data sent from botched tracking implementations (which can also appear as self-referrals)

It’s also worth noting the trend towards genuinely direct traffic that would historically have been organic. For example, due to increasingly sophisticated browser autocompletes, cross-device history, and so on, people end up “typing” a URL that they’d have searched for historically.

Attribution

I’ve written about this in more detail here, but in general, a session in Google Analytics (and any other platform) is a fairly arbitrary construct — you might think it’s obvious how a group of hits should be grouped into one or more sessions, but in fact, the process relies on a number of fairly questionable assumptions. In particular, it’s worth noting that Google Analytics generally attributes direct traffic (including dark traffic) to the previous non-direct source, if one exists.

Discussion

I was quite surprised by some of my own findings when researching this post, but I’m sure I didn’t get everything. Can you think of any other ways in which data can end up missing from analytics?

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

How Do You Set Smart SEO Goals for Your Team/Agency/Project? - Whiteboard Friday

Posted by randfish

Are you sure that your current SEO goals are the best fit for your organization? It's incredibly important that they tie into both your company goals and your marketing goals, as well as provide specific, measurable metrics you can work to improve. In this edition of Whiteboard Friday, Rand outlines how to set the right SEO goals for your team and shares two examples of how different businesses might go about doing just that.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Howdy, Moz fans, and welcome to another edition of Whiteboard Friday. This week we're chatting about SEO goals, how to set smart ones, how to measure your progress against them, how to amplify those goals to the rest of your organization so that people really buy in to SEO.

This is a big challenge. So many folks that I've talked to in the field have basically said, "I'm not sure exactly how to set goals for our SEO team that are the right ones." I think that there's a particularly pernicious problem once Google took away the keyword-level data for SEO referrals.

So, from paid search, you can see this click was on this keyword and sent traffic to this page and then here's how it performed after that. In organic search, you can no longer do that. You haven't been able to do it for a few years now. Because of that removal, proving the return on investment for SEO has been really challenging. We'll talk in a future Whiteboard Friday about proving ROI. But let's focus here on how you get some smart SEO goals that are actually measurable, trackable, and pertain intelligently to the goals of the business, the organization.

Where to start:

So the first thing, the first problem that I see is that a lot of folks start here, which seems like a reasonable idea, but is actually a terrible idea. Don't start with your SEO goals. When your SEO team gets together or when you get together with your consultants, your agency, don't start with what the SEO goals should be.

• Start with the company goals. This is what our company is trying to accomplish this quarter or this year or this month.
• Marketing goals. Go from there to here's how marketing is going to contribute to those company goals. So if the company has a goal of increasing sales, marketing's job is what? Is marketing's job improving the conversion funnel? Is it getting more traffic to the top of the funnel? Is it bringing back more traffic that's already been to the site but needs to be re-earned? Those marketing goals should be tied directly to the company goals so that anyone and everyone in the organization can clearly see, "Here's why marketing is doing what they're doing."
• SEO goals. Next, here's how SEO contributes to those marketing goals. So if the goal is around, as we mentioned, growing traffic to the top of the funnel, for example, SEO could be very broad in their targeting. If it's bringing people back, you've got to get much more narrow in your keyword targeting.
• Specific metrics to measure and improve. From those SEO goals, you can get the outcome of specific metrics to measure and improve.

Measurable goal metrics

So that list is kind of right here. It's not very long. There are not that many things in the SEO world that we can truly measure directly. So measurable goal metrics might be things like...

1. Rankings. Which we can measure in three ways. We can measure them globally, nationally, or locally. You can choose to set those up.

2. Organic search visits. So this would be just the raw traffic that is sent from organic search.

3. You can also separate that into branded search versus non-branded search. But it's much more challenging than it is with paid, because we don't have the keyword data. Thus, we have to use an implied or inferred model, where essentially we say, "These pages are likely to be receiving branded search traffic, versus these pages that are likely to be receiving non-branded search traffic."

A good example is the homepage of most brands is most likely to get primarily branded search traffic, whereas resource pages, blog pages, content marketing style pages, those are mostly going to get unbranded. So you can weight those appropriately as you see fit.

Tracking your rankings is crucially important, because that way you can see which pages show up for branded queries versus which pages show up for unbranded queries, and then you can build pretty darn good models of branded search versus non-branded search visits based on which landing pages are going to get traffic.

4. SERP ownership. So ideas around your reputation in the search results. So this is essentially looking at the page of search results that comes up for a given query and what results are in there. There might be things you don't like and don't want and things you really do want, and the success and failure can be measured directly through the rankings in the SERP.

5. Search volume. So for folks who are trying to improve their brand's affinity and reputation on the web and trying to grow the quantity of branded search, which is a good metric, you can look at that through things like Google Trends or through a Google AdWords campaign or through something like Moz's Keyword Explorer.

6. Links and link metrics. So you could look at the growth or shrinkage of links over time. You can measure that through things like the number of linking root domains, the total number of links. Authority or spam metrics and how those are distributed.

7. Referral traffic. And last, but not least, most SEO campaigns, especially those that focus on links or improving rankings, are going to also send referral traffic from the links that are built. So you can watch referral traffic and what those referrers are and whether they came from pages where you built links with SEO intent.

So taking all of these metrics, these should be applied to the SEO goals that you choose that match up with your marketing and company goals. I wanted to try and illustrate this, not just explain it, but illustrate it through two examples that are very different in what they're measuring.

Example one

So, first off, Taft Boots, they've been advertising like crazy to me on Instagram. Apparently, I must need new boots.

• Grow online sales. Let's say that their big company goal for 2018 is "grow online sales to core U.S. customers, so the demographics and psychographics they're already reaching, by 30%."
• Increase top of funnel website traffic by 50%. So marketing says, "All right, you know what? There's a bunch of ways to do that, but we think that our best opportunity to do that is to grow top of funnel, because we can see how top of funnel turns into sales over time, and we're going to target a number of 50% growth." This is awesome. This can turn into very measurable, actionable SEO goals.
• Grow organic search visits 70%. We can say, "Okay, we know that search is going to contribute an outsized quantity of this 50% growth. So what we want to do is take search traffic up by 70%. How are we going to do that? We have four different plans.
  • A. We're going to increase our blog content, quality and quantity.
  • B. We're going to create new product pages that are more detailed, that are better optimized, that target good searches.
  • C. We're going to create a new resources section with some big content pieces.
  • D. We're going to improve our link profile and Domain Authority."

Now, you might say, "Wait a minute. Rand, this is a pretty common SEO methodology here." Yes, but many times this is not directly tied to the marketing goals, which is not directly tied to the business goals. If you want to have success as an SEO, you want to convince people to keep investing in you, you want to keep having that job or that consulting gig, you've got to connect these up.

From these, we can then say, "Okay, for each one, how do we measure it?" Well...

• A. Quantity of content and search visits/piece. Blog content can be measured through the quantity of content we produce, the search visits that each of those pieces produce, and what the distribution and averages are.
• B. Rankings and organic traffic. Is a great way to measure product pages and whether we're hitting our goals there.
• C. Link growth, rankings, and traffic. That's a great way to measure the new resources section.
• D. Linking root domains plus the DA distribution and maybe Spam Score distribution. That's a great way to measure whether we're improving our link profile.

All of these, this big-picture goal is going to be measured by the contribution of search visits to essentially non-homepage and non-branded pages that contribute to the conversion funnel. So we have a methodology to create a smart goal and system here.

Example two

Another example, totally different, but let's try it out because I think that many folks have trouble connecting non-e-commerce pages, non-product stuff. So we're going to use Book-It Theatre. They're a theater group here in the Seattle area. They use the area beneath Seattle Center House as their space. They basically will take popular books and literature and convert them into plays. They'll adapt them into screenplays and then put on performances. It's quite good. We've been to a few shows, Geraldine and I have, and we really like them.

So their goal — I'm making this up, I don't actually know if this is their goal — but let's say they want to...

• Attract theater goers from outside the Seattle area. So they're looking to hit tourists and critics, people who are not just locals, because they want to expand their brand.
• Reach audiences in 4 key geographies — LA, Portland, Vancouver, Minneapolis. So they decide, "You know what? Marketing can contribute to this in four key geographies, and that's where we're going to focus a bunch of efforts — PR efforts, outreach efforts, offline media, and SEO. The four key geographies are Los Angeles, Portland, Vancouver, and Minneapolis. We think these are good theater-going towns where we can attract the right audiences."

So what are we going to do as SEOs? Well, as SEOs, we better figure out what's going to match up to this.

• Drive traffic from these regions to Book-It Theatre's pages and to reviews of our show. So it's not just content on our site. We want to drive people to other critics and press that's reviewed us.
  • A. So we're going to create some geo landing pages, maybe some special offers for people from each of these cities.
  • B. We're going to identify third-party reviews and hopefully get critics who will write reviews, and we're going to ID those and try and drive traffic to them.
  • C. We're going to do the same with blog posts and informal critics.
  • D. We're going to build some content pages around the books that we're adapting, hoping to drive traffic, that's interested in those books, from all over the United States to our pages and hopefully to our show.

So there are ways to measure each of these.

• A. Localized rankings in Moz Pro or a bunch of other rank tracking tools. You can set up geo-specific localized rankings. "I want to track rankings in Vancouver, British Columbia. I want to track rankings from Los Angeles, California." Those might look different than the ones you see here in Seattle, Washington.
• B. We can do localized rankings and visits from referrals for the third-party reviews. We won't be able to track the visits that those pages receive, but if they mention Book-It Theatre and link to us, we can see, oh yes, look, the Minneapolis Journal wrote about us and they linked to us, and we can see what the reviews are from there.
• C. We can do localized rankings and visits from referrals for the third-party blog posts.
• D. Local and national ranking, organic visits. For these Book-It content pages, of course, we can track our local and national rankings and the organic visits.

Each of these, and as a whole, the contribution of search visits from non-Seattle regions, so we can remove Seattle or Washington State in our analytics and we can see: How much traffic did we get from there? Was it more than last year? What's it contributing to the ticket sales conversion funnel?

You can see how, if you build these smart goals and you measure them correctly and you align them with what the company and the marketing team is trying to do, you can build something really special. You can get great involvement from the rest of your teams, and you can show the value of SEO even to people who might not believe in it already.

All right, everyone. Look forward to your thoughts and feedback in the comments, and we'll see you again next week for another edition of Whiteboard Friday. Take care.

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

The MozCon 2018 Final Agenda

Posted by Trevor-Klein

MozCon 2018 is just around the corner — just over six weeks away — and we're excited to share the final agenda with you today. There are some familiar faces, and some who'll be on the MozCon stage for the first time, with topics ranging from the evolution of searcher intent to the increasing importance of local SEO, and from navigating bureaucracy for buy-in to cutting the noise out of your reporting.

We're also thrilled to announce this year's winning pitches for our six MozCon Community Speaker slots! If you're not familiar, each year we hold several shorter speaking slots, asking you all to submit your best pitches for what you'd like to teach everyone at MozCon. The winners — all members of the Moz Community — are invited to the conference alongside all our other speakers, and are always some of the most impressive folks on the stage. Check out the details of their talks below, and congratulations to this year's roster!

Still need your tickets? We've got you covered, but act fast — they're over 70% sold!

Pick up your ticket to MozCon!

The Agenda

---

Monday, July 9

---

8:30–9:30 am

Breakfast and registration

Doors to the conference will open at 8:00 for those looking to avoid registration lines and grab a cup of coffee (or two) before breakfast, which will be available starting at 8:30.

---

9:30–9:45 am

Welcome to MozCon 2018!
Sarah Bird

Moz CEO Sarah Bird will kick things off by sharing everything you need to know about your time at MozCon 2018, including conference logistics and evening events.

She'll also set the tone for the show with an update on the state of the SEO industry, illustrating the fact that there's more opportunity in it now than there's ever been before.

---

9:50–10:20 am

The Democratization of SEO
Jono Alderson

How much time and money we collectively burn by fixing the same kinds of basic, "binary," well-defined things over and over again (e.g., meta tags, 404s, URLs, etc), when we could be teaching others throughout our organizations not to break them in the first place?

As long as we "own" technical SEO, there's no reason (for example) for the average developer to learn it or care — so they keep making the same mistakes. We proclaim that others are doing things wrong, but by doing so we only reinforce the line between our skills and theirs.

We need to start giving away bits of the SEO discipline, and technical SEO is probably the easiest thing for us to stop owning. We need more democratization, education, collaboration, and investment in open source projects so we can fix things once, rather than a million times.

---

10:20–10:50 am

Mobile-First Indexing or a Whole New Google
Cindy Krum

The emergence of voice-search and Google Assistant is forcing Google to change its model in search, to favor their own entity understanding or the world, so that questions and queries can be answered in context. Many marketers are struggling to understand how their website and their job as an SEO or SEM will change, as searches focus more on entity-understanding, context and action-oriented interaction. This shift can either provide massive opportunities, or create massive threats to your company and your job — the main determining factor is how you choose to prepare for the change.

---

10:50–11:20 am

AM Break

---

11:30–11:50 am

It Takes a Village:
2x Your Paid Search Revenue by Smashing Silos
Community speaker: Amy Hebdon

Your company's unfair advantage to skyrocketing paid search revenue is within your reach, but it's likely outside the control of your paid search team. Good keywords and ads are just a few cogs in the conversion machine. The truth is, the success of the entire channel depends on people who don't touch the campaigns, and may not even know how paid search works. We'll look at how design, analysis, UX, PM and other marketing roles can directly impact paid search performance, including the most common issues that arise, and how to immediately fix them to improve ROI and revenue growth.

---

11:50 am–12:10 pm

The #1 and Only Reason Your SEO Clients Keep Firing You
Community speaker: Meredith Oliver

You have a kick-ass keyword strategy. Seriously, it could launch a NASA rocket; it's that good. You have the best 1099 local and international talent on your SEO team that working from home and an unlimited amount of free beard wax can buy. You have a super-cool animal inspired company name like Sloth or Chinchilla that no one understands, but the logo is AMAZING. You have all of this, yet, your client turnover rate is higher than Snoop Dogg's audience on an HBO comedy special. Why? You don't talk to your clients. As in really communicate, teach them what you know, help them get it, really get it, talk to them. How do I know? I was you. In my agency's first five years we churned and burned through clients faster than Kim Kardashian could take selfies. My mastermind group suggested we *proactively* set up and insist upon a monthly review meeting with every single client. It was a game-changer, and we immediately adopted the practice. Ten years later we have a 90% client retention rate and more than 30 SEO clients on retainer.

---

12:10–12:30 pm

Why "Blog" Is a Misnomer for Our 2018 Content Strategy
Community speaker: Taylor Coil

At the end of 2017, we totally redesigned our company's blog. Why? Because it's not really a blog anymore - it's an evergreen collection of traffic and revenue-generating resources. The former design catered to a time-oriented strategy surfacing consistently new posts with short half-lives. That made sense when we started our blog in 2014. Today? Not so much. In her talk, Taylor will detail how to make the perspective shift from "blog" to "collection of resources," why that shift is relevant in 2018's content landscape, and what changes you can make to your blog's homepage, nav, and taxonomy that reflect this new perspective.

---

12:30–2:00 pm

Lunch

---

2:05–2:35 pm

Near Me or Far:
How Google May Be Deciding Your Local Intent For You
Rob Bucci

In August 2017, Google stated that local searches without the "near me" modifier had grown by 150% and that searchers were beginning to drop geo-modifiers — like zip code and neighborhood — from local queries altogether. But does Google still know what searchers are after?

For example: the query [best breakfast places] suggests that quality takes top priority; [breakfast places near me] indicates that close proximity is essential; and [breakfast places in Seattle] seems to cast a city-wide net; while [breakfast places] is largely ambiguous.

By comparing non-geo-modified keywords against those modified with the prepositional phrases "near me" and "in [city name]" and qualifiers like "best," we hope to understand how Google interprets different levels of local intent and uncover patterns in the types of SERPs produced.

With a better understanding of how local SERPs behave, SEOs can refine keyword lists, tailor content, and build targeted campaigns accordingly.

---

2:35–3:05 pm

None of Us Is as Smart as All of Us
Lisa Myers

Success in SEO, or in any discipline, is frequently reliant on people's ability to work together. Lisa Myers started Verve Search in 2009, and from the very beginning was convinced of the importance of building a diverse team, then developing and empowering them to find their own solutions.

In this session she'll share her experiences and offer actionable advice on how to attract, develop, and retain the right people in order to build a truly world-class team.

---

3:05–3:35 pm

PM Break

---

3:45–4:15 pm

Search-Driven Content Strategy
Stephanie Briggs

Google's improvements in understanding language and search intent have changed how and why content ranks. As a result, many SEOs are chasing rankings that Google has already decided are hopeless. Stephanie will cover how this should impact the way you write and optimize content for search, and will help you identify the right content opportunities. She'll teach you how to persuade organizations to invest in content, and will share examples of strategies and tactics she has used to grow content programs by millions of visits.

---

4:15–4:55 pm

Ranking Is a Promise: Can You Deliver?
Dr. Pete Meyers

In our rush to rank, we put ourselves first, neglecting what searchers (and our future customers) want. Google wants to reward sites that deliver on searcher intent, and SERP features are a window into that intent. Find out how to map keywords to intent, understand how intent informs the buyer funnel, and deliver on the promise of ranking to drive results that attract clicks and customers.

---

7:00–10:00 pm

Kickoff Party

Networking the Mozzy way! Join us for an evening of fun on the first night of the conference (stay tuned for all the details!).

---

---

Tuesday, July 10

---

8:30–9:30 am

Breakfast

---

9:35–10:15 am

Content Marketing Is Broken
and Only Your M.O.M. Can Save You
Oli Gardner

Traditional content marketing focuses on educational value at the expense of product value, which is a broken and outdated way of thinking. We all need to sell a product, and our visitors all need a product to improve their lives, but we're so afraid of being seen as salesy that somehow we got lost, and we forgot why our content even exists. We need our M.O.M.s! No, not your actual mother. Your Marketing Optimization Map — your guide to exploring the nuances of optimized content marketing through a product-focused lens.

In this session you'll learn data and lessons from Oli's biggest ever content marketing experiment, and how those lessons have changed his approach to content; a context-to-content-to-conversion strategy for big content that converts; advanced methods for creating "choose your own adventure" navigational experiences to build event-based behavioral profiles of your visitors (using GTM and GA); and innovative ways to productize and market the technology you already have, with use cases your customers had never considered.

---

10:15–10:45 am

Lies, Damned Lies, and Analytics
Russ Jones

Search engine optimization is a numbers game. We want some numbers to go up (links, rankings, traffic, and revenue), others to go down (bounce rate, load time, and budget). Underlying all these numbers are assumptions that can mislead, deceive, or downright ruin your campaigns. Russ will help uncover the hidden biases, distortions, and fabrications that underlie many of the metrics we have come to trust implicitly and from the ashes show you how to build metrics that make a difference.

---

10:45–11:15 am

AM Break

---

11:25–11:55 am

The Awkward State of Local
Mike Ramsey

You know it exists. You know what a citation is, and have a sense for the importance of accurate listings. But with personalization and localization playing an increasing role in every SERP, local can no longer be seen in its own silo — every search and social marketer should be honing their understanding. For that matter, it's also time for local search marketers to broaden the scope of their work.

---

11:55 am–12:25 pm

The SEO Cyborg:
Connecting Search Technology and Its Users
Alexis Sanders

SEO requires a delicate balance of working for the humans you're hoping to reach, and the machines that'll help you reach them. To make a difference in today's SERPs, you need to understand the engines, site configurations, and even some machine learning, in addition to the emotional, raw, authentic connections with people and their experiences. In this talk, Alexis will help marketers of all stripes walk that line.

---

12:25–1:55 pm

Lunch

---

2:00–2:30 pm

Email Unto Others:
The Golden Rules for Human-Centric Email Marketing
Justine Jordan

With the arrival of GDPR and the ease with which consumers can unsubscribe and report spam, it's more important than ever to treat people like people instead of just leads. To understand how email marketing is changing and to identify opportunities for brands, Litmus surveyed more than 3,000 marketers worldwide. Justine will cover the biggest trends and challenges facing email today and help you put the human back in marketing’s most personal — and effective — marketing channel.

---

2:30–3:00 pm

Your Red-Tape Toolkit:
How to Win Trust and Get Approval for Search Work
Heather Physioc

Are your search recommendations overlooked and misunderstood? Do you feel like you hit roadblocks at every turn? Are you worried that people don't understand the value of your work? Learn how to navigate corporate bureaucracy and cut through red tape to help clients and colleagues understand your search work — and actually get it implemented. From diagnosing client maturity to communicating where search fits into the big picture, these tools will equip you to overcome obstacles to doing your best work.

---

3:00–3:30 pm

PM Break

---

3:40–4:10 pm

The Problem with Content &
Other Things We Don't Want to Admit
Casie Gillette

Everyone thinks they need content but they don't think about why they need it or what they actually need to create. As a result, we are overwhelmed with poor quality content and marketers are struggling to prove the value. In this session, we'll look at some of the key challenges facing marketers and how a data-driven strategy can help us make better decisions.

---

4:10–4:50 pm

Excel Is for Rookies:
Why Every Search Marketer Needs to Get Strong in BI, ASAP
Wil Reynolds

The analysts are coming for your job, not AI (at least not yet). Analysts stopped using Excel years ago; they use Tableau, Power BI, Looker! They see more data than you, and that is what is going to make them a threat to your job. They might not know search, but they know data. I'll document my obsession with Power BI and the insights I can glean in seconds which is helping every single client at Seer at the speed of light. Search marketers must run to this opportunity, as analysts miss out on the insights because more often than not they use these tools to report. We use them to find insights.

---

---

Wednesday, July 11

---

8:30–9:30 am

Breakfast

---

9:35–10:15 am

Machine Learning for SEOs
Britney Muller

People generally react to machine learning in one of two ways: either with a combination of fascination and terror brought on by the possibilities that lie ahead, or with looks of utter confusion and slight embarrassment at not really knowing much about it. With the advent of RankBrain, not even higher-ups at Google can tell us exactly how some things rank above others, and the impact of machine learning on SEO is only going to increase from here. Fear not: Moz's own senior SEO scientist, Britney Muller, will talk you through what you need to know.

---

10:15–10:45 am

Shifting Toward Engagement and Reviews
Darren Shaw

With search results adding features and functionality all the time, and users increasingly finding what they need without ever leaving the SERP, we need to focus more on the forest and less on the trees. Engagement and behavioral optimization are key. In this talk, Darren will offer new data to show you just how tight the proximity radius around searchers really is, and how reviews can be your key competitive advantage, detailing new strategies and tactics to take your reivews to the next level.

---

10:45–11:15 am

AM Break

---

11:25–11:45 am

Location-Free Local SEO
Community speaker: Tom Capper

Let's talk about local SEO without physical premises. Not the Google My Business kind — the kind of local SEO that job boards, house listing sites, and national delivery services have to reckon with. Should they have landing pages, for example, for "flower delivery in London?"

This turns out to be a surprisingly nuanced issue: In some industries, businesses are ranking for local terms without a location-specific page, and in others local pages are absolutely essential. I've worked with clients across several industries on why these sorts of problems exist, and how to tackle them. How should you figure out whether you need these pages, how can you scale them and incorporate them in your site architecture, and how many should you have for what location types?

---

11:45 am–12:05 pm

SEO without Traffic:
Community speaker: Hannah Thorpe

Answer boxes, voice search, and a reduction in the number of results displayed sometimes all result in users spending more time in the SERPs and less on our websites. But does that mean we should stop investing in SEO?

This talk will cover what metrics we should now care about, and how strategies need to change, covering everything from measuring more than just traffic and rankings to expanding your keyword research beyond just keyword volumes.

---

12:05–12:25 pm

Tools Change, People Don't:
Empathy-Driven Online Marketing
Community speaker: Ashley Greene

When everyone else zags, the winners zig. As winners, while your 101+ competitors are trying to automate 'til the cows come home and split test their way to greatness‚ you're zigging. Whether you're B2B or B2C, you're marketing to humans. Real people. Homo sapiens. But where is the human element in the game plan? Quite simply, it has gone missing, which provides a window of opportunity for the smartest marketers.

In this talk, Ashley will provide a framework of simple user interview and survey techniques to build customer empathy and your "voice of customer" playbook. Using real examples from companies like Slack, Pinterest, Intercom, and Airbnb, this talk will help you uncover your customers' biggest problems and pain points; know what, when, and how your customers research (and Google!) a need you solve; and find new sources of information and influencers so you can unearth distribution channels and partnerships.

---

12:25–1:55 pm

Lunch

---

2:00–2:30 pm

You Don't Know SEO
Michael King

Or maybe, "SEO you don't know you don't know." We've all heard people throw jargon around in an effort to sound smart when they clearly don't know what it means, and our industry of SEO is no exception. There are aspects of search that are acknowledged as important, but seldom actually understood. Michael will save us from awkward moments, taking complex topics like the esoteric components of information retrieval and log-file analysis, pairing them with a detailed understanding of technical implementation of common SEO recommendations, and transforming them into tools and insights we wish we'd never neglected.

---

2:30–3:00 pm

What All Marketers Can Do about Site Speed
Emily Grossman

At this point, we should all have some idea of how important site speed is to our performance in search. The recently announced "speed update" underscored that fact yet again. It isn't always easy for marketers to know where to start improving their site's speed, though, and a lot of folks mistakenly believe that site speed should only be a developer's problem. Emily will clear that up with an actionable tour of just how much impact our own work can have on getting our sites to load quickly enough for today's standards.

---

3:00–3:30 pm

PM Break

---

3:40–4:10 pm

Traffic vs. Signal
Dana DiTomaso

With an ever-increasing slate of options in tools like Google Tag Manager and Google Data Studio, marketers of all stripes are falling prey to the habit of "I'll collect this data because maybe I'll need it eventually," when in reality it's creating a lot of noise for zero signal.

We're still approaching our metrics from the organization's perspective, and not from the customer's perspective. Why, for example, are we not reporting on (or even thinking about, really) how quickly a customer can do what they need to do? Why are we still fixated on pageviews? In this talk, Dana will focus our attention on what really matters.

---

4:10–4:50 pm

Why Nine out of Ten Marketing Launches Suck
(And How to Be the One that Doesn't)
Rand Fishkin

More than ever before, marketers are launching things — content, tools, resources, products — and being held responsible for how/whether they resonate with customers and earn the amplification required to perform. But this is hard. Really, really hard. Most of the projects that launch, fail. What separates the wheat from the chaff isn't just the quality of what's built, but the process behind it. In this presentation, Rand will present examples of dismal failures and skyrocketing successes, and dive into what separates the two. You'll learn how anyone can make a launch perform better, and benefit from the power of being "new."

---

7:00–11:30 pm

MozCon Bash

Join us at Garage Billiards to wrap up the conference with an evening of networking, billiards, bowling, and karaoke with MozCon friends new and old. Don't forget to bring your MozCon badge and US ID or passport.

---

---

Grab your ticket today!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

Backlink Blindspots: The State of Robots.txt

Posted by rjonesx.

Here at Moz we have committed to making Link Explorer as similar to Google as possible, specifically in the way we crawl the web. I have discussed in previous articles some metrics we use to ascertain that performance, but today I wanted to spend a little bit of time talking about the impact of robots.txt and crawling the web.

Most of you are familiar with robots.txt as the method by which webmasters can direct Google and other bots to visit only certain pages on the site. Webmasters can be selective, allowing certain bots to visit some pages while denying other bots access to the same. This presents a problem for companies like Moz, Majestic, and Ahrefs: we try to crawl the web like Google, but certain websites deny access to our bots while allowing that access to Googlebot. So, why exactly does this matter?

Why does it matter?

As we crawl the web, if a bot encounters a robots.txt file, they're blocked from crawling specific content. We can see the links that point to the site, but we're blind regarding the content of the site itself. We can't see the outbound links from that site. This leads to an immediate deficiency in the link graph, at least in terms of being similar to Google (if Googlebot is not similarly blocked).

But that isn't the only issue. There is a cascading failure caused by bots being blocked by robots.txt in the form of crawl prioritization. As a bot crawls the web, it discovers links and has to prioritize which links to crawl next. Let's say Google finds 100 links and prioritizes the top 50 to crawl. However, a different bot finds those same 100 links, but is blocked by robots.txt from crawling 10 of the top 50 pages. Instead, they're forced to crawl around those, making them choose a different 50 pages to crawl. This different set of crawled pages will return, of course, a different set of links. In this next round of crawling, Google will not only have a different set they're allowed to crawl, the set itself will differ because they crawled different pages in the first place.

Long story short, much like the proverbial butterfly that flaps its wings eventually leading to a hurricane, small changes in robots.txt which prevent some bots and allow others ultimately leads to very different results compared to what Google actually sees.

So, how are we doing?

You know I wasn't going to leave you hanging. Let's do some research. Let's analyze the top 1,000,000 websites on the Internet according to Quantcast and determine which bots are blocked, how frequently, and what impact that might have.

Methodology

The methodology is fairly straightforward.

1. Download the Quantcast Top Million
2. Download the robots.txt if available from all top million sites
3. Parse the robots.txt to determine whether the home page and other pages are available
4. Collect link data related to blocked sites
5. Collect total pages on-site related to blocked sites.
6. Report the differences among crawlers.

Total sites blocked

The first and easiest metric to report is the number of sites which block individual crawlers (Moz, Majestic, Ahrefs) while allowing Google. Most site that block one of the major SEO crawlers block them all. They simply formulate robots.txt to allow major search engines while blocking other bot traffic. Lower is better.

Of the sites analyzed, 27,123 blocked MJ12Bot (Majestic), 32,982 blocked Ahrefs, and 25,427 blocked Moz. This means that among the major industry crawlers, Moz is the least likely to be turned away from a site that allows Googlebot. But what does this really mean?

Total RLDs blocked

As discussed previously, one big issue with disparate robots.txt entries is that it stops the flow of PageRank. If Google can see a site, they can pass link equity from referring domains through the site's outbound domains on to other sites. If a site is blocked by robots.txt, it's as though the outbound lanes of traffic on all the roads going into the site are blocked. By counting all the inbound lanes of traffic, we can get an idea of the total impact on the link graph. Lower is better.

According to our research, Majestic ran into dead ends on 17,787,118 referring domains, Ahrefs on 20,072,690 and Moz on 16,598,365. Once again, Moz's robots.txt profile was most similar to that of Google's. But referring domains isn't the only issue with which we should be concerned.

Total pages blocked

Most pages on the web only have internal links. Google isn't interested in creating a link graph — they're interested in creating a search engine. Thus, a bot designed to act like Google needs to be just as concerned about pages that only receive internal links as they are those that receive external links. Another metric we can measure is the total number of pages that are blocked by using Google's site: query to estimate the number of pages Google has access to that a different crawler does not. So, how do the competing industry crawlers perform? Lower is better.

Once again, Moz shines on this metric. It's not just that Moz is blocked by fewer sites— Moz is blocked by less important and smaller sites. Majestic misses the opportunity to crawl 675,381,982 pages, Ahrefs misses 732,871,714 and Moz misses 658,015,885. There's almost an 80 million-page difference between Ahrefs and Moz just in the top million sites on the web.

Unique sites blocked

Most of the robots.txt disallows facing Moz, Majestic, and Ahrefs are simply blanket blocks of all bots that don't represent major search engines. However, we can isolate the times when specific bots are named deliberately for exclusion while competitors remain. For example, how many times is Moz blocked while Ahrefs and Majestic are allowed? Which bot is singled out the most? Lower is better.

Ahrefs is singled out by 1201 sites, Majestic by 7152 and Moz by 904. It is understandable that Majestic has been singled out, given that they have been operating a very large link index for many years, a decade or more. It took Moz 10 years to accumulate 904 individual robots.txt blocks, and took Ahrefs 7 years to accumulate 1204. But let me give some examples of why this is important.

If you care about links from name.com, hypermart.net, or eclipse.org, you can't rely solely on Majestic.

If you care about links from popsugar.com, dict.cc, or bookcrossing.com, you can't rely solely on Moz.

If you care about links from dailymail.co.uk, patch.com, or getty.edu, you can't rely solely on Ahrefs.

And regardless of what you do or which provider you use, you can't links from yelp.com, who.int, or findarticles.com.

Conclusions

While Moz's crawler DotBot clearly enjoys the closest robots.txt profile to Google among the three major link indexes, there's still a lot of work to be done. We work very hard on crawler politeness to ensure that we're not a burden to webmasters, which allows us to crawl the web in a manner more like Google. We will continue to work more to improve our performance across the web and bring to you the best backlink index possible.

Thanks to Dejan SEO for the beautiful link graph used in the header image and Mapt for the initial image used in the diagrams.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!