Connect with us

Hi, what are you looking for?

The American GeniusThe American Genius

SEO Myth: Dynamic URLs Can’t be Crawled

Dynamic URLs Cannot be Crawled....BUSTED!


Myth – BUSTED!

OK Realtors, let’s see a quick show of hands – how many of you have been told, overheard or just assumed that the search engines cannot crawl and index pages if the URL contains a question mark?  Don’t get nervous, my hand is up too.  How many of you think it’s true?  According to Google, this is one of the most prevalent myths there are concerning what search engines can and cannot do.

As is true of most myths, it is based in fact.  When the first robots began to crawl the web they were rather dumb and not able to follow URLs if they had a “?” in them.  Because of this, dynamic pages were not included in the search indexes.  Thankfully, this changed a few years ago as the robots began to mature.  Unfortunately for us, that information did not trickle down to all the web site builders and owners and the myth continued to flourish.

In a post on the Google webmaster blog, they state that if you have a URL that reads something like “…?entertainment=2&artist=14” they can follow it and index the resulting content.  However, they also say that having URLs like this is not recommended.  Google suggests it is wiser, and more reader friendly to use a keyword rich URL like “/entertainment/music/punk/the-ramones/ .”  Reader friendly content…where have I heard that before?  The suggested format, often referred to as “permalinks,” also looks much better in the Search Engine Result Pages (SERPs) and visitors are more likely to click on a link if they see what appears to be relevant information.

What Does This Mean to You?

If you’ve been worried about “fixing” your site so pages get listed…relax (a little).  They are probably already in the indexes and the content is helping you get a better placement in the rankings.  But, as you have read from me already (and will again), your goal should be the reader, not the robot.  Your click-thrus from searches will increase if you use human-speak in your URLs, not geek-speak.

Still Want to Fix Those URLs?

Good, I hoped you would.  When you create your new, reader-friendly URLs, you cannot just delete the old ones.  And, you also cannot just leave them alone either.  Hey, I never said that all SEO would be easy.

Advertisement. Scroll to continue reading.

If you delete all the old ones you lose all of the inbound link credit you have earned, lowering your search rankings and certainly frustrating anyone that gets a “file not found” error when they try to follow a link to your site.

If you leave all the old links alone, then you’ll probably get penalized by the search engines for having duplicate content on your site.

Fixing the URLs correctly will probably require the help of a web geek to create what’s called a “301 redirect.”  This is special code that will take your visitors to the correct place if they follow a now incorrect inbound link, and also tells the search engines where the new file is located (and to update the location in the index).  Depending on the size of your site, this may take some time as it will need to be done for every URL you are updating. If you are using a Content Management System (CMS), like WordPress, it may be a bit easier for you. There are plugins available that allow you to create and manage your 301 redirects from within your WordPress admin (Hey Lani & Benn…did you see that?).

I don’t think I need to say it, but I will anyway to to be sure – BEFORE you make any changes that will effect your entire site in one fell swoop, make and verify a back up.

How Important is it to Update my Dynamic URLs?

On a scale of 1 to 10, I’d say updating your dynamic URLs is only about a 4.  The engines can read your content so you are getting the full benefit of the page from that perspective.  However, there is the possibility of a slight increase in ranking if you were to use the key word rich URLs as suggested by Google.  If feel you have done all you can already and need to squeeze out a bit more SEO to nudge past the competition, then yes – you should do this.

Advertisement. Scroll to continue reading.

The best thing is to not let your site get out of control with the dynamic URLs to begin with.  If you are new to this, as some of you have said you are, talk to your technology providers and figure out up front how to use permalinks on your site.

Written By

Jack Leblond is a SEO/SEM professional working for a large corporation full time in Austin, TX. He is not a Realtor, he is our in-house SEO expert. Jack is the Director of Internet Strategy and Operations for TG (www.tgslc.org). In addition to managing the team that develops and maintains the company's multiple Web sites, he focuses on Search Engine Optimization (SEO), e-marketing and Social Media. Jack's background ranges from Submarine Sonar Technician/Instructor for the United States Navy, technical writer, pioneer in internet/intranet creation for McGraw-Hill and Times Mirror Higher Education, former Adjunct Professor for two Universities teaching web-related courses, has served as a city council member and co-founded Net-Smart, a web design and hosting company, where he managed networks and oversaw the development of hundreds of Web sites. As a free-lance SEO consultant, Jack performs SEO Site Audits for small/medium businesses that want their web sites to perform better in the search engine listings.

19 Comments

19 Comments

  1. Benn Rosales

    September 17, 2008 at 10:01 am

    We rank it at about 1 for changing (for us) based on this video which was one of a few primary references we used when rolling out Ag. https://www.viddler.com/explore/jpozadzides/videos/2/

    WP default was billed as the perfect seo engine in 06, and 07, and based on what we’ve seen there is a lot of truth in that position which includes the wp suggested link structure. Over time it has become more easy to change structures, but the risk of position makes the proposition of changing a suicide to a site of this magnitude.

    We have a strategy in the future where links are concerned that makes sense, that is based more on user ease and a little less on whether google sees the words in links as relivant.

  2. Todd

    September 17, 2008 at 12:20 pm

    Anyone wishing to step up from WordPress to Drupal will find lots of tools for 301’s and aliases;

    https://drupal.org/project/globalredirect

    Drupal renders pages just beautifully ( markup ) – I would be interested in Mr. Leblond’s opinion of how good he thinks Drupal’s SEO is.

  3. Jay Thompson

    September 17, 2008 at 2:07 pm

    Benn –

    I don’t have time right now to watch an hour long video. Can you summarize what Cutts said and what you’ve seen that supports a URL like “AgentGenius.com/?p=4571” would be better for SEO than say “AgentGenius.com/dynamic-url-myths-exposed” ?? There is no question that Google can “see” words in a URL, and little debate that keywords in URLs are part of the algorithm. So I struggle that “?p=####” would be better for SEO that some pertinent keyword(s). (not to mention that for me at least, I like to see a URL that makes sense as I tend to hover over links and look at the URL before clicking. But maybe that’s just me)

  4. Jack Leblond

    September 17, 2008 at 2:37 pm

    Jay – I took time today to watch the video. In it Matt Cutts, the Google spam lord talks about two types of URLs. First, those with dashes,underscores, or words all crammed together. He does say that it is better to use hyphens. But also says it’s not worth the trouble to change to them if you have one of the others. He also touches on dynamic URLs and says that the GoogleBot can index them – I didn’t hear mention of if they are good, better,best. Or if they should be left alone.

    I’ve attempted to contact Matt directly for an explanation and also reached out to a few A-List SEOs for their opinions. I’ll keep everyone updated as to what they say.

  5. Benn Rosales

    September 17, 2008 at 3:06 pm

    “default wordpress install is best” along with “if you’re using the ? post number you’re fine, google sees you” Btw, I didn’t say he said anything about better, what I am saying he said was it made no difference in googles eyes.

    Simply referring to the ? myth- we picked our url because or CM needs, it works- and God bless us we have a lot of content to manage.

    But if I may, I’ll go one step further on SEO for a wordpress BLOG (not static)- a BLOG is a wall of words, rich wonderful keyword rich words that make walls- if I may be so bold- your post TITLE matters, not the url behind it- otherwise thousands of CMS sites would be in a lot of trouble and rank at the very bottom of scale.

    Where I do believe it matters is in reverse use, an example would be if I were to track back to your recent post where you have 25 real estate related keywords in your url, but you can mitigate and maximize that by use of title=”the full title”.

    I’m probably right and probably wrong on some fronts, but the bottom line is the ? is fine with google. I would watch the video, there are a lot of little tidbits to be had.

    As I said earlier, we’ve got something more useful coming down the pike in the coming weeks but as for messing with it right now, it ranks a 1. I I set one up for a client today, I would go with a longer keyword url as Jack advises because in a local race, “every ounce” does matter.

  6. Jay Thompson

    September 17, 2008 at 4:07 pm

    I’ll watch the Cutts video (actually, I’m pretty sure I’ve seen it before, but any hour with Cutts is generally worth repeating).

    To b honest, I don’t pay a great deal of attention to SEO when I write. The “wall of words” Benn mentions does a pretty good job, and Google bots are pretty darn smart — they generally can tell what a post is about.

    Changing URL structure is a scary thought. 301 redirects? I’ve read horror stories about people incorrectly 301’ing .htaccess’ing and sending their sites into oblivion. Unless someone (or several someones) said whatever I was doing was fundamentally harmful, I wouldn’t touch them. Heck, I don’t even like opening the permalinks page in WP, much less changing anything on it. When I set up blogs for others, I use %postname% just because it seems like keywords in the URL (assuming the writer has used them in the title) can’t hurt and can only help.

  7. Bob

    September 17, 2008 at 5:12 pm

    Matt has stated more than once that urls can make a difference.

    They also play a part in conversion, but that really wouldn’t apply to AG.

  8. Hawaii real estate company

    September 17, 2008 at 5:28 pm

    I am in the camp that the URL’s that have words in them are better than dynamic ones. Here are a couple reasons why.

    1. Some experts say search engines rank them better if they actual have the keywords in them.
    2. More importantly: If the URL actual has words that the page is about, it will convert more traffic. Now that is what most people want.

    Same goes with the title as well.

  9. Bob

    September 17, 2008 at 9:01 pm

    Ey brah – AG comments are no follow. The comment spam doesn’t work here.

  10. Hawaii real estate company

    September 17, 2008 at 9:09 pm

    Hows it Bob? Nice pigeon. I know the comments are no follow. Just wanted to contribute to the conversation 🙂 Just like to sign as Hawaii real estate company. I am glad you have my back.

    Aloha Cuz

  11. Bob

    September 17, 2008 at 9:55 pm

    just joking.

  12. Jack Leblond

    September 18, 2008 at 7:58 am

    Hello all – Still nothing back from Matt Cutts, but the other SEOs I contacted agree that while it is definitely better to have keywords in the URL, if you site was created without them by using dynamic URLs, and you have a lot of pages – leave them alone, it’s not worth the trouble to change them.

  13. ines

    September 18, 2008 at 8:16 am

    Is there a maximum recommended number of 301 redirects? or better yet, would google penalize you for too many of them?

  14. Lenn Harley

    October 22, 2008 at 6:33 am

    I’ve had dynamic navigation on my web sites since about 2008. Google crawls almost every day. I believe that it helps to have text links and not images links.

    Lenn Harley
    Broker
    Homefinders.com

  15. Lenn Harley

    October 22, 2008 at 10:49 am

    I mispoke in my last comment.

    I’ve had dynamic navigation on my web sites since about 1998.

Leave a Reply

Your email address will not be published.

AdBlocker Message

Our website is kept FREE to you by displaying online ads to our visitors. Please consider supporting us by disabling your ad blocker OR subscribing to our email newsletter: https://theamericangenius.com/get-american-genius-newsletter/

The
American Genius
news neatly in your inbox

Subscribe to our mailing list for news sent straight to your email inbox.

Advertisement

KEEP READING!

Business Marketing

(MARKETING) SEO is more complex today than it ever has been, and whether doing it yourself or hiring a pro, you need to know...

Business Marketing

(MARKETING) Despite sexier emerging trends, the reality is that you can’t build out a successful online presence and marketing strategy without SEO.

Business News

(BUSINESS) Now when ordering online through the Google featured snippet, the tech giant is pushing 3rd party apps, costing owners cold-hard cash in fees.

Tech News

(TECH NEWS) Google rolled out a new core SEO algorithm update this month, but here’s why you shouldn’t freak out just yet.

The American Genius is a strong news voice in the entrepreneur and tech world, offering meaningful, concise insight into emerging technologies, the digital economy, best practices, and a shifting business culture. We refuse to publish fluff, and our readers rely on us for inspiring action. Copyright © 2005-2022, The American Genius, LLC.