IdeaOverTen
Linking Your Business to a World of Customers...
Get More Info
  
Corporate Office
Our Partners    |
article
Search Engine Optimization and Marketing Part 2Common problems
Introduction to common problems
There are a number of things that can impede a search engine from indexing your website correctly. The following section will address some of the most common problems.
Frames
Frames have always been problematic for search engines when they are crawling for information. Some studies have indicated that users also have difficulty performing simple navigational tasks when they are confronted with a website using frames.
Jakob Nielsen, an acknowledged usability expert, addresses problems with frames in an important article titled: Why Frames Suck (Most of The Time) [http://www.useit.com/alertbox/9612.html/]. One of the peskiest problems with frames occurs when users want to bookmark the Web page. When users save a Web page from a frames website into their browser’s bookmarks they are directed to the website’s homepage whenever they use the bookmark.
Likewise, search engines cannot "bookmark" a deep page (using frames) of your website to reference it within a search listing. Search engines will try to browse through the site and look for individual pages they can refer to, but users that follow search results can then get trapped without navigation! So, search engines prefer to avoid ranking frames-based sites altogether.
Only if there are some compensating factors (like substantial popularity) will a frames site perfom well with the search engines.
We recommend avoiding frames if you want good visibility in search engines. There is a work-around for frames but problems remain. A clever developer can use Javascript and re-fetch content pages when search engines index individual pages from a frames site. This solves part of the equation. Still, getting those pages indexed and ranking well is a bit of a problem.
Some search engines record content in an alternate container called NOFRAMES. NOFRAMES is meant for browsers that don't support frames technology. Text and links in NOFRAMES can help a search engine index a frameset document. However, a frameset document for every page of your site can be unwieldy.
The HTML code looks like this:
<noframes>Place content here</noframes>
Browsers – and search engines – not supporting frames, can display the content you place between the two NOFRAMES containers.
The suggested use of this space is to:
• Fill the section with the same text that you have on the visible Web page. Use the raw text without images – but do not use any text that is not visible to the users.
• Remember to add links in the NOFRAMES container too that can be followed by search engines. Use the same navigational links as you do on the page you show your visitors but remember to keep them as normal text-links.
Using these suggestions is not a guarantee that your Web pages will get indexed and rank well. Many search engines will not read the content nor follow links in the NOFRAMES-section.
Dynamic websites and Web pages
Dynamic Web pages usually contain content (e.g., images, text, form fields, etc.) that can change/move without the Web page being reloaded. These pages are often produced on-the-fly by server-side programs.
A traditional static website is made up of a number of individual files usually ending with .html. For example, index.html, products.html etc. Each page is a unique file and usually has unique “static” content.
On the other hand, a dynamic website very often only has one or a few files –-called “templates.” The template dictates how to present the visible content but contains no content itself. All the content is stored in a database. To show a page on a dynamic website your template code knows what to load from the database. Technically, that is done by adding “parameters” to the url while browsing page to page.
If the template is called: pages.asp and you would want to load the content with ID #54 the URL could end up looking something like this:
www.YourDomain.com/pages.asp?page_id=54
This seems fairly simple, but can get very complicated when many parameters are used to support the navigation of the underlying database. The same URL with a few more parameters may look like this:
www.YourDomain.com/pages.asp?page_id=54?manufcturer_id=acmeco?color_code=brown?style_code=modern?size_code=xxl?upc=12345678
This more complex URL (Web address) makes it difficult for search engines to crawl. There is simply no way for a search engine to ascertain which parameters identify a new page and which parameters are just a sort order of the content, a navigation setting or something else that does not justify indexing the page as a unique Web page.
There are other complicating factors related to having dynamic websites and websites built on content management systems indexed in search engines. This tutorial is unable to cover all of them.
There are a growing number of tools, techniques and services to help get dynamic websites indexed in the search engines. For further information on how to get your dynamic website indexed we recommend that you hire an experienced search engine optimization expert with expertise in these areas. Most often it takes a closer examination of your website to determine the best strategy.
Flash
Search engines do not read Flash files; therefore, content and links that you place in any of these formats will not be accessible to search engines.
You can read more about Flash, Macromedia and Java Applets in the Links and Navigation section.
Java Applets and other client side applications.
Search engines basically read regular text on a Web page. They do not read text or follow links in Java Applets or any other formats that require the user to have a program, run-time environment or plugin installed.
IP-delivery, agent-delivery and personalization
It has become increasingly popular for websites to implement varying forms of personalization for their visitors. These websites gather numerous pieces of information about individuals which can influence the way they will serve Web pages to individual users.
A simple example is a server-side program that checks what browser people are using, and serve up a specially tailored version for that browser. The same kind of program can also be used to check if people have specific plug-ins installed so that they get a version of the website they can read.
A more advanced use would be to check what country the user is coming from to serve a local version of a Web page. Some portals, search engines and cross-national websites have employed these techniques. There are many legitimate reasons to do so including business strategies, marketing, and legal issues with products only allowed in certain countries.
Growing sophistication in personalization is evidenced by some of the larger e-commerce websites that track a variety of individual information, like purchase history, buying behaviors, useage patterns, etc.
Cloaking
Similar techniques for tracking information about website visitors can be applied to the search engines that crawl a website. A Web server can detect a visit from a crawler and selectively serve different content. This technique is referred to as “cloaking”.
There may be some legitimate reasons to use cloaking. However, in most cases search engines do not like the use of cloaking and it is considered a spamming tactic. We recommend that you do not use cloaking unless you have a very good reason to do so, that you fully master the necessary techniques and that you understand the possible consequences.
Cookies
A cookie is a small text file that a web server can save in a users’ browser for later retrieval when that same user has a subsequent visit. It can be used to store log-in information or other preferences to make it easier or better for users to use a particular website.
Cookies are safe to use in the sense that they cannot be read or shared across different users or websites. If a cookie is set on a browser then the website that wrote it is the only website that can read it. Also, other users of a website cannot get to the information in my cookie – it can’t be transferred or shared.
Important to note is that Spiders do not accept cookies. Therefore, if your website uses cookies you have to make sure that all of the Web pages that you want to have indexed can be accessed without accepting the cookie.
Tip: You can turn off cookies in your own browser to test if your website can be accessed without them. Refer to your manual or help files of the browser you are using. This information can usually be found in “advanced options”.
Submit and index pages
Introduction
Search engines do not always include all Web pages from a website. Usually they will only include a sample of your pages – the ones they determine to be most valuable.
Some of your Web pages will be more important to have indexed than others. A product information page is far more important to have indexed than a contact form, as it is more likely someone will search for your products than your contact form texts.
Search engines do not always find the right pages to index by themselves. Sometimes they need a little help and guidance. In this section you will learn how to submit your Web pages to search engines to get them indexed.
Introduction to submission
Most search engines advertise a free feature to submit your website by adding your URL. Submitting a website to the search engines is not guarantee that it will be included in the Index. Unfortunately, a significant volume of undesirable websites are submitted in this fashion. Therefore, the search engines will generally evaluate a candidate website, before including it in their Index.
How to submit to search engines
Most search engines have a form you must fill out to submit your website. You will usually find the link in the bottom of the front page labeled “Add URL”.
Generally you should only submit your front page as the search engines will follow links from that page to the rest of your website.
However, if you have important sections of your website that are not directly accessible through the regular navigation you can also submit them. If you have a site map (a page with links to all the Web pages on your website) you can submit that too, to help the search engine spiders find all of your content.
TIP: The easiest way to get your website into the search engines is by having it included in the major directories, such as Yahoo and ODP. Many directories are used to provide input to the search engines. Most search engines will consider a website that is included in the major directories to be of higher value.
Do not over-submit
You should never submit a Web page to a search engine if it is already indexed. Some questionable SEOs will recommend that you resubmit your Web pages on a regular basis to keep your rankings. This is simply not true and can cause undesirable results.
You can buy software or services that make it easy for you to submit all of your Web pages to hundreds of search engines as often as you like. BUYER BEWARE, do not use such programs or services!
The reality is that there are only a handful of major search engines and directories that serve the vast majority of all search users – either directly through their own search portals or through distribution of their indexes to other portals.
Paid Inclusion (PFI) programs
Some of the search engines (e.g. Yahoo) and their Partners offer a commericial service to submit your Web pages. This is a time-tested service that offers some measureable benefits. You can pick exactly the pages from your website you want to have indexed, including deep, dynamic pages. As long as you have the direct address (URL) to the page, you can have it included through a Paid inclusion program. Generally, these programs offer some ancillary benefits, like regular refreshes, performance reporting, etc. The Yahoo Search Submit Basic program is such a service.
Excluding pages from getting indexed
Introduction to exclusion of Web pages
There are instances when you may not want a website crawled and indexed by the Search engines. This can be accomplished with one of the following methods:
• Robots.txt
• META robots code
Robots.txt
Robots.txt is a file that provides instructions for all robots that attempt to crawl your website. It is placed in the root directory of your Web server. The file uses a simple syntax to exclude specific types of user agents from parts of your website. You can either exclude specific search engine spiders or all spiders. To exclude all search engine spiders as well as all other robotic user agents from all directories on your web server, the robots.txt file should contain the following:
User-agent: *
Disallow: /
Note: This would disallow everything including your home page!
Learn more about how to write robots.txt files at SearchTools.com
http://www.searchtools.com/robots/robots-txt.html
TIP: We recommend that you validate your robots.txt file before uploading it. There is no way to predict how a search engine will interpret a robots.txt file with errors.
META robots
META robots is a container that you can place in the HEAD container of your HTML documents. You can use the META robots container if you don’t have access to your web server’s root directory or if you want to exclude single pages on your website. You can read more about how to use the META robots code at http://www.searchtools.com/robots/robots-meta.html
FAQ
Can I pay the search engines to improve my rankings?
No, major search engines like Yahoo do not sell improvement of rankings. Some of the search engines have inclusion programs in which you pay to get included in the index but you are not guaranteed a specific ranking.
Furthermore, you can buy sponsor links and keyword targeted banner ads in most major search engines and this way get a premium position for your favorite keywords. However, don’t confuse this with the regular search results as such listings are clearly labeled as ads – separated from the free search results.
Why do my Web pages not rank well?
Unfortunately there is not one simple answer to that question. The first place to start would be with a question: “For what keywords?”
Maybe you have been unrealistic about the keywords you are targeting. Maybe the competition for the keywords you are trying to rank well for is just too high. You should probably not expect to rank as number one for “Microsoft”, just because your online bookstore has a used book about Windows XP on stock, just as you should not expect to get top rankings for broad competitive terms such as “Travel” and “Mortgage”.
Be realistic about your goals!
If that’s not the problem then maybe you need to optimize your website further. This tutorial can serve as a good starting point for the basic things to learn. If and when you want to learn more we recommend that you visit us often as we will be posting more information on a regular basis and recommending other good sites for specific topic information.
I have paid for inclusion, why do I not rank better?
For those search engines that offer this type of program you are guaranteed a fast inclusion and refresh in their index. However, you are not guaranteed top rankings for your keywords. That alone is determined by the search engine’s relevancy algorithms.
How deep do search engines browse my website?
There is nothing to keep search engines from reading and indexing content on your website – even in very deep directory structures. There are no set rules on how deep search engines go. In some cases – with some websites - they might go deeper than in others.
Generally a good internal link-structure will help you get as many pages as possible indexed. If you have valuable pages buried deep down in sub-directories it might be worth linking directly to them from other pages higher in the directory structure, to “point out” to the search engines (as well as human visitors) how good these pages are.
However, we recommend that you do not put your most important content in deep sub-directories. You should place your most important pages in the root, or in a directory directly below the root, as in the example below.
http://www.YourCompany.com/index.html
http://www.YourCompany.com/about_us.html
http://www.YourCompany.com/news.html
http://www.YourCompany.com/products/index.html
http://www.YourCompany.com/services/index.html
http://www.YourCompany.com/shopping/index.html
What do the search engines consider spam?
There are unfortunately not any agreed standards on this topic. However, there are a few basic rules that you should follow to stay out of trouble.
Do not …
• repeat keywords over and over just to get better rankings
• target Web pages to keywords that are unrelated to the exact Web page
• use hidden text or links on your Web pages
• submit Web pages that are already indexed
• participate in link-exchange programs to boost Link popularity
• link to criminal, illegal or very offensive websites
We recommend that you look at the search engines ‘Help’ pages for more information. Many of them have published guidelines on the web for what they consider to be spam.
TIP: If you are ever in doubt that what you are about to do is spam, don’t do it! Instead, study and investigate the technique or method further. Do not always blindly trust what people tell you in public forums – use your own judgment and common sense.
If something sounds too good to be true it probably is. Easy tricks never work in the long run.
Can search engines index my dynamic website?
Most search engines will have difficulties interpreting a dynamic website. Dynamic websites generally have a much more complex structure, which can easily make the search engine spiders get lost or confused trying to read and browse through such websites. When a spider becomes confused it messes up the index, so most search engines tend to stay away from dynamic websites.
However, if you are using Pay For Inclusion or Trusted Feeds you will be able to tell the search engine exactly what URLs you want indexed and thereby making it safe for the search engines to spider those Web pages.
There are also a number of other solutions you can implement to improve the search engine friendliness of your dynamic website. It is however beyond the scope of this tutorial to get into the details of these methods …
You can learn more about dynamic websites in the Optimization section of this tutorial.
Can you tell me how the search engine algorithms work?
No, the algorithms are well kept secrets of the search engines. However, there are a lot of similarities in the way they work that we recommend you study further if you want to improve your rankings.
The number of daily visitors from search engines has dropped – why?
Search engines are constantly adding new pages to their indexes and changing the algorithms to improve search results. That is part of the job of maintaining a good search engine.
If the number of visitors you receive has dropped the reason could be that you do not have as good rankings as you did before or that less of your Web pages are included in the search engine indexes.
If there have been a lot of new Web pages added within your category it will be harder for you to rank well. In that case you may have to optimize your Web pages further to keep decent rankings for your most relevant keywords.
But the drop in visitors can also be due to changes in search trends. If you have a website that sells Christmas decoration you will most likely (hopefully) get a lot of search visitors in December. But you should not get surprised, or blame your rankings, if you experience a dramatic decrease in traffic by mid January and February.
Searches for ski-tours, summer vacations, camping, boating, swim suites and fur coats all have their own season. People search for what they need – when they need it.
Help! My site got dumped – what do I do?
First, make sure that your website is actually not in the index any more, what search engines it has been dropped from and how many pages were lost – if not all. It may be that your website was not excluded at all, but your rankings just changed a bit. If that is the case you will have to analyze why that is and make the necessary changes to your website or search engine optimization strategy.
If some, or all, of your pages have been excluded from a search engine there can be several reasons for that. Most commonly it is because your web server was down the last time the search engine spider was trying to browse your website. If that is the case then you should not worry – the next time the spider is out it will most likely pick up your site again and re-index your Web pages.
It is not very often that a website gets banned by search engines. Compared to the billions of Web pages that do get indexed, it’s only a small fraction of sites that search engines chose choose to permanently exclude from their indexes. These include extreme adult sites, hate, violence, racism - sites that most of us can agree search engines should not link to. In some cases search engines also ban websites that have been taking radical steps to manipulate their rankings.
If you have been optimizing your website to the limit and suspect you have been banned for crossing the line then you should contact the search engine for verification. In some cases you can get back in if you make the necessary changes to the website and promise not to mess up again.
Conclusion
SEO and SEM in general are not for the faint of heart. It is a process that takes knowledge and continued education-however, there are fundamental things that site owners can do or ask their webmaster to do.
Long term commitment, consistency and persistence with a touch of patience will get you to where you want to be.
We recommend using the correct tools for the job-Google and Yahoo are putting more emphasis on the structure of the url and other onsite and offsite factors relating to your shopping cart software or content management software. You need to use the latest search engine friendly shopping cart and website content management software that has these features already built in. You want to be better than your competitors, better optimized, using the best tools available to increase your return on investment.
Successful online businesses start their online Holiday marketing and software implementation plans in March. A good solid search engine marketing plan should start as early in the year as possible. If you need any help with your search engine optimization, search engine marketing or pay per click plan, or would like to upgrade to one of the best shopping cart software and website content management software platforms available, please contact one of our Internet Marketing Specialist at our toll free number, 866.864.2836, or email us and we will contact you at your convenience. Ed Kundahl Related Features
|
1994-2010© IdeaOverTen, LLC