Software
to help you create a robots.txt file
By
Sumantra Roy
Robogen
is a software for Windows 95, Windows 98 and Windows NT based computers
which can help you create robots.txt files without having to worry about the
correct syntax of a robots.txt file. It comes in two editions: the free
Limited edition and the Standard edition which costs $12.95. You can get
Robogen here.
In
this article, I'll tell how you can use Robogen to simplify the process of
creating a robots.txt file. This article assumes that you have read my
article on what a
robots.txt file is and how you can manually create it.
I
would recommend that you download the trial version of the Standard Edition.
The Limited edition does not support as many search engines as the Standard
Edition.
Once
you have downloaded Robogen and installed it in your hard drive, start the
program. Here are the steps that you should follow to create a robots.txt
file:
Step
1: Click on the Logon FTP Server button. In the dialog box that appears,
type in the address of the FTP server for your web site, make sure that the
"This server requires me to log in" option is checked, and then
type in the username and the password which you use to access the server in
the appropriate boxes.
Now,
depending on how your web host has set up the server, the files for your web
site may be present in the root directory of the server itself. Or, it may
be present in a sub-directory of the root directory. In the latter case,
click on the "Use specified root" option and type the relevant
sub-directory in the text box besides the option. If you don't know in which
sub-directory the files for your web site are present, ask your web host to
give it to you.
Done?
Now click on OK. After a while, Robogen will display the directories and
files which are present in your site in the right pane of the main Robogen
program window. If you find that Robogen has not displayed the files in your
site, it means that you did not enter the correct sub-directory in which the
files of your web site are present. In this case, ask you web host to give
the correct sub-directory to you.
Step
2: In the left pane of the main Robogen window, click on the small button
present to the right of the drop-down combo box at the top. In the menu that
appears, select "Popular Search Engines".
Step
3: Now, click on the drop-down combo box and choose the search engine that
you are targeting (say, AltaVista). In the right pane, select a page which
is NOT meant for AltaVista. Now click on the "<<<"
button. You will find that the file which you had selected in the right pane
is now present in the left pane. In this way, click on each of the pages
which are not meant for AltaVista and press the "<<<"
button.
Repeat
step 3 for each of the search engines that you are targeting. Just select
the search engine that you are targeting from the drop-down combo box,
select the pages which are not meant for the search engine one by one from
the right pane, and click on the "<<<" button after
clicking on each age. Repeat this process for all the search engines that
you are targeting.
Step
4: Upload the file to your server using the "Upload Robots.Txt"
button. If you want, you can also use the Save menu item from the File menu
to save a copy of the robots.txt file in your hard drive.
Sumantra
Roy is one of the most respected and recognized search
engine positioning specialists on the Internet. For more articles on search
engine placement, subscribe to his 1st Search Ranking Newsletter by sending
a blank email to mailto:1stSearchRanking.999.99@optinpro.com
