Now you're ready to search. Signing up with a remote site search service and getting your site indexed is easy, no matter which service you choose. Most just ask you to register with your name, a password, and a URL for the robot to start at. The indexer will usually visit your site soon afterward, quickly making its way through all your links — so try to do this during a quiet time.
If you are serving very long HTML files and your server is relatively slow or if you get charged by the byte, consider a robot which sends an HTML HEAD or CONDITIONAL GET request. These ask the server to send just the meta information about an HTML file, which includes the date modified. Assuming your server does this thing right (which is not, unfortunately, a certainty), the robot will only ask for the contents of files which have been changed. This technique can lighten the load on your server and may save you some money.
As the robot follows links on your site, you can watch it work if you have access to a site monitor. Otherwise, check your site log, which will allow you to follow the progress of the robot and learn where it had problems.
If you have trouble, you may want to create one or more crawl pages or site maps or a table-of-contents page to make sure that each page is found by the robot (for instructions, see Sending Search-Engine Traffic to Your Site).
next page»
