How do i choose which pages of my site to be crawled

<hidden> anonymous
Created: 4 years and 5 months ago • Updated: 4 years and 5 months ago
here is my site http://www.techhapa.com

This forum has been archived

Thank you all for the many comments, questions and suggestions. Particular thanks go to user x.15a2 for constantly monitoring, replying and helping so many users here. To continue these discussions, please head over to the DuckDuckGo subreddit.


x.15a2
DDG uses multiple sources for results and they may not have indexed your site yet. As soon as they do, it will flow through to DDG: https://duck.co/help/results/sources
posted by x.15a2 Community Leader4 years and 5 months ago Link
jbarrett
Any well behaved web crawler will respect a site's robots.txt file. If you wish to restrict crawler access to certain directories, this is a good place to start:

http://www.robotstxt.org/

If you want to have your site discovered by crawlers, get it linked from other sites.
posted by jbarrett Staff4 years and 5 months ago Link