DDG won't search my website

[Old Forum bgmcoder] anonymous
Created: 5 years and 1 month ago
Salvete!  I have been looking for a solution to this for a long time.

I want to use the site search on my website, but Duck Duck Go won't search my site at all:
try this:
  1. site:sspxusa.org nerd

There are several pages there that have "nerd" in the html text. 
What can I do to get it to work?

This forum has been archived

Thank you all for the many comments, questions and suggestions. Particular thanks go to user x.15a2 for constantly monitoring, replying and helping so many users here. To continue these discussions, please head over to the DuckDuckGo subreddit.


crazedpsyc
Looks like your site is completely password-protected (at least at the root), which makes it nearly impossible to crawl. If you do have public resources, either (a) Link to them from the root and remove the password protection, or (b) Create a sitemap.xml and submit it with the search engines' webmaster tools (see https://en.wikipedia.org/wiki/Sitemaps#File_format)
posted by crazedpsyc 5 years and 1 month ago Link
anonymous
Password protected?  How is that?  You can browse it yourself:  http://sspxusa.org

What makes you see it as password protected?

We have another site at https://rch.sspxusa.org that IS password protected however.  Maybe DDG is reading that one for some reason?

I already have a web.sitemap page there, too.
posted by [Old Forum bgmcoder] • 5 years and 1 month ago Link
crazedpsyc
When I visit that first link (normally, in a web browser), I see a username/password prompt.
posted by crazedpsyc 5 years and 1 month ago Link
anonymous
Really?  I don't know how that is possible!  There is no code or anything to present any kind of login box, and in all the time I've had this site up, noone has ever had to log in.

I've browsed that website, and directed others to do so to, and never has anyone said there was a login box.

The second link has a login box because it is a sharepoint portal.

I am absolutely bewildered that you would say there is a login box.

Can you give me some details (browser, OS, I don't know... that is weird).
I simply cannot reproduce your claim.
posted by [Old Forum bgmcoder] • 5 years and 1 month ago Link
crazedpsyc
Archlinux+Firefox/Chromium/cURL, Android+Firefox/Default browser. Pretty wide variety to be related to the software ;-)

Maybe there's some sort of IP-based authentication? I don't know. Google's webmaster tools let you "fetch as GoogleBot" and see the result -- might want to check that out.
posted by crazedpsyc 5 years and 1 month ago Link
anonymous
Thanks for your time, Mr. Bizarre.

No, there's no authentication.  I've never heard of your problem before at all!  Really, I don't understand that.  Even in IIS, the authentication is marked for anonymous.

If the router were blocking your connection, you wouldn't get a login box.

Anyway, I only want to get duckduckgo to search my website so I can use the DDG search engine with it - and I don't really want to submit my sitemap.xml to google or bing. 

Now, about a year an a half ago - DDG was able to search my site with no problems.

That gives me a clue, at least - Maybe it is the router that is blocking the search engine...
posted by [Old Forum bgmcoder] • 5 years and 1 month ago Link
anonymous
Ah!  It gets a little stranger.

I see now that my site never turns up in ANY search results anywhere.  It cannot be searched by search engines at all.

Now, about a year and a half ago, this was not the case.
The only thing that I can think of is that I moved our site from a hosted server to our own server.

Could it be that there is some DNS issue?
Maybe somehow our router is preventing it?
posted by [Old Forum bgmcoder] • 5 years and 1 month ago Link
anonymous
  http://sspxusa.org definitely requires authentication. Using Firefox.
posted by [Old Forum guest] • 5 years and 1 month ago Link
anonymous
Okay, I've been looking at this problem extensively.

The login problem is intermittent - some people are asked to authenticate, others aren't.  In my web.config, the authentication is set to None - and to allow all users = anonymous authentication.

I've discovered that duckduckgo WILL find ONE pdf file using this:
site:sspxusa.org help

But there are LOTS of pages with "help" written on them.

But this doesn't turn up anything:
site:sspxusa.org nerd
It should find this page:
http://www.sspxusa.org/help/computerhelp.aspx

So what's with that?  Does anyone have a clue?

Any clues?
posted by [Old Forum bgmcoder] • 5 years and 1 month ago Link
x.15a2
I wish that I had more to add, but I get the same non-results from Y!, Bing and the other search engine and when I attempted to go to your URL, I was presented with a login credentials box (I tried using 4 different browsers, just for kicks). You may need to seek help from your web host to determine how to configure your site settings.

x-15a2
_______________________________________________________________
Death is conquered while you slumber...seven is the perfect number
posted by x.15a2 Community Leader5 years and 1 month ago Link
anonymous
Funny thing is that the pdf that gets discovered is not even there any longer.  That means the search engines are searching an old cache or something.
posted by [Old Forum bgmcoder] • 5 years and 1 month ago Link
anonymous
We fixed a DNS issue that may have been causing some folks to have to authenticate when access my site at http://sspxusa.org

Would one of you kind folks who had trouble before, please try the site again?

However, I still can't get DDG to search the site.
posted by [Old Forum guest] • 5 years and 1 month ago Link
crazedpsyc
Looking good from here! It may take around a week to re-index the site now, since search engines have now cached the previous errors. If you have direct access to the web server logs, I suggest skimming through them every so often looking for "GoogleBot," "BingBot," etc. to make sure there are no more errors in their requests.
posted by crazedpsyc 5 years and 1 month ago Link
anonymous
Well, I discovered that my router was blocking access to the googlebot. 
Fixed.

Now, Google works:
https://encrypted.google.com/search?hl=en&q=site:sspxusa.org nerd

DuckDuckGo gives less results:
https://duckduckgo.com/?q=site:sspxusa.org+nerd

but the DDG based searchbox on my own site doesn't work (you can see the query string at the top):
http://www.sspxusa.org/help/shell.aspx?search=nerd

posted by [Old Forum guest] • 5 years and 21 days ago Link