Too many trash sites in the results and no flagging/blocking/report option!

Working on it

Temporary fix
Not a problem
Analyzing
Need more info
Solved
Working on it

in DuckDuckGo Forum • 1 year ago

Hi,
I've noticed that every time you look for something like an album, a book or a movie (not necessarily a way to download them, I'm speaking about looking for them generically, for info, like if I type ), after a while malicious links will pop up, usually disguised as warez sites of some kind and containing catchy keywords like "megaupload", "torrent" or "download". It's easy to spot and avoid them if you're experienced enough and care about security (have a good browser with the right addons, use Linux, etc.) but, otherwise, they can easily fool some users. Their urls are usually structured like this:
http://www.sitename.domain/?q=your+search+query
From there you can see that they don't really link anywhere but to their fake search engine that probably brute-forced its way to DDG's results.
Will you do something about it? Google tells you that "this site may harm your computer" and automatically puts the malicious results in the backyard.
Also, I've been playing around with DDG for a couple of minutes and found out that words usually linked to warez are the ones that bring the most trash-results out, even if you don't look for anything concrete. For example:
Query for "pdf rapidshare" - pdf per se is just a file format and rapidshare is just a file hosting site. Yet, a lot of fake-warez sites will show up in the results.
Query for "music mediafire" - Hindi movies and cellphone-apps fake-warez sites show up. WTF?
And so on. Looking for matching occurrences in the results in found out that the names of popular file hosting services pop up on almost every trash site - probably because they are very popular these days among regular folk. And I don't even want to think about what would happen if I typed something more...spicy! I don't know about your stats but I assume a lot of people use DDG to search for porn (it's the Internet, after all) and I'm afraid the problem would be even bigger with those queries - especially considering that, sadly, people who look for porn on search engines are usually too young or too old to have a good understanding of security on the Internet. Right now I don't really want to delve into that kind of research but you guys should do it and see what kind of trash it brings out, since even looking for something as innocuous as "pdf rapishare" brings you malware-ridden trash.
Can you guys do something about it? A moderated report system, maybe? So that you get noticed when a site gets reported.
Also, I'm very glad to see you fixed the broken safesearch, it was very annoying when it removed the words you typed, even when they were legit!
DDG is improving a lot, I hope I can be at least a bit helpful.

Replies(3)

yegg13

Re: Too many trash sites in the results and no flagging/blocking/report option!

1 year ago

Absolutely. This is an ongoing issue, and I'd be more than happy to remove those sites and the networks behind them. Our main crawler is dedicated to removing this kind of stuff.

We are looking towards a reporting mechanism. The challenge is we don't want to clutter the UI.

In the mean time, I'd appreciate any listing of domains here so I can add them to the training set and blacklist.

Comment

Guest

Re: Too many trash sites in the results and no flagging/blocking/report option!

1 year ago

Hello and thanks for the reply, Mr. yegg13!
I'm here to help but It would require too much effort to list all the malware sites by hand, however what about filtering all those sites that just send your query to their own search engine, to start with? They don't offer any concrete results and their queries are usually fake.I wonder if it's possible for you to write a regex that filters out sites using "this+kind+of+url" or "this worse one" in a malicious way? A well-written Genetic Algorithm could also be a solution to improve automation when DDG will get bigger, I believe.

I see you use the SimilarSites API...you could use it on your blacklist to find out sites similar to those you already blocked. I'm testing it right now on a couple of bad sites, for example (I'm replacing Os with *s because I don't want to give them free SEO)

mediafireb*t(dot)com
Is for example a trash site and SimilarSites found a lot of matching sites, probably run by the same shady guys.
The problem could be bigger than I thought, as I'm noticing a lot of BlogSpot sites are basically mirrors for bigger trash sites.
Another trash site that seems to pop up a lot in the results are
m*x*lbums(dot)com (fake queries)
g*gab*twarez(dot)com (trash)
studiaw*b(dot)com (fake queries)
Basically 90% of the sites having "warez" in their URL

Also, more importantly, I've seen very, very suspicious-looking pornographic URLs and titles in the results, even with the SafeSearch turned On (which at least partially reduces the trash links) . Like, VERY suspicious-looking, if you know what I mean. I used to think nobody would actually click on those links but you should never underestimate the stupidity (or ignorance/inexperience) of people. Those links should definitely not appear in the results, whether they are fake (and they probably are) or legitimate or whether the filter is on or off. Right now I am not seeing one but I would definitely report it if I had the chance.

Finally, I understand what you guys are trying to do with the UI and it's awesome, however DDG definitely could use some help from its users and at least in this phase, it needs a report function. Possible solutions without cluttering the UI:

Report page on its own
Greasemonkey script (only those who really wanted to contribute would install it but it's good enough)
Semi-visible Warning sign next to "Similar Sites". Could probably be done changing the alpha value of the image, so you won't need to have more than one.

And these are just generic ideas, I bet you guys can be more creative.

Be sure to specify that the reports are just for spam/trash/extremely illegal sites, otherwise you'll find yourself flooded with reports made by RIAA lawyers to censor actually good sites like TPB, and that would be wrong and time-wasting.

That's all for now. Thanks for listening and I hope I didn't sound too demanding, I'm just trying to be a good user :)

Response title

Attachments

New to this Portal?