Tools to scan web for DDG by volunteers.

[Old Forum guest] anonymous
Created: 5 years and 7 months ago
As you said on some page - you don't have a millions of dollars to own heavy machine to scan web for you search engine. So maybe could write some tools that everyone could install in computer and run to help duckduckgo parse the Internet? :)

This forum has been archived

Thank you all for the many comments, questions and suggestions. Particular thanks go to user x.15a2 for constantly monitoring, replying and helping so many users here. To continue these discussions, please head over to the DuckDuckGo subreddit.

It would be hard to establish the trust that way
posted by dashesy 5 years and 7 months ago Link
Distributed crawling is a great idea, but it would still require a central location to store the data collected; that would mean, maybe a few (hundred?) petabytes, with backups, ever-changing, etc. So, the processing and bandwidth would be mostly taken care of, but there's still that. 
As you may know, Google has quite a few massive buildings full of servers dedicated to this.
posted by crazedpsyc 5 years and 7 months ago Link