I'm familiar with YaCy, when I first heard of it I was very excited, but the execution and vision is not quite the same.
When you go to YaCy.net and try to do a search, first thing you see is all the stuff about the project and when you try to search (without running your instance, at least as of 2 mins ago, I don't get a single search result), the regular user doesn't really care about how it works, they just want to search, and DDG provides this very well, the service is never in your way.
Also, the world runs on this fake thing called money and it motivates people to leave their comfy beds and commute to work all over the planet. There's no real motivation for anybody to run YaCy, also it's slow, they made installers for Windows, Mac OSx, and desktops... you can't run a fast search engine like that, at least not today. This thing I'm talking about is meant for sysadmins and people with linux machines with great connections speeds to guarantee your network will be competitive to other search engines.
I'm also familiar with Nutch, we use it internally and we think it's actually quite a dangerous tool in the wrong hands ;). I see Nutch more as a search engine project, probably something that I'd use If I were to write a search engine starting today and then evolve and evolve to my own needs (awesome for me since it's written in java, I understand DDG is written in Perl)
(If you had a change to read an email about stickers from the FrostWire Team you should have an idea on how we're about to use Nutch to index free content on the internet)
Cost not a problem, growth is a problem though.
I bet with how well you're executing so far, and with your contacts (and also being a hacker-angel) costs are not that big an issue for your current infrastructure, in fact such a hot project should have VCs and Angels drewling over it, even more if they read this conversation. However, growth is a bitch, word of mouth happens but it takes time, and I think that getting big fast the old fashioned way does take influence, power and probably lots of marketing money. With the Shared Cake approach it's a win-win for everyone. The user gets a search engine that may work as fast or even faster than google once it becomes ubiquitous across all the backbones of the internet, Co-Hosters get to use their resources and get paid by just providing hardware and hopefully little to none maintenance since it's all managed remotely by DDGs engineers, and DDG gets to grow super fast in servers and word of mouth without spending a dime. I was having breakfast and thinking that it's like a weird hybrid of communism and capitalism, instead of everyone getting the same rights and no money in mind instead you get a lot of people to work for the same goal and eat the cake together, the bigger the cake, the bigger DDG gets, and DDG can also re-invest the money to grow specialized infrastructure to route and distribute the load fairly.
Another thought about cost, in order to compete with Google you will need faster and faster crawls. I just ran a search for "frostwire bitbucket" on google and DDG and google is already indexing the new frostwire bitbucket repository, I still don't see the expected search result on DDG, probably since it's slower to crawl, more machines would probably help on crawling more and more constantly.
How I imagine it happening
I'm sure there's thousands of mid-level hosting resellers that have plenty of empty racks and unused bandwidth and power out there, biting each other to get customers to buy their services. Once the word gets out, many of these fellas would jump right in and you'd probably have prime hosting power all over the internet backbones, whatever they can't sell on their own, they can just use that space themselves and help host DDG (If you have your own hosting company wouldn't it be cool to know you're hosting Google machines?, I think this could happen for DDG)
Say I'm SmallHost Inc, which sub-leases 20 closets from Level3 in Fort Lauderdale, out of which during the last 6 months I've not been able to find customers to rent the last 3 closets, and each closet holds about 48 1U slots, I could start by filling out one of those slowly with more and more machines. If I know that DDG's routing algorithm will send me all the traffic coming from the South Florida area, you betcha I'm gonna do as much noise as I can with local advertising say on Facebook (super targetted) to have people search on my instances. The same effect would repeat all over the place starting with small hosting companies.
If I saw that I'm actually making more money hosting DDG myself than re-selling the space, I'd probably start dumping those customers if they want to leave and then use all the space for myself. And DDG has to hire nobody, just smart a** coders to control the whole thing with amazingly good written software.
Then add to that your small time web dev, with re-seller hosting plans on Dreamhosts and the likes, if their VPSes pass the DDG test (enough speed, storage, minimum uptime, average load, etc) your instances would be ubiquitous. I think Google could not possibly beat having 10s of millions of servers (I'd love to see the hacks for DNS servers and routing for this thing, I'd even serve search one page of results from many different servers at the same just to be safe and maybe get more speed)
Loading User Profile...
Login with Facebook