index and search the web

[mose]

What are p2p alternatives for web index/search, to defeat corporate strategies on control ond conquer over web searching ? (namely, google, yahoo, microzoft, and concerns about censorship in china).

[AGNUcius]

Howdy mose,

I remember seeing somthing like this once, but can't find that same project, and don't remember the name...

But Google helped me again, and I found
GPU.sf.net and the search-engine plugin at http://gpu.sourceforge.net/search_engine.php "Using the Search Frontend, or the web interface of the Search Frontend, one can use the GPU network as a search engine.".

If we could get this running on as many machines as the protien research "Folding at Home" or what was probably the founding project in this genre = the SETI@Home http://SETIatHome.Berkeley.edu which now has 3 million members looking at the stars instead of fixing our planet.

[mose]

Superb, I'll have to test that. It's real truth that seti@home is such a waste ! GPU looks like the way to go, but this is a .exe thing, and don't seem to be free software.

thanks for pointer anyway, amigo

[AGNUcius]

Everything on SourceForge is Free Software (or at least Open Source). For future reference, I would never knowingly suggest non-free software.

The source is at sf.net/projects/gpu.

Let me know if you need help compiling.

[AGNUcius]

I found the project I was thinking of Grub.org. The most recent source is not yet available because Jimmy Wales is apparently getting it ready for his Wikia search engine. I hope he releases it under the GNU AGPL. With such a big name behind it, it may be worth watching this project.

Grub.org/html/news.php?op=show&id=1
"Wikia has aquired the Grub sourcecode from LookSmart. We will be posting the complete, current codebase as soon as possible, here on Grub.org. In the meantime, signup and stay tuned to developments regarding getting Grub going again. You can also check out the Wikia site at Search Wikia Labs."

[Michel Bauwens]

Hi,

You may want to check the tag http://del.icio.us/mbauwens/P2P-Search, where I monitor search-related developments.

Here is what I have in the section http://p2pfoundation.net/Category:Technology:

# For the most fully distributed Peer to Peer Computer Networks at present, please check out Tribler ;Peerple ; Wipeer
# Research into more fully distributed P2P systems for the future: Chord, CX Project, Farsite, Globe Project, Oceanstore, Pastry

There is an important open search initiative based in Amsterdam as well. (should be able to find it through google or in the bookmarking tag above, I also wrote about it in the blog)

Finally, there are people in our network knowledgeable on this: Olivier Auber, Gwendal Simon, "Olivier Auber" "Anh-Tuan GAI" . I'll invite them, but otherwise, send me an email and I'll forward your query.

[mose]

excellent. Hey olivier is also a someone I know (if it's the same guy that made the generateur poietique years ago), but I didn't have direct news from him for a while.

Thank you for those pointers, agnucius and michel, I'll dig further but this topic is not central for me (yet). I just suspect it will soon appear as vital, when google will have revealed its real face a little more.

mose

[AGNUcius]

Yahoo is now using the Free Hadoop.Apache.org which is 'distributed' in that it is designed to run on thousands of machines, though usually in the same building...

http://developer.yahoo.net/blog/archives/2008/02/hadoop_production_...

Some big news in the world of Hadoop comes out of Yahoo! today. We believe we're now running the world's largest Hadoop application, a 10,000 core Linux cluster producing data used by the Yahoo! Search Webmap.

[Sepp Hasslberger]

There is a p2p web indexer and distributed search index at

Yacy - http://www.yacy.net/en/

YaCy is a free search engine that anyone can use to build a search portal for their intranet or to help search the public internet. When contributing to the world-wide peer network, the scale of YaCy is limited only by the number of users in the world and can index billions of web pages. It is fully decentralized, all users of the search engine network are equal, the network does not store user search requests and it is not possible for anyone to censor the content of the shared index. We want to achieve freedom of information through a free, distributed web search which is powered by the world's users.