Filter Bubble Experiment II

yegg Staff yegg
Created: 4 years and 9 months ago
We're going to run another filter bubble test tomorrow. To participate, please join the testing list: https://duckduckgo.com/support (Level 6).

I will report back here with explicit directions tonight, which will involve taking screenshots at a certain time for a few searches.

This forum has been archived

Thank you all for the many comments, questions and suggestions. Particular thanks go to user x.15a2 for constantly monitoring, replying and helping so many users here. To continue these discussions, please head over to the DuckDuckGo subreddit.


anonymous
Signed up!
Will we get something through e-mail or something?
What's the time it's going to start? (Europe, Portugal) or I just can use a time converter.


Proud duck duck go user!
posted by [Old Forum rickysfunbox] • 4 years and 9 months ago Link
yegg
Yup, once we finalize the directions I'll send it to that list.
posted by yegg Staff4 years and 9 months ago Link
anonymous
Here's the email I sent to the list:
DuckDuckGo testing team,

As you may know, we ran a filter bubble experiment last month. For background on it and the filter bubble in general, check out this video we made: https://vimeo.com/duckduckgo/bubble

Tomorrow we're running a second filter bubble experiment (on Friday, October 19th), and we would love your participation. We plan to produce more quantitative results and have them published in a major publication. Here are the directions for the second experiment (which vary a bit from the first experiment, so please pay close attention):

1) Wait until 14:00 (2pm) Eastern Time, 11:00am Pacific Time, 18:00 (6pm) GMT on Friday, October 19th, which is in about 19 hours from the send time of this email. If you cannot take screen-shots at that time, please ignore this email. It's very important that we get results from around the same time.
If you're having trouble figuring out the exact time in your local timezome, you can use DuckDuckGo to help: https://duckduckgo.com/?q=2pm+et+in+london%2C+england


2) Sign into Google, which means you should see your name or username in the upper right of the screen when you go to http://google.com. If you instead see a button to sign in, then do so before proceeding. If you do not have a Google account, you can skip the next step.

Use http://google.com/ for these tests. That is, if you usually use a special Google URL, be it a country domain like google.co.uk or a special search like /ncr, please do not for the purposes of this test.


3) Search for these searches on Google in this order, and take screen-shots that include all of the links on the first page of the Google results. (Detailed instructions on how to take screen-shots effectively are at the bottom of this message.)

global warming
gay marriage
obamacare
affirmative action supreme court


4) Go into private browsing mode on your browser, which is called incognito mode when using Google Chrome. We are using private mode to get as de-personalized as possible. As such, you should NOT log in for this part. (Detailed instructions on how to get into this mode are at the bottom of this message.)


5) Re-search Google for the same four searches (now that you are in private browsing mode), and again take screen-shots that include the links on the first page of the Google results.

global warming
gay marriage
obamacare
affirmative action supreme court


6) Pick a random number between 1 and 1 million, which you can get from 
https://duckduckgo.com/?q=rand+1+1000000


7) Name your files like this

49324_c_1.jpg (for global warming signed in)
49324_c_2.jpg (for global warming private signed out)
49324_g_1.jpg (for gay marriage signed in)
49324_g_2.jpg (for gay marriage private signed out)
49324_o_1.jpg (for obamacare signed in)
49324_o_2.jpg (for obamacare private signed out)
49324_a_1.jpg (for affirmative action supreme court signed in)
49324_a_2.jpg (for affirmative action supreme court private signed out)

The first part of the file names is your random number from #6 (in this case 49324). The second part is the first letter of the search, e.g. o for obamacare. However, since there are two searches starting with g, please use c for global warming (since it is related to climate change). The third part (1 or 2) is 1 for signed in and 2 for private mode.

If you need a second screen-shot per search (to get all of the results) just put a b on the end like

49324_c_1.jpg
49324_c_1b.jpg

This naming scheme makes it much easier for us to process :)

If your screen-shot capture method doesn't use JPG (and therefore doesn't end the files with .jpg) that is OK. We just care about the first part of the file name. For example, PNG or PDF is fine.


8) Please email all the screen-shots as attachments in an email to: bubble@duckduckgo.com. In the email, please include the city and country that you're currently in, e.g. Valley Forge, Pennsylvania, United States or Ontario, Canada.


We realize this is a lot to ask, but know that you're helping break new ground here. As far as we know, this will be the first quantitative analysis of the filter bubble via analyzing results from searching at the same time.


ADDITIONAL NOTES

-- We will not share your personal information without your permission, but we cannot use doctored screen shots, so please do not alter the screen shots.

-- We need screen-shots at approximately the same time, so if you cannot start at that time, please do not participate in this experiment (early or late) -- there will, however, be other future experiments.

-- We do not have a political affiliation of any kind. These searches were chosen because they were associated with results that people had reported significantly different in the past (see figs 8 & 10 on 
http://dontbubble.us/) and because right now they are searched for very often due to the US presidential election.


-- For the signed in part, it would be ideal if you would disable any extensions that could mess with Google personalization. For this part we want to get as personalized results as possible. We realize our sample will be generally less-personalized because our users generally use us and are privacy-aware. As such, if you know anyone who uses Google a lot, we'd appreciate you involving them in this study.
When signed in, Google can personalize results based on your use of Google services (news, google+, youtube, web history, etc.) as well as your connected accounts (facebook, twitter, etc.). Check out this stuff here: https://www.google.com/preferences (search preferences, look for personal results), http://www.google.com/ads/preferences/ (ad preferences), https://history.google.com/history/ (web history), https://profiles.google.com/connectedaccounts (connected accounts), http://news.google.com/ (click on gears icon).


HOW TO TAKE SCREEN-SHOTS

-- If you can use Firefox, there is a quick and easy way to capture the whole page (including hidden areas) in a screen-shot. Here it is:
 A) Go to the page you want to screen-shot.
 B) Go to Tools -> Web Developer -> Developer Toolbar
 C) In the black box at the bottom, paste: screenshot 12345_c_1.jpg 0 true without quotes 
 D) Change 12345_c_1.jpg to the right name with your number and such. 
 E) Press enter and the pop-up tells you where the file was saved.

-- It works best if you maximize the browser to your screen so you capture the most results at once.

-- If you use Windows, the easiest way is to use the Snipping Tool that comes with Windows 7, which you can access by searching for it via the start menu/windows icon in the lower left. Check out this article for more specific instructions: http://www.wikihow.com/Take-a-Screenshot-in-Microsoft-Windows

-- If you use Windows and don't have the Snipping Tool, you can use the Print Screen button on your keyboard and copy and paste it into Paint. Again, the bottom of this article has instructions:
http://www.wikihow.com/Take-a-Screenshot-in-Microsoft-Windows

-- On Mac, the easiest way is to do Command-Shift-4, then select the area (and then automatically saves the screen shot to the desktop).

-- On some other operating system, you probably know what you're doing :)



HOW TO GO INTO PRIVATE BROWSING MODE

-- On Chrome, go to New Incognito Window

-- On Firefox, go to Start Private Browsing

-- On IE9, go to InPrivate Browser

-- On Safari, go to Private Browsing

-- For more specific instructions on each of these platforms, check out this article: 
https://megatechtoday.wordpress.com/2012/01/19/private-browsing-in-different-web-browsers/
posted by [Old Forum guest] • 4 years and 9 months ago Link
anonymous
I've got some questions.

Isn't 14:00 Eastern time equivalent to 19:00 GMT, and not 18:00? Also, are we supposed to be logged out of our Google accounts when taking the incognito screenshots?
posted by [Old Forum guest] • 4 years and 9 months ago Link
yegg
The time is right -- I triple checked it and just did it again because last time I messed this up :)

Eastern Time in the US is now 4hours behind GMT so 14:00 + 4 is 18:00 GMT. If anyone thinks I am wrong on this please follow-up ASAP!

And yes, for the private mode you should not log in. I will make this more clear in the directions,

Thanks for the questions!
posted by yegg Staff4 years and 9 months ago Link
anonymous
Does it matter what browser we use?  EG, I tend to use different browser for different things (work, personal, play), so they may actually all have different results.  Also, Opera doesn't really have an "incognito mode" -- that makes me think of Chrome...?  Or do you just mean logged out of any Google Accounts?
posted by [Old Forum guest] • 4 years and 9 months ago Link
yegg
For the signed in state, you should use the browser that has the most Google info about you.

For the private state, you should go into a private browsing mode that clears the cookies.
posted by yegg Staff4 years and 9 months ago Link
anonymous
I've joined the list after the above instructions were posted.
But I assume I shall participate anyway, is that right?

posted by [Old Forum guest] • 4 years and 9 months ago Link
yegg
Yes, anyone is encouraged to pariticpate!
posted by yegg Staff4 years and 9 months ago Link
anonymous
18:00 GMT will be 19:00 BST as we're on DST until 28/10/2012
posted by [Old Forum guest] • 4 years and 9 months ago Link
anonymous

Here with another question. (Sorry. :p) I'm in a foreign country, so I usually use Google.com/ncr which prevents redirection to the local version of Google. Should I simply type in "Google.com" and go with wherever that takes me, for the experiment?

posted by [Old Forum guest] • 4 years and 9 months ago Link
yegg
Np, thanks! Yes, simply type in google.com and use whatever comes up for the purposes of this experiment.
posted by yegg Staff4 years and 9 months ago Link
yegg
I just sent this to the test team list (clarifications based on initial feedback):

DuckDuckGo testing team,

The filter bubble experiment today (Fri, Oct 19th) is in about 5hr from the send time of this email, at 2pm Eastern Time. 

The latest (updated) and complete directions are now available at http://duck.co/topic/filter-bubble-experiment-ii

Below are a few clarifications to the original directions. Sorry about not getting the directions perfect the first time!


A) As you may have noticed, two of the searches start with g so using the first letter as the identifier in the file name doesn't quite work :). Instead, we will keep the naming as originally written, and just use c for global warming. Here's how it should look in the end:

49324_c_1.jpg (for global warming signed in)
49324_c_2.jpg (for global warming private signed out)
49324_g_1.jpg (for gay marriage signed in)
49324_g_2.jpg (for gay marriage private signed out)
49324_o_1.jpg (for obamacare signed in)
49324_o_2.jpg (for obamacare private signed out)
49324_a_1.jpg (for affirmative action supreme court signed in)
49324_a_2.jpg (for affirmative action supreme court private signed out)


B) Use http://google.com/ for these tests. That is, if you usually use a special Google URL, be it a country domain like google.co.uk or a special search like /ncr, please do not for the purposes of this test.


C) For the private browsing part, you should be signed out of Google. This happens by default when you enter that mode, so just don't sign in for that part. The idea here is to de-personalize Google as much as possible.


D) For the signed in part, it would be ideal if you would disable any extensions that could mess with Google personalization. For this part we want to get as personalized results as possible. We realize our sample will be generally less-personalized because our users generally use us and are privacy-aware. As such, if you know anyone who uses Google a lot, we'd appreciate you involving them in this study.

When signed in, Google can personalize results based on your use of Google services (news, google+, youtube, web history, etc.) as well as your connected accounts (facebook, twitter, etc.). Check out this stuff here: https://www.google.com/preferences (search preferences, look for personal results),http://www.google.com/ads/preferences/ (ad preferences),https://history.google.com/history/ (web history),https://profiles.google.com/connectedaccounts (connected accounts), http://news.google.com/ (click on gears icon).


E) If your screen-shot capture method doesn't use JPG (and therefore doesn't end the files with .jpg) that is OK. We just care about the first part of the file name. For example, PNG or PDF is fine.


F) If you can use Firefox, there is a quick and easy way to capture the whole page (including hidden areas) in a screen-shot. Here it is:

1) Go to the page you want to screen-shot.

2) Go to Tools -> Web Developer -> Developer Toolbar

3) In the black box at the bottom, paste: screenshot 12345_c_1.jpg 0 true without quotes 

4) Change 12345_c_1.jpg to the right name with your number and such.

5) Press enter and the pop-up tells you where the file was saved.


G) If you're having trouble figuring out the exact time in your local timezome, you can use DuckDuckGo to help:https://duckduckgo.com/?q=2pm+et+in+london%2C+england


Again, the latest (updated) and complete directions are now available at http://duck.co/topic/filter-bubble-experiment-ii -- I will answer any further questions there.
posted by yegg Staff4 years and 9 months ago Link
anonymous
Ah, looks like I'm gonna have to stay up late tonight. If possible I want my friends to take part in this too but it seems that everyone else is busy.
posted by [Old Forum guest] • 4 years and 9 months ago Link
anonymous
Seems like there is mistake in step F.3
My Firefox doesn't know screen-shot, but only screenshot :)
English (auto-detected) » Russian

posted by [Old Forum guest] • 4 years and 9 months ago Link
yegg
You are absolutely right! Changed in the instructions above.

Damn spell-check :(
posted by yegg Staff4 years and 9 months ago Link
anonymous
i dont wanna miss the experiment again, cause i cant acces to a pc at that time. more over, ive got a smart phone with 3g. its androir, so im not quite sure how to take screenshot nor logout from google. any suggestons????
posted by [Old Forum guest] • 4 years and 9 months ago Link
yegg
On Android, you can download Google Chrome and go into incognito mode on it for the private piece.

To take screenshots on later versions of Android you just hold down the power and volume down buttons for one sec.
posted by yegg Staff4 years and 9 months ago Link
dashesy
I signed up but I have received no email, nor any confirmation.
posted by dashesy 4 years and 9 months ago Link
yegg
Check your spam folder. Was it in there?

There are no confirmations though. And in any case, the directions are listed above so you can still participate!
posted by yegg Staff4 years and 9 months ago Link
anonymous
I signed up before you posted the clarifications.  And I also didn't get any mail.
I checked the spam folder, nothing there.  But since the address I used was gmail, maybe it's just Google showing it's evil side…

posted by [Old Forum guest] • 4 years and 9 months ago Link
yegg
This one appears to be my fault :(. I just sent out an email to new people on the list. Did you get it?
posted by yegg Staff4 years and 9 months ago Link
anonymous
I was about to write that still nothing, but it just came in. Thanks.
posted by [Old Forum guest] • 4 years and 9 months ago Link
anonymous
To anyone saying that the time is wrong, it isn't. It is right. Britain, right now, is in GMT+1. GMT stays the same in the whole year, it is Britain itself which moves from GMT (British Winter time) to GMT+1 (British Summer Time). 2pm Eastern Time, as it is currently, is 18:00 GMT, and 19:00 BST. So if you are in Britain, the experiment will take place at 19:00.
To confuse things further, GMT and UTC are one in the same.

It's confusing, I know. I was confused by it too when I moved from South Africa, but we can blame the government for that, just like everything else :)
posted by [Old Forum guest] • 4 years and 9 months ago Link
anonymous
To be exact, time in GMT is wrong because there is no such thing as GMT in modern use.
For time zone names in form GMT±XX it's exactly the same as UTC, but writing GMT is not strictly correct.

GMT was identical to solar time known today as UT1 and even though nobody uses solar time for civil timekeeping any more, definition of GMT was never changed.  Of course, for many purposes the difference (kept under 1s with leap seconds) is negligible, but using GMT is confusing.  Not to mention the historical ambiguity of noon and midnight in GMT…

Therefore one should always write UTC when they mean UTC.
Otherwise some of the British people are bound to get confused as seen above.

posted by [Old Forum guest] • 4 years and 9 months ago Link
yegg
My bad 

But duly noted.
posted by yegg Staff4 years and 9 months ago Link
anonymous
I just wanted to reiterate that google.com/ncr stands for No Country Redirect so it does not take the person to localized search.

Also, is it okay to do the (non logged in) searches by using the !g bang on duckduckgo?
posted by [Old Forum guest] • 4 years and 9 months ago Link
yegg
It would be easiest to just go to google.com and then search from whatever they take you to by default.
posted by yegg Staff4 years and 9 months ago Link
dashesy
Can I do it now time-wise? it is 9:15 Mountain Time here now
posted by dashesy 4 years and 9 months ago Link
yegg
No, please wait until 2pm ET -- it is important it all occurs at about the same time, since search results, e.g. news, can change quickly with time.
posted by yegg Staff4 years and 9 months ago Link
dashesy
Ok, sure
posted by dashesy 4 years and 9 months ago Link
anonymous
Does you care if we use SSL?  httpS://google.com instead of http://.... ?
posted by [Old Forum guest] • 4 years and 9 months ago Link
yegg
No, don't care about that.
posted by yegg Staff4 years and 9 months ago Link
anonymous
I created a different profile for Firefox in order to take the screenshots. Is that proper? Do you like it that sterile? Could it be that my add-ons on firefox and/or my history, cookies etc. are making my results more personalized, even if they're not from google?
posted by [Old Forum guest] • 4 years and 9 months ago Link
yegg
I think that is fine -- the idea is it is a fresh space with no cookies.
posted by yegg Staff4 years and 9 months ago Link
anonymous
Ok, searching in 4.
posted by [Old Forum guest] • 4 years and 9 months ago Link
anonymous
Screenshot Utility

It is a little late for this experiment (since it is starting in five minutes) but Gadwin Printscreen can really help out with the screenshots.  It can be configured to save with preconfigured file names, so you can just enter your random number and then append the c_1, o_2, and so on, as you take them.  It can also put a timestamp of the screen capture in the lower right corner.


Look for 'Gadwin PrintScreen 4.7' on the right-hand menu; this version is freeware (thanks Gadwin!) and is a tiny download (just a few meg).


posted by [Old Forum upstatescott] • 4 years and 9 months ago Link
anonymous
You say no alterations, but my screenshots are of the entire (two monitor) display!  Is cropping okay?
posted by [Old Forum guest] • 4 years and 9 months ago Link
yegg
Yes, that is OK! Just meant no alteration of the Google results themselves.
posted by yegg Staff4 years and 9 months ago Link
anonymous
Hope it helps!
posted by [Old Forum mack.stump] • 4 years and 9 months ago Link
anonymous
I had some interesting behaviour. I accidentally took a screenshot before hitting enter and then took another one after hitting enter. Some results are different between the two. Should I include the screenshot with instant or will it confuse more? :)
posted by [Old Forum guest] • 4 years and 9 months ago Link
yegg
Yes, would just send the one after enter please :)
posted by yegg Staff4 years and 9 months ago Link
anonymous
Done. :)

I definitely got different results for both searches, which I think is mostly due to localization of the signed-out results, as it got redirected to the local version of Google. I hate location-based personalization with a passion, more so than other types.
posted by [Old Forum guest] • 4 years and 9 months ago Link
anonymous
Hm, where do we send the files to??
posted by [Old Forum guest] • 4 years and 9 months ago Link
yegg
posted by yegg Staff4 years and 9 months ago Link
anonymous
Done, hope it helps because Google may have hard time personalizing for me and I use it only trough !g now for many months.
posted by [Old Forum guest] • 4 years and 9 months ago Link
dashesy
And that was me
posted by dashesy 4 years and 9 months ago Link
anonymous
It's a little strange how Google is actually more personalized when I'm in Firefox's private mode. :p But then again I've gone to great lengths to prevent Google from Dutchifying my normal search results.
posted by [Old Forum guest] • 4 years and 9 months ago Link
anonymous
I have the same feeling.  And not only because it displays in my native language when in private mode, but the results themselves seem better fit then when logged in.  But then again, these are topics which Google had low chances of getting right for me (unless there are some side channels I'm not aware of).
posted by [Old Forum guest] • 4 years and 9 months ago Link
anonymous
Want a tgz?  A zip?  What?
posted by [Old Forum guest] • 4 years and 9 months ago Link
yegg
Zip is best.
posted by yegg Staff4 years and 9 months ago Link
anonymous
Also done!

I have just signed on step 6 and did the screenshots. Hope it helps.
I got slighty different results, but see by yourself. I'm now sending by email.

Regards,
malevolent
posted by [Old Forum malevolent] • 4 years and 9 months ago Link
anonymous
Mail sent!!
Hope it helps :)

Niket
posted by [Old Forum guest] • 4 years and 9 months ago Link
anonymous
Hi there!

Sorry I just missed the experminent,
Please keep us posted about the results!
I would like to use some of it as teaching material (i<m teaching about rationality, information and democracy this semester!)
posted by [Old Forum guest] • 4 years and 9 months ago Link
zac
Thanks to everyone who participated! We're still going through the results --which means a lot of you got involved. We really appreciate you taking the time to help out : )

merriest of weekends,

-Zac

DuckDuckGo Community Manager
posted by zac Staff4 years and 9 months ago Link
yegg
Seconded!
posted by yegg Staff4 years and 9 months ago Link
anonymous
Oh dear - I've missed out on some fun
And then for some bashing ;-)
Use http://google.com/ for these tests. That is, if you usually use a special Google URL, be it a country domain like google.co.uk or a special search like /ncr, please do not for the purposes of this test.

Erhmm - google.com & google.com are not same.
Say what(?)
It depends on where you are, as reflected by Google's mapping of your IP to location.
  • If you're in the US then google.com is Google US.
  • If you're outside the US then google.com is Google Generic.
    (and no, it does unfortunately not imply that Google skips location awareness; "Google World" does not exist)
While using an inPrivate browsing, try these two different searches:
Yes, the searches are different - and the results are a bit so too.
Note: I'm in Europe (i.e. neither in A:Japan nor B:USA)
And if you want to try something fun, then try:
C: [ gay marriage ]
You'll probably see a lot of results from only few sites.
: It's one of the bad bugs currently being a plague to Google users, - know as the Diversity Bug.   (I should know, as I was the one who did the initial diagnosis; a public thread reflecting that is at https://productforums.google.com/d/topic/websearch/0IWfgSReofY/discussion )
   
OK, What I'm saying is: You absolutely need to be clear on what personalisation is, if you're going to evaluate such effects.
Some (e.g. Google) may say that physical location is a matter of regionalisation, not personalisation (I somewhat disagree on such perception).
Number of results per page, is a setting which I'm somewhat hesitant of calling personalisation (and I most certainly do not consider it a privacy issue) - but the settings can (and does as demonstrated) affect results in more ways that just the number of results per page.
 
Cutting the write a bit short, then:
My recommendation would be to make and have the guinea pigs click specific crafted URLs, instead of just tell them so search for e.g. [ global warming ].
One of the primary concerns with bubbling is with the cumulative effect of past observed personal behaviour and the use of such memory data.
In contrast: If you (like I) often search using English search words, within the set of English pages on WWW, then I could certainly argue that results are bubbled in a very major way; It is however not that kind of customisation & personalisation which we normally refer to as "Bubbling".  
 
Seriously: If you're going to compare signed-in Google users SERPs for differences, looking for Bubbling effects, - then you need to know what custom settings are in effect for each user's search.
posted by [Old Forum thomas-p..] • 4 years and 9 months ago Link
anonymous
Gosh! Can't believe I missed it!!!
It seemed so exciting! :D

I'd love to see the resulting ad...

--
Moderator
Dumb Search ad-free search engine
posted by [Old Forum sean-anderson] • 4 years and 9 months ago Link