allourideas

Here’s a video of my recent talk about wiki surveys at...

Mon, 21 May 2012 10:47:45 -0400

Here’s a video of my recent talk about wiki surveys at Collective Intelligence 2012. Also, check out the videos of all the other interesting talks.

allourideas in finnish

Tue, 08 May 2012 07:27:31 -0400

I am happy to announce that the voter-facing portions of the site have now been translated into Finnish. Thank you to volunteer translators Juho Salminen and Mirka Kiljala.

All Our Ideas is now available in eight languages other than English thanks to the great work of volunteers. If you would like to help translate the site into another language, please send me an email.

API v 3.0.1 released

Mon, 30 Apr 2012 10:10:49 -0400

We are happy to announce that version 3.0.1 of our API has been released. You can use the API to create your own pairwise comparison application.

Major improvements include:

Upgrading Rails to 2.3.14 for security fixes.
Switching to Bundler for managing dependencies. This should make it much easier for you to install and use the API.
Adding several new API calls for richer data analytics.

Also, going forward we’ve got a better system in place for future releases. We’re now utilizing semantic versioning for our release numbers. We are also documenting changes in CHANGELOG.md. Finally, our API documentation will include versioning information.

All API releases can be found here:
https://github.com/allourideas/pairwise-api/tags

If you make something cool with the API or want to add more functionality, please let us know.

Pacific Time (US) to UTC Time

Tue, 27 Mar 2012 16:17:48 -0400

Posted by Matt

We’ve recently converted all the times in our database from Pacific Time (US) to UTC time. This change should not affect how you use the site in any way, and it prevents weird things from happening during changes to and from Daylight Savings Time. This conversion was completed Wednesday, March 21, 2012 at 21:00 UTC.

first paper

Wed, 08 Feb 2012 21:33:04 -0500

We’ve just posted our first paper about the research behind allourideas.org. Comments and suggestions are welcome.

Wiki surveys: Open and quantifiable social data collection

Matthew J. Salganik and Karen E. C. Levy

Abstract: Research about attitudes and opinions is central to social science and relies on two common methodological approaches: surveys and interviews. While surveys enable the quantification of large amounts of information quickly and at a reasonable cost, they are routinely criticized for being “top-down” and rigid. In contrast, interviews allow unanticipated information to “bubble up” directly from respondents, but are slow, expensive, and difficult to quantify. Advances in computing technology now enable a hybrid approach that combines the quantifiability of a survey and the openness of an interview; we call this new class of data collection tools wiki surveys. Drawing on principles underlying successful information aggregation projects, such as Wikipedia, we propose three general criteria that wiki surveys should satisfy: they should be greedy, collaborative, and adaptive. We then present results from www.allourideas.org, a free and open-source website we created that enables groups all over the world to deploy wiki surveys. To date, about 1,500 wiki surveys have been created, and they have collected over 60,000 ideas and 2.5 million votes. We describe the methodological challenges involved in collecting and analyzing this type of data and present case studies of wiki surveys created by the New York City Mayor’s Office and the Organisation for Economic Co-operation and Development (OECD). We conclude with a discussion of limitations, many of which may be overcome with additional research.

Wikipedia Banner Challenge: Results

Fri, 20 Jan 2012 12:09:00 -0500

Congratulations to Wikipedia for a successful fundraiser. They raised 20 million dollars with donations from more than one million people. Now that the fundraiser is complete, we have archived the Wikipedia Banner Challenge; you can still vote and upload new banners, but those contributions will not be recorded. Below, I’ll present some analysis of the data and provide links to the raw data so that you can analyze it too.

Over the approximately two weeks that the site was active, there were about 100,000 votes cast and about 1,500 banners uploaded. Of these 1,500, I activated about 1,000. Basically, I activated every banner that was in English, did not have any grammatical errors, and was not obscene. In general, the number of banners uploaded per day closely matched the number of votes per day and participation was from all over the world.

Votes over time

Banners uploaded over time

Map of votes

Map of banner uploads

First, let’s look at some broad patterns in the results. We started the process with 300 banners based on Wikipedia’s previous research, and in that set of seed banners we included every possible combination of 12 images and 23 pieces of text. The heat-map below shows the scores of those 276 banners. Recall that the score ranges from 0 to 100 and represents the estimated chance that a banner will win against a randomly chosen banner. Here’s the graph (blue is lower score and red is higher score):

You can click on the heat-map to see a larger version. If the only thing that matters for a banner is the image, then we would see clear vertical bands of color. If the only thing that matters is the text, on the hand, we would see clear horizontal bands of color. In fact, we see something in between. A few of the clearest patterns are:

Across a range of messages, images of Jimmy and the Earth seem to do better than average and images of Susan seem to do worse than average.
Across a range of images, some text seems to do above average (“Imagine a world in which every person on the planet had free access to all human knowledge.” & “Let’s make a world in which every person on the planet has free access to all human knowledge.”) and some text seems to do worse than average (“Want to make the world a better place? What are you waiting for?” & “Let’s keep Wikipedia growing”).

These results are about the seed banners, but what about the more than 1,000 uploaded banners? Were any good banners uploaded? It seems that the answer is yes: the top 10 scoring banners were all uploaded by users. In other words, we seeded the Wikipedia Banner Challenge with 300 banners building on Wikipedia’s extensive earlier research, but not one of these 300 banners was in the top 10. Unfortunately, some of these uploaded banners had very few votes for or against them because they were uploaded close to the end of the process (this is a pattern we see frequently and something we are working on solving). Therefore, it is probably better to restrict our attention to banners that had more than 50 completed contests. In this case, 9 of the top 10 banners were uploaded by users, and here are the three with the top scores:

There is also the question of whether the scores from the Wikipedia Banner Challenge can predict click rate during the fundraiser. For example, you might wonder if these three high scoring banners had higher click rates and were able to raise more money than the banners Wikipedia are currently using. Unfortunately, we don’t know. The fundraiser reached its target, and thus ended, much faster than previous years so there was no time to run any banners from our site.

Even though we don’t know these banners would have done, we do have some data about the relationship between score and click rate during the fundraiser. Wikipedia had done some previous banner testing experiments, and they made some of this data publicly available. We included these previously tested banners in our set of seed banners, so we can see if the score from our site can correctly “predict” these experiments that have already happened. To summarize our findings, if anything the relationship seems to be opposite of expected: banners with lower score seem to have higher click rates.

The Wikipedia fundraising page reports results from three banner tests that we were able to replicate. In the first test run by Wikipedia, three banners with the picture of Jimmy Wales had different texts:

[Jimmy] Please read: A personal appeal from Wikipedia founder Jimmy Wales
[Jimmy] Please read: Advertising isn’t evil but it doesn’t belong on Wikipedia
[Jimmy] Advertising isn’t evil but it doesn’t belong on Wikipedia

In the second test, Jimmy was compared to Wikipedia contributors Susan and James:

[Susan] Please read: A personal appeal from an author of 549 Wikipedia articles
[Jimmy] Please read: A personal appeal from Wikipedia founder Jimmy Wales
[James] Please read: A personal appeal from Wikipedia editor Dr. James Heilman

In the third test by Wikipedia, Jimmy was compared to Wikipedia contributor Sarah.

[Sarah] Please read: A personal appeal from an author of 159 Wikipedia articles
[Jimmy] Please read: A personal appeal from Wikipedia founder Jimmy Wales

We included all of these banners in our site, so we can compare the score on our site to the click rates during the fundraiser. Since these outcomes are not measured on the same scale, we would not expect them to match exactly, but we would expect to see that the banners that had higher score also had higher click rate. Here’s the graph:

In fact, the click rate seems to be negatively related to score (r=-0.77). The numbers and colors also provide additional information. Each number represents one specific banner, so this plot makes it clear that one of the banners — “[Jimmy] Please read: A personal appeal from Wikipedia founder Jimmy Wales” — appeared in three separate tests. And, somewhat surprisingly, the click rate for this banner varied by almost a factor of 2, probably because the tests were run at different times (the exact time of all of the experiments is not publicly available as far I can tell). The color of each marker allows you to group results by test. For example, all the banners marked in green were run in the same test and therefore at the same time. By looking at the results for each test individually, we can see that the negative relationship between click rate and score holds true in two of the three experiments.

There are lots of reasons why the score on our site might not be predictive of the click rate on Wikipedia, and here are three that I think are most likely. First, I think that on our site, we were estimating the preferences of die-hard Wikipedians — our big sources of traffic were the Wikipedia blog and the thank you page that people saw after making a donation — and these Wikipedians might have different preferences from the more numerous casual Wikipedia users. Second, our site and the actual fundraiser are different contexts so it might be the case that a banner that does well when being directly compared to other banners might not do well when embedded at the top of a real Wikipedia page. Finally, I think that the specific banners that we have data about were pretty similar — 7 of the 9 were “Please read: A personal appeal from X” — so it might be that we are unable to detect the relationship between score and click rate for this narrow set of banners. Without more research, it will be hard to distinguish between these three and the many other possible explanations for this pattern.

However, carrying this negative relationship to its full conclusion, one might want to run the lowest scoring banners. Here they are (again with a minimum of 50 completed contests). I don’t think these would raise a lot of money, but who knows?

The undoubtedly complex relationship between score on our site and click rate leads to some natural questions about how Wikipedia should use these results. My recommendation would be to think about Wikipedia Banner Challenge (and All Our Ideas more generally) as a decision making guide, not a decision making machine. That is, All Our Ideas + Wisdom does better than just Wisdom or just All Our Ideas (a related point was made in this interesting blog post by Sharad Goel). The ability of wisdom to supplement All Our Ideas seems self-evident, but the ability of All Our Ideas to supplement wisdom comes from providing a decision maker access to a wide range of inputs that have been sufficiently filtered to be useful. Both components are key. A wide range of inputs helps expose the decision maker to new ideas — what Donald Rumsfeld might call “unknown unknowns” — and filtering, even imperfect filtering, helps make those inputs useful. Inputs without filtering are noise. For example, compare your ability to make a decision based on the Wikipedia Banner Challenge results page and Wikipedia’s effort to crowdsource banners in 2010. While the attempt from 2010 provides lots of information, it is hard to use for decision making because it has not been sufficiently filtered.

The analysis that we have done so far is just a first step; we know that there is much more to learn from the data so here it is:

banner file (one row for each banner)
votes file (one row for each vote)
non-votes file (one row for each non-vote, e.g., I can’t decide)

We are placing this data in the public domain under the CC0 License (for more on licensing for open data, see Molley (2011)). Here’s some documentation explaining exactly what is in these files. And, to help get you started, here’s the R code that I used. Please let us know if you do anything with the data; we’d love to see it.

Wikipedia Banner Challenge

Tue, 20 Dec 2011 14:28:52 -0500

As you can tell from the banners appearing all over Wikipedia, their fundraiser is in full swing. Despite Wikipedia’s importance as a global resource, only about one-in-a-thousand Wikipedia readers donate. One way to improve that would be better banners, and that’s why we are launching the Wikipedia Banner Challenge, a website to collect and prioritize banner ideas for Wikipedia. You can participate by voting on banners and suggesting new ones. It is quick, easy, and even a little fun.

The Wikipedia Banner Challenge builds on previous innovative efforts by Wikipedia to involve their community in the design of the fundraiser, especially during the 2010 fundraiser. In a continuation of that community-driven spirit, Wikipedia announced on their blog that they will be watching the results from the Wikipedia Banner Challenge closely and will use some of the most promising banners during the fundraiser. In other words, your banner could appear in front of Wikipedia users around the world.

The Wikipedia Banner Challenge is a customization of the core allourideas.org code that was completed in about a week by two awesome web developers: Chap Ambrose and Luke Baker. In addition to furthering our research and helping Wikipedia, we hope that this project will also encourage others to customize our open source software for their own purposes.

Wikipedia has always been an inspiration for this research, so we are very excited that they decided to use allourideas for such an important project. Wikipedias, if you want better banners, give us your ideas.

Here are links to more information about:
* The 2011 fundraiser
* The 2010 fundraiser
* Banner ideas from 2010 and banner test results
* Banner ideas from 2009

Governor Genro tops President Obama on Citizen Feedback: "The Governer Asks" vs. "Open for Questions"

Wed, 14 Dec 2011 22:38:00 -0500

Something neat is happening in Porto Alegre, Brazil today. Governor Tarso Genro, of the state of Rio Grande do Sul, is meeting with some of his constituents. Of course, that’s pretty normal; governors meet with constituents all the time. What is neat is how those constituents were selected. They are not the ones with the most money or influence, they are the ones with the best ideas.

These 50 constituents were selected to meet with Governor Genro through a process called Governador Pergunta (The Governor Asks). The process started when citizens suggested 1,300 ideas related to five different aspects of health care (e.g., access to care, family health). Next, the Governor’s office launched a major public outreach campaign to encourage residents to prioritize these ideas through an online voting process. To broaden participation, there were public events and even a “voting van” packed with Internet-connected computers that drove around the state. In just 30 days, Governador Pergunta collected 120,000 votes, and these votes were used to select the top 10 ideas in each of the five categories.

To readers in the US, Governador Pergunta might sound like President Obama’s Open for Questions, and the two did have the same admirable goal: to increase public participation in government. But, there were important differences in their implementation that lead me to conclude that Governor Genro’s Governador Pergunta topped President Obama’s Open for Questions.

The first big difference between the two projects was their voting mechanisms. Here’s what they looked like:

Governador Pergunta

Open for Questions

Open for Questions used single-column, approval voting. Visitors to the site could find the items that they wanted and then vote for them. Governador Pergunta used pairwise comparison, meaning that voters were presented with two options and are asked to choose between them. These mechanism may seem similar, but the Governador Pergunta voting system is better than Open for Questions in important ways. (Disclaimer: Of course, readers of this blog know that pairwise comparison is the voting mechanism for allourideas, and it turns out that that Governador Pergunta used the allourideas pairwise API. But, more on that later.)

One reason that the voting mechanism in Governador Pergunta is better is that voters made their decisions independently; they had no information about how others had voted. In Open for Questions, in contrast, voters made their decisions interdependently; items were sorted by popularity and this popularity was shown to voters (see screenshot above). This type of interdependent voting system, unfortunately, can lead to strong and unpredictable fads where some ideas get additional support mainly because they had been supported in the past. As I’ve shown in some earlier web-based experiments, the stronger the interdependence of decision-making, the weaker the relationship between underlying quality and ultimate success. In other words, interdependent voting systems are not good for finding the best ideas.

Further, the pairwise comparison voting mechanism used by Governador Pergunta is more manipulation resistant. Recall that in the approval voting system used in Open for Questions, the voters chose which items to consider. This feature makes it easy for a small group of people to rush to a single idea and push it to the top. This weakness was quickly discovered and exploited by the National Organization for the Reform of Marjuana Laws (NORML). In the midst of a financial crisis and national debate about health reform, many of the highest scoring items in Open for Questions were focused on the legalization of marijuana.

With a pairwise comparison voting mechanism, such as the one used by Governador Pergunta, it would have been much harder (but not impossible) for NORML, or any other group, to skew the results because a voter would have had to cast many, many votes before she would get a chance to vote for the idea she wanted to push to the top. Whatever you think about the fairness of marijuana laws in the US, having a system of public participation that is open to manipulation by a small group is clearly not ideal.

Finally, in addition to using a superior voting system, Governador Pergunta topped Open for Questions in another way: it was open-source. Just as black-box electronic voting machines threaten public confidence in elections, so to do black-box systems for other forms of public participation in democratic governance. Any effort to make government more open and transparent using processes that are not open and not transparent seems destined to fail. The source code for Governador Perguntaand the source code for the Pairwise API, used by used Governador Pergunta, are both open-source. The Governor’s team and I hope that other public officials will build on our work to develop even better ways of making government more open, transparent, and effective.

Note: I posted a very similar version of this article on the Freedom to Tinker blog.

great news: another grant from Google

Wed, 07 Dec 2011 22:40:14 -0500

Our project has just been awarded another Google Faculty Research Award. This grant will allow us to continue our research and keep improving allourideas.org. Thank you Google.

new code patch from the community

Wed, 23 Nov 2011 14:32:57 -0500

Thank you to Lucas D’Avila from Brazil for submitting an improvement to our code-base. Here are the details of the patch.

All Our Ideas is an open source project, and we welcome contributions from everyone in our community. [The image of Tux the Linux penguin is from Wikimedia Commons.]

server improvements

Wed, 02 Nov 2011 14:37:55 -0400

We took the site down today for about an hour for some server improvements. Everything went as planned, and we are now running Passenger 3. This should lead to improved performance, especially when we have multiple concurrent sessions.

Thank you to the good folks at Phusion who released Passenger 3 open-source.

occupy wall street

Wed, 19 Oct 2011 17:00:00 -0400

Members of the Occupy Wall Street movement have started using allourideas.org to help articulate goals. You can:

Participate
Read more on the Digital Democracy blog
Follow @OccupyVotes on twitter
Help them collect even more information by embedding a widget in your website or blog. Here’s the embed code:

Here’s more about how to customize the widget.

The Junior League and all our ideas

Mon, 10 Oct 2011 14:31:19 -0400

The Association of Junior Leagues International Inc. is a non-profit that improves communities through volunteer projects and builds the leadership skills of its 155,000 members. To provide additional training opportunities to members, the Junior League recently started a Wednesday Webinar series, and the first challenge for Becki Fleischer, the leader of the series, was to decide on speakers. Of course Becki could have sat down at her desk and created a list of people. But, these would have been the people that she thought were best, and what Becki really wanted was a list of the people that her members thought were best.

Becki had heard about allourideas.org from a Beth Kanter blog post about crowdsourcing, and she decided to give it a try. 24,000 votes and 60 ideas later, the choices were clear. Shar McBee, the top scoring suggestion, is already booked to give the webinar on April 18th, and Becki is in the process of inviting other members from the list.

allourideas in hebrew

Sun, 11 Sep 2011 14:34:29 -0400

I am happy to announce that the voter-facing portions of the site have now been translated into Hebrew. Thank you to the volunteer translator Uri Shwed.

All Our Ideas is now available in seven languages other than English thanks to the great work of volunteers. If you would like to help translate the site into another language, please send me an email.

contribute to all our ideas

Tue, 16 Aug 2011 11:05:09 -0400

All Our Ideas is an open-source project. That means that you can contribute to the code base to help the project grow. We’ve already had volunteer translators who have internationalized the site, but now we are asking for contributions from coders too. Here’s a list of open coding tasks that you can do. Have other ideas for improving the project? If so, let us know.

The photo of pins is by “Gmaxwell” and is available from Wikimedia Commons.

maps

Wed, 03 Aug 2011 13:33:55 -0400

Above is a map of the 2.5 million votes that we have received so far at allourideas.org. How does this map compare to other maps like the luminosity of the world at night, the map of Facebook users, or the map of Wikipedia edits? It turns out that they all look pretty similar.

The earth at night in the year 2000, source: NASA

Facebook users and the connections between them, source: Facebook

Wikipedia edits in a single day, source Erik Zechte from the Wikimedia Foundation

Catholic Relief Services and allourideas

Wed, 06 Jul 2011 08:56:19 -0400

Catholic Relief Services (CRS) is an NGO that provides assistance to poor and vulnerable people in nearly 100 countries. To provide these services, CRS employs more than 4,000 people in 150 offices around the world. Collecting feedback in a bottom-up, participatory way from such a large and diverse group is a huge challenge.

As part of an organizational development project, CRS used All Our Ideas to ask their employees what makes an excellent worker, more specifically, “Which phrase better describes an EXEMPLARY CRS staff member (any CRS staff member, anywhere)?”. To answer this question Stephen Moles from CRS launched three idea marketplaces—in English, French, and Spanish—that collected more than 20,000 votes and 100 new ideas from all over the organization. The results from the idea marketplaces were then refined and combined through a series of participatory focus groups. CRS has now completed the process, and here are the four key competencies that will be used to guide recruitment and career development globally:

Serves with Integrity: Manifests CRS mission, values, and guiding principles to help improve the lives of the poor, vulnerable, and voiceless.
Models Stewardship: Makes responsible and efficient use of time, talent, money, assets, and natural resources to achieve plans and goals.
Develops Constructive Relationships: Builds and maintains mutually beneficial relationships through solidarity.
Promotes Learning: Builds the capacity of self, staff and partners to continue learning and innovating to better fulfill our mission.

Even cooler than this list, I think, was the process used to create it.

democratic democracy index, part 2

Mon, 04 Jul 2011 12:04:14 -0400

Crowdsourced democracy rankings, darker countries were voted as more democratic (Source: Xavier Marquez)

In an earlier post, I descried an effort by Xavier Marquez, a political science professor in New Zealand, to produce a crowdsourced democracy index of countries. Since then, he has had many more votes, some from readers of this blog. With all of this new data, he compared the crowdsourced rankings to the Freedom House rankings, a widely used source. He found:

The correlation between Freedom House and the crowdsourced index is a fairly high 0.84 (which is about as high as the correlation between the combined Freedom House score and the Polity2 score for 2008: 0.87).

You can read more about the results—there is lots of cool analysis—at Xavier’s blog: http://abandonedfootnotes.blogspot.com/2011/05/crowdsourcing-democracy-index-update.html

This might be the kind of thing that would be cool to do in different languages.

customized urls and data

Thu, 30 Jun 2011 18:22:00 -0400

One of the nice things about All Our Ideas is that we don’t have accounts and log-ins for voters; they just visit the idea marketplace and start voting. This is great because it maximizes the amount of data collected and helps minimizes barriers to participation.

The absence of accounts can be a limitation, however, because it makes it hard to collect demographic information about voters. However, we’ve now taken steps to solve this problem. Using customized urls, you can now have better information about the characteristics of your voters.

If you spread the url of your idea marketplace through email—and this is one of the most effective ways that we have seen—you can now append information to the urls that you send out. For example, if your organization has offices in London, New York, and Tokyo, you can record which votes are coming from which offices by sending a different url to each office:

http://www.allourideas.org/[your_url]?info=london

http://www.allourideas.org/[your_url]?info=newyork

http://www.allourideas.org/[your_url]?info=tokyo

Basically anything that you put after “?info=” will be attached to each vote, and it will show up when you download your data. [Actually, it is a little more complicated than that. Anything between “?info=” and an ampersand (“&”) or then end of the URL (whichever comes first) will show up in the data file.]

You could even create a code that is specific to each person. For example, you could send me the following url:

http://www.allourideas.org/[your_url]?info=matt@allourideas.org

If your group has information about each of its members—for example, a university might have information about class year, gender, and major of all students—that information could now be linked to the votes. This linkage of additional information would allow you to estimate more subtle features of public opinion such as which ideas are most favored by women or which ideas have the biggest male/female difference in support. If you are worried about privacy, you can also do this matching anonymously by using a cryptographic hash. This hash would mean that you would send me an email with the url:

http://www.allourideas.org/[your_url]?info=fgksdfg8234kadfgSDF24t

Then the person providing you with the database would remove names and replace them with hashes (like fgksdfg8234kadfgSDF24t). This procedure allows you to match votes with demographics without easily identifying the voters. I am looking forward to seeing all the creative ways that this new capability get used.

The photo of footprints on the beach is by Pete and Linz.

all our ideas in italian

Mon, 20 Jun 2011 12:01:53 -0400

I am happy to announce that the voter-facing portions of the site have now been translated into Italian. Grazie to the volunteer translator Gianluca Torresani.

All Our Ideas is now available in six languages other than English thanks to the great work of volunteers. If you would like to help translate the site into another language, please send me an email.