Posted by blosint on August 14, 2008
Thursday, Oct. 4, 2007 , 5 – 7 p.m. Abstract
A conversation about the theory and practice of collective intelligence, with emphasis on Wikipedia, other instances of aggregated intellectual work and on recent innovative applications in business.
Speakers: Karim R. Lakhani - Thomas W. Malone – Alex (Sandy) Pentland
Posted in Collective Intelligence | Leave a Comment »
Posted by blosint on August 9, 2008
Making the Most of Open Private Sector Knowledge
Despite the fact that U.S. taxpayers have been paying more than $30 billion a year for a national intelligence and counterintelligence community to protect it from both traditional state-based threats and unconventional non-state actors, the events of 9-11 demonstrated our inability to detect and prevent bold asymmetric attacks that used our own airliners as precision missiles. Armed with new concepts, money, and suicidal pilots, Osama bin Laden has cost us at least $20 billion in damages.
The problem with spies is they only know secrets
Unfortunately, our spies and our satellites have lost touch with reality, for they collect less than 10% of the relevant information that we must digest to understand the complex multi-cultural world that is now capable of producing very wealthy and suicidal terrorists. We need a “new craft of intelligence” that can access and digest the broad historical, cultural, and current events knowledge that is available openly in over twenty-nine languages — by exploiting these open sources we can create open source intelligence, or OSINT, suitable for informing our public as well as our state and local authorities and our international partners, as to the threats to our nation.
What are open sources? Open sources go well beyond the Internet (3 billion pages of substance and rising) and premium online services (ten times what is on the Internet, with value-added) to include “gray literature” (limited edition publications including dissertations and local directories from around the world); specialized market research, private investigations, and other information broker services; and geo-spatial information services including commercial imagery and Russian military maps for all countries of the world (the U.S. does not have military maps for 90% of the world.) Open sources include experts on any subject, in any language. Shocking as it may seem, our intelligence community does not routinely strive to identify the top people in the world (not just Americans) on the various topics of concern — from terrorism to the environment to human trafficking to corruption to disease and public health — with the result that our analysis tends to be shallow and incestuous, relying on the same consultants again and again.
Where’s the action?
Why is this not obvious, and, more importantly, why is it not being acted upon? Although the bipartisan Aspin-Brown Commission on intelligence reform (reporting in March 1996) found that our intelligence community is “severely deficient” in its access to open sources of information, and also found that the various departments and agencies of government have failed to fulfill their responsibilities for collecting, processing, and analyzing open source information relevant to their missions, nothing has been done to implement the Commission’s recommendations for reform. The Commission specifically stated that OSINT should be a top priority for funding within our $30 billion a year intelligence budget, and that it should be a top priority for the attention of the Director of Central Intelligence.
- Digital History Project ($5M) to digitize and translate key Islamic, Chinese, and other foreign language historical, political, economic, cultural, social, and technical materials.
- Non-Governmental Organization Data Warehouse ($10M) to provide free storage and network access to the various international organizations whose “local knowledge” is vital to U.S. understanding.
- Global Coverage Distance Learning and Expert Forum Network ($10M) that will establish open ethical boards of review for all countries and topics, including distance learning and expert forums.
- Generic Open Source Training Initiative ($10M) to create both distance learning modules accessible by our state and local, armed forces and diplomatic personnel and our public.
- Public Information Sharing and Collaboration Toolkit ($10M) comprised of a generic set of industry standards and related tools for desktop level exploitation and analysis of digital foreign information.
- Regional Open Source Information Networks for Africa, Asia, Europe, and Latin America ($40M) , each with an open source collection and processing center in partnership with local governments who will provide regional language skills and access to gray literature and local experts.
- International Trade and Chamber of Commerce Network ($5M) to establish a web-based network maximizing access by U.S. businesses to foreign economic, regulatory and taxation information.
- Digital Marshall Plan ($20M) to provide direct assistance and subsidies to extend the Internet to every corner of the world (including rural areas in America) via wireless delivery means.
- University of the Republic & Global Outreach Program ($15M) that will bring together and educate “cohorts” of mid-career subject-matter experts from state and local governments, the federal government, and the business, academic and media communities, as well as foreign professionals.
Source: Time
Posted in Intelligence, OSINT | Leave a Comment »
Posted by blosint on August 9, 2008
We need to rethink the distinction between open sources and secrets. Too many policymakers and intelligence officers mistake secrecy for intelligence and assume that information covertly acquired is superior to that obtained openly. Yet, the distinction between overt and covert sources is less clear than such thinking suggests. Open sources often equal or surpass classified information in monitoring and analyzing such pressing problems as terrorism, proliferation, and counterintelligence. Slighting open source intelligence (OSINT) for secrets, obtained at far greater expense when available at all, is no way to run an intelligence community. Also, we must put to rest the notion that the private sector is the preferred OSINT agent. In the end, I would contend, the Intelligence Community (IC) needs to assign greater resources to open sources.
Judging from their words, too many policymakers and intelligence officers mistake secrecy for intelligence. President Nixon, for example, once belittled the Central Intelligence Agency (CIA) in words that capture the common mistake: “What use are they? They’ve got over 40,000 people over there reading newspapers.”[1] The president’s remarks, reflecting a persistent misperception, echo even now within the Intelligence Community. Recent CIA recruiting literature suggests to applicants: “You can be on the sidelines, reading about global events in the newspaper. Or you can be at the heart of world-shaping events . . . ” in the CIA. The brochure proposes a world divided between those who read newspapers “on the sidelines” and those with access to “intelligence” within the Agency. George Tenet, a recent director of central intelligence, was fond of defining the CIA to audiences both within and outside the Intelligence Community with a curt phrase: “We steal secrets.” Neither from reading the CIA’s recruiting brochure nor listening to its chief would one learn that the Agency includes an OSINT service that produces the lion’s share of its intelligence.[2]
Deeds also reflect the mistaken notion that secrets are all important. The Intelligence Community now includes large, well-funded agencies for overhead imagery intelligence (IMINT), signals intelligence (SIGINT), and human intelligence (HUMINT). By all accounts, most resources in the Intelligence Community go to such IMINT and SIGINT activities as developing reconnaissance satellites, collecting signals, and analyzing the take. OSINT, the stepchild of the Intelligence Community, lacks its own agency. The Foreign Broadcast Information Service (FBIS), the largest IC organization devoted to open sources, resides in the CIA. Other OSINT units are scattered within the Department of Defense and the State Department. Alone and in the aggregate, OSINT organizations have few people and little funding. Despite numerous surveys putting the contribution of open sources anywhere from 35 to 95 percent of the intelligence used in the government, OSINT’s share of the overall intelligence budget has been estimated at roughly 1 percent.[3]
Those who swear that secrets are the only true intelligence, in contrast to mere “information” found through open means, would do well to consider the indistinct character of the categories of overt and covert in intelligence. Information hidden behind walls of classification and special access programs may prove no more than equal in value to material available to the public.
Overt and covert streams of intelligence are by no means completely parallel and distinct; they often mingle and meander over one another’s territory. Covert reports at times are amalgams of press clippings. And newspaper editors, for their part, frequently publish stories based on accurate leaks of classified material. Examples abound. Veteran CIA case officer James Lilley learned early in his career how Chinese agents had “swindled” his office with supposedly inside information on Chinese developments that later proved to be “embroidered versions of articles from provincial Chinese newspapers.”[4] Similarly, European con men reportedly passed off Soviet newspaper articles as intelligence from behind the Iron Curtain to operatives of the CIA and the West German Gehlen Organization in the 1950s.[5] More recently, journalist Bill Gertz of the Washington Times has leaked classified information in his stories. His published photocopies of actual intelligence documents underscore how the overt and covert streams mingle.[6]
The more one considers the problem, the less distinct appears the distinction between open information and secrets. Let us consider the case of the B-29 bomber aircraft, whose use in the Second World War was reportedly classified. Samuel Halpern, an officer of the Office of Strategic Services (OSS), recalled how he once surprised an admiral by referring in his briefing to the B-29 Super Fortress bombers. When the admiral demanded to know how Halpern knew of the “highly classified” aircraft, the OSS officer replied that he had learned of the bomber through monitored Japanese radio broadcasts.[7] In short, what is classified to some is open information to others. This can lead to the absurd situation where foreigners learn details of US intelligence operations in their country through their national media, while the American public and Intelligence Community remain unaware of the overseas exposure. Perhaps “unilateral secret” would be the proper term for this phenomenon![8]
Not only are open sources at times indistinguishable from secrets, but OSINT often surpasses classified information in value for following and analyzing intelligence issues. By value, I am thinking in terms of speed, quantity, quality, clarity, ease of use, and cost.
Speed: When a crisis erupts in some distant part of the globe, in an area where established intelligence assets are thin, intelligence analysts and policymakers alike will often turn first to the television set and Internet.[9]
Quantity: There are far more bloggers, journalists, pundits, television reporters, and think-tankers in the world than there are case officers. While two or three of the latter may, with good agents, beat the legions of open reporters by their access to secrets, the odds are good that the composite bits of information assembled from the many can often approach, match, or even surpass the classified reporting of the few.
Quality: As noted above, duped intelligence officers at times produce reports based on newspaper clippings and agent fabrications. Such reports are inferior to open sources untainted by agent lies.
Clarity: An analyst or policymaker often finds even accurate HUMINT a problem. For example, when an officer of the CIA’s Directorate of Intelligence (DI), reads a report on a foreign leader based on “a source of unproven reliability,” or words to that effect, the dilemma is clear. Yet, the problem remains with a report from a “reliable source.” Who is that? The leader’s defense minister? The defense minister’s brother? The mistress of the defense minister’s brother’s cousin? The DI analyst will likely never know, for officers of the Directorate of Operations (DO) closely guard their sources and methods. This lack of clarity reportedly contributed, for example, to the Iraqi WMD debacle in 2002-03. The DO reportedly described a single source in various ways, which may have misled DI analysts into believing that they had a strong case built on multiple sources for the existence of Iraqi weapons of mass destruction.[10] With open information, sources are often unclear. With secrets, they almost always are.
Ease of use: Secrets, hidden behind classifications, compartments, and special access programs, are difficult to share with policymakers and even fellow intelligence officers. All officials may read OSINT.
Cost: A reconnaissance satellite, developed, launched, and maintained at a cost of billions of dollars, can provide images of a weapons factory’s roof or a submarine’s hull. A foreign magazine, with an annual subscription cost of $100, may include photographs of that factory’s floor or that submarine’s interior.
Beyond this general argument for open sources, I would maintain that OSINT often equals or surpasses secrets in addressing such intelligence challenges of our day as proliferation, terrorism, and counterintelligence. When a nation develops a weapon of mass destruction, for example, hundreds or even thousands of engineers, scientists, and manufacturers may join the program. Bureaucrats and traders may sell the weapons abroad. The OSINT target is immense. Engineers attend conferences; scientists publish scholarly articles; manufacturers build production lines; bureaucrats issue guidelines; and traders print brochures for prospective clients. Many paper trails wind around the world beyond whatever may surface in the media.
Before terrorists act, they issue warnings, religious leaders of their community deliver sermons, and political leaders plead their cause. Open sources, while they may not tell us where the next bomb will explode, do allow us to understand the terrorist agenda and act thereby to address grievances or launch competing campaigns for hearts and minds.
When foreigners seek to tap US technology abroad or on our soil in order to evade embargoes or leapfrog the R&D process at our expense, open sources may alert counterintelligence officers to their activities. For example, the National Counterintelligence Executive has published reports based on Korean media from both sides of the DMZ to bring to light North and South Korean efforts to acquire Western technology both abroad and in the United States.[11]
Arguing that we need to rethink the distinction between open information and secrets, which is more blurred than many think, and that OSINT is often more useful in addressing intelligence challenges, I would further maintain that Washington’s slighting of open sources is no way to run an intelligence community. In earmarking only one of every hundred dollars in the intelligence budget and assigning some similarly meager percentage of IC personnel to OSINT, our policymakers and intelligence executives are learning less than possible about our nation’s challenges while paying a higher price than necessary. DO officers without access to foreign media published uncounted numbers of bogus reports based on Chinese, Soviet, and other newspaper articles. We are also almost certainly spending large sums today to obtain covertly information similar or identical to that openly available. Rather than learn through HUMINT or SIGINT that a scientist of interest attended an international conference, for example, would it not be better simply to acquire, then print or report the contents of the conference proceedings? Open acquisition would likely be less expensive, and all policymakers and analysts would have access to the information.
Policymakers and intelligence executives would also do well to resist the siren call of those who argue that we should simply privatize OSINT. Private corporations are an excellent source of dictionaries, software, and contractors for our government. But private companies alone are no substitute for accountable, dedicated OSINT professionals in government offices.[12] Let us take the vital issue of translation as an example. Contractors— whether individuals, translation agencies, or research companies (the latter generally subcontracting with translation agencies or independent translators for the talent they lack in house)—today translate most of the foreign newspapers, scientific journals, and other open information for the Intelligence Community. They do so under the lead of cleared OSINT officers who, knowing both the requirements of the Intelligence Community and the mysteries of the foreign media, manage the translation flows to provide answers to intelligence questions. Staff officers are also available to translate priority items themselves on a crash basis when contractors are unavailable. Staff officers serve one master. Contractors, busy with a mix of assignments from corporate and government customers, often are unavailable when most needed.
Ideally, in my view, the government should develop its own sizeable cadre of translators. Yet, that would be much more expensive than the present system. Some would argue for the opposite path of privatizing OSINT, which would mean intelligence analysts, case officers, and others buying their translations directly from the private sector without OSINT officers to apply their general requests against the appropriate media for the right information or to edit, often heavily, contractor translations that are frequently of poor quality.
The above logic, which applies to media analysis, targeting, and other OSINT functions as well as to translation, suggests that the government should retain its OSINT capabilities. The Intelligence Community requires staff officers to lead the contractors. To use an analogy from history, private corporations may have supplied the aircraft, landing craft, and rifles for D-Day, but General Eisenhower, his military staff, and soldiers in uniform took the beaches at Normandy.
I have maintained that (1) secrets are not identical to intelligence; (2) the distinction between overt and covert sources is more blurred than commonly imagined; (3) open information often equals or surpasses classified material; (4) slighting OSINT is no way to run an intelligence community; and (5) the private sector is no substitute for the government in applying open sources to address today’s intelligence challenges. I can only conclude that Washington needs to assign greater resources to open sources. Whether we create a national OSINT center or leave FBIS and its counterparts right where they are is less important than the issue of dollars and people. Putting all the meager OSINT offices together in a single center, without added funding, would be analogous to a poor man combining several small bank accounts into one—he would still be poor. With greater resources, perhaps a doubling of OSINT spending to roughly 2 percent of the intelligence budget, we would see an impressive increase in intelligence available to all in government. It would even permit covert collectors to focus with greater precision on areas truly beyond the reach of open sources.
Footnotes
[1]Charles E. Lathrop, The Literary Spy: The Ultimate Source for Quotations on Espionage & Intelligence (New Haven: Yale University Press, 2004).
[2]I came across the brochure, Challenges for a Changing World, while on a college recruiting trip in 2003. For an example of DCI Tenet’s fondness for the phrase “We steal secrets,” see an account of his 1999 speech at his alma mater, Cardozo High School in Queens, New York, in Vernon Loeb’s “Back Channels” column, The Washington Post, 18 June 1999. In my use of the word intelligence, I mean vetted information from any source. For more on how I define intelligence, see my article “Sailing the Sea of OSINT in the Information Age,” Studies in Intelligence 48, no. 3 (2004): 50–51.
[3]The National Reconnaissance Office (NRO) develops reconnaissance satellites. The National Geospatial-Intelligence Agency (NGA) provides IMINT. The National Security Agency (NSA) handles SIGINT. The CIA devotes most of its resources, apart from its analytical directorate, to collecting HUMINT and developing technical support for such operations. FBIS, established in 1941 in the Federal Communications Commission, exists today as an OSINT service within a HUMINT organization. For OSINT’s contribution and its funding, see Joseph Markowitz, “The Open Source Role,” Horizons 1, 2 (Summer 1997): 1–2.
[4]James Lilley, with Jeffrey Lilley, China Hands: Nine Decades of Adventure, Espionage, and Diplomacy in Asia (New York: Public Affairs, 2004), 82–83.
[5]For reference to the European cons and an earlier account by Ambassador Lilley of the China debacle, see Evan Thomas, The Very Best Men: Four Who Dared: The Early Years of the CIA (New York: Simon & Schuster, 1995), 154.
[6]For an example of Gertz’s use of leaks, see the copies of classified documents in the appendix to his book Betrayal: How the Clinton Administration Undermined American Security (Washington, DC: Regnery, 1999). The appendix, titled “The Paper Trail,” covers pages 217–84 of a book running 291 pages. In other words, his classified appendix accounts for nearly one page of every four in the book.
[7]Samuel Halpern, “Remembering 109,” OSS Society (Fall 2001), 5; accessed at www.oss.org.
[8]While the US media has no habit of exposing CIA and DoD actions and organizations in East Asia, to pick a region, the same cannot be said of the Asian media. Pyongyang’s press, for example, publishes articles detailing the alleged flights each month of US reconnaissance aircraft against North Korea. Seoul reporters, too, engage in the sport of “spot the spook” by telling Korean readers that this or that office of the US Embassy is a CIA station. Even the media in Japan, our bedrock partner in the Pacific, repeatedly treat the public to stories of US officers under non-official cover and other sensitive topics.
[9]The image of CIA officers tuning in CNN to watch the breaching of the Berlin Wall underscores this point. See my article, “Sailing the Sea of OSINT in the Information Age,” Studies in Intelligence 48, no. 3 (2004): 47.
[10]Walter Pincus, “CIA Alters Policy After Iraq Lapses,” The Washington Post, 12 February 2004: A1.
[11]For a report from the National Counterintelligence Executive on Pyongyang mobilizing overseas Koreans to acquire foreign S&T information, see “Overseas Koreans Contributing Technical Literature to DPRK,” Counterintelligence News & Developments, September 2001. For a story about South Korean plans to build a biotechnology facility in California, see “South Korea: Biotech Consortium To Build Tech-Transfer Facility in San Diego,” News & Developments, January 2002. Both reports are available on line at www.ncix.gov.
[12]I do not maintain that contractors are a different species from intelligence officers. Indeed, they are often our retired colleagues who have traded staff badges for contractor badges.
Stephen C. Mercado is an analyst in the CIA Directorate of Science and Technology.
Posted in Intelligence, OSINT | Leave a Comment »
Posted by blosint on August 9, 2008
The Open Source movement has established over the last decade a new collaborative approach, uniquely adapted to the Internet, to developing high-quality informational products. Initially, its exclusive application was the development of software (GNU/Linux and Apache are among the most prominent projects), but increasingly we can observe this collaborative approach being applied to areas beyond the coding of software. One such area is the collaborative gathering and analysis of information, a practice we term “Open Source Intelligence”. In this article, we use three case studies – the nettime mailing list, the Wikipedia project and the NoLogo Web site – to show some the breadth of contexts and analyze the variety of socio-technical approaches that make up this emerging phenomenon.
Open Source Collaborative Principles
A Few Examples of Open Source Intelligence
The Future of OS-INT
Source: First Monday – Open Source Intelligence by Felix Stalder and Jesse Hirsh
In the world of secret services, Open Source Intelligence (OS-INT) means useful information gleaned from public sources, such as scientific articles, newspapers, phone books and price lists. We use the term differently. In the followings OS-INT means the application of collaborative principles developed by the Open Source Software movement [1] to the gathering and analysis of information. These principles include: peer review, reputation- rather than sanctions-based authority, the free sharing of products, and flexible levels of involvement and responsibility.
Like much on the Internet in general, including the Open Source Software movement, practice preceded theory also in the case of OS-INT.
Many of the Internet’s core technologies were created to facilitate free and easy information sharing among peers. This always included two-way and multicast communication so that information could not only be distributed efficiently, but also evaluated collaboratively.
E-mail lists – the most simple of all OS-INT platforms – have been around since the mid 1970s [2]. In the 1980s, bulletin boards, FidoNet and Usenet provided user-driven OS-INT platforms with more sophisticated and specialized functionality.
In the 1990s, many of these platforms were overshadowed by the emergence of the Word Wide Web. Tim Berners-Lee’s foundational work on Web standards was guided by a vision of peer collaboration among scientists distributed across the globe [3].
While OS-INT’s precedents reach back through the history of the Internet – and if one were to include peer-reviewed academic publishing, much beyond that – a series of recent events warrant that it be considered a distinct phenomenon that is slowly finding its own identity, maturing from a practice “in itself” to one “for itself.”
The culture of the Internet as a whole has been changing. The spirit of free sharing that characterized the early days is increasingly being challenged by commodity-oriented control structures which have traditionally dominated the content industries.
At this point, insatead of being the norm, free sharing of information is becoming the exception, in part because the regulatory landscape is changing. The extension of copyrights and increasingly harsh prosecution of violations are attempts to criminalize early Net culture in order to shore up the commodity model, which is encountering serious difficulties in the digital environment [4].
In other areas, years of experience with the rise and fall of “proto-OS-INT” forums has accumulated to become a kind of connective social-learning process. Uncounted e-mail lists went through boom and bust cycles, large numbers of newsgroups flourished and then fell apart due to pressures from anti-social behavior. Spam became a problem. Endless discussions raged about censorship imposed by forum moderators, controversial debates erupted about ownership of forums (is it the users or the providers?), difficulties were encountered when attempting to reach any binding consensus in fluctuating, loosely integrated groups.
The condensed outcome of these experiences is a realization that a sustainable, open and collaborative practice is difficult to achieve and that new specialized approaches must be developed in order to sustain the fine balance between openness and a healthy signal/noise ratio.
In other words, self-organization needs some help.
The emerging field of OSI-INT is made up of numerous, independent projects. Each of them, such as the Nettime e-mail list, Wikipedia and the NoLogo.org Web site which will be discussed in the following, has a distinct history that led them to develop different technical and social strategies, in order to realize some or all of the open source collaborative principles.
One of the early precedents of open source intelligence is the process of academic peer review. As academia established a long time ago, in the absence of fixed and absolute authorities, knowledge has to be established through the tentative process of consensus building. At the core of this process is peer review, the practice of peers evaluating each other’s work, rather than relying on external judges.
The specifics of the reviewing process are variable, depending on the discipline, but the basic principle is universal. Consensus cannot be imposed, it has to be reached. Dissenting voices cannot be silenced, except through the arduous process of social stigmatization.
Of course, not all peers are really equal, not all voices carry the same weight. The opinions of those people to whom high reputation has been assigned by their peers carry more weight. Since reputation must be accumulated over time, these authoritative voices tend to come from established members of the group. This gives the practice of peer review an inherently conservative tendency, particularly when access to the peer group is strictly policed, as it is in academia, where diplomas and appointments are necessary to enter the elite circle.
The point is that the authority held by some members of the group – which can, at times, distort the consensus-building process – is attributed to them by the group, therefore it cannot be maintained (easily) against the will of the other group members.
If we follow Max Weber’s definition that power is the ability to “impose one’s will upon the behavior of other persons,” [5] this significantly limits the degree to which established members can yield power. Eric Raymond had the same limitations in mind when he noted that open source projects are often run by “benevolent dictators” [6]. They are not benevolent because the people are somehow better, but because their leadership is based almost exclusively on their ability to convince others to follow. Thus the means of coercion are very limited. Hence, a dictator who is no longer benevolent, i.e. who alienates his or her followers, loses the ability to dictate.
The ability to coerce is limited, not only because authority is reputation-based, but also because the products that are built through a collaborative process are available to all members of the group. Resources do not accumulate with the elite. Therefore, abandoning the leader and developing the project in a different direction – known as “forking” in the Open Source Software movement – is relatively easy and always a threat to the established players. The free sharing of the products produced by the collaboration among all collaborators – both in their intermediary and final forms – ensures that that there are no “monopolies of knowledge” that would increase the possibility of coercion.
The free sharing of information has nothing to do with altruism or a specific anti-authoritarian social vision. It is motivated by the fact that in a complex collaborative process, it is effectively impossible to differentiate between the “raw material” that goes into a creative process and the “product” that comes out.
Even the greatest innovators stand on the shoulders of giants. All new creations are built on previous creations and provide inspiration for future ones. The ability to freely use and refine those previous creations increases the possibilities for future creativity. Lawrence Lessig calls this an “innovation commons,” and cites its existence as one of the major reasons why the Internet as a whole developed so rapidly and innovatively [7].
It is also important to note that an often overlooked characteristic of open source collaboration is the flexible degree of involvement in and responsibility for the process that can be accommodated. The hurdle to participating in a project is extremely low. Valuable contributions can be as small as a single, one-time effort – a bug report, a penetrating comment in a discussion.
Equally important, though, is the fact that contributions are not limited to just that. Many projects also have dedicated, full-time, often paid contributors who maintain core aspects of the system – such as maintainers of the kernel, or editors of a slash site.
Between these two extremes – one-time contribution and full-time dedication – all degrees of involvement are possible and useful. It is also easy to slide up or down the scale of commitment. Consequently, dedicated people assume responsibility when they invest time in the project, and lose it when they cease to be fully immersed.
Hierarchies are fluid and merit-based, however and whatever merit means to the peers. This also makes it difficult for established members to continue to hold onto their positions when they stop making valuable contributions. In volunteer organizations, this is often a major problem, as early contributors sometimes try to base their influence on old contributions, rather than letting the organizations change and develop.
None of these principles were “invented” by the Open Source Software movement. However, they were updated to work on the Internet and fused into a coherent whole in which each principle reinforces the other in a positive manner. The conservative tendencies of peer review are counter-balanced with relatively open access to the peer group: a major difference from academia, for instance.
Most importantly, the practice of Open Source has proved that these principles are a sound basis for the development of high-end content that can compete with the products produced by commodity-oriented control structures [8].
Nettime is an e-mail list founded in the summer of 1995 by a group of cultural producers and media activists during a meeting at the Venice Biennale. As its homepage states, the list focuses on “networked cultures, politics, and tactics” [9]. Its actual content is almost entirely driven by members’ submissions. It is a good example of true many-to-many communication.
Nettime calls its own practice “collaborative text filtering.” The filter is the list itself – or to be more precise, the cognitive capacities of the people on the list. The list consists of peers with equal ability – though not necessarily interest – to read and write. The practice of peer review takes place on the list and in real time.
The list serves as an early warning system for the community, a discussion board for forwarded texts as well as a sizeable amount of original writing, and, equally importantly, an alternative media channel. This last function became most prominent during the war against Yugoslavia, when many of members living in the region published their experiences of being on the receiving end of not-so-smart, not-so-precise bombs.
By March 2002, the number of subscribers had grown to 2,500. The number of people who read nettime posts, however, is higher than the number of subscribers to the list. Nettime maintains a public Web-based archive that is viewed extensively, and some of the subscriber addresses are lists themselves. Also, as a high-reputation list, many of the posts get forwarded by individual subscribers to more specialized lists (another kind of collaborative text filtering), in addition to being published in print and other electronic media.
The majority of subscribers come from Western Europe and North America, but the number of members from other regions is quite sizeable [10]. Over the years, autonomous lists have been spun off in other languages: Dutch, Romanian, Spanish/Portuguese, French and Mandarin. A Japanese list is currently in preparation. Despite its growth and diversity, nettime has retained a high degree of coherent culture and developed an original of technology-savvy, leftist media critique, stressing the importance of culture and social aspects of technology, as well as the importance of art, experimentation and hands-on involvement. This flexible coherence has been strengthened through a series of real-life projects, such as paper publications including a full-scale anthology [11], and a string of conferences and “nettime-meetings” in Europe during the 1990s.
Since its inception, the list has been running on majordomo, a then popular open source e-mail list package, and assorted hypermail and mhonarc based Web archives. Technically, the list has undergone little development. Initially, for almost three years, the list was open and unmoderated, reflecting the close-knit relationships of its small circle of subscribers and the still “clubby” atmosphere of netculture.
However, after spam and flame wars became rampant, and the deteriorating signal/noise ratio began to threaten the list’s viability, moderation was introduced. In majordomo, moderation means that all posts go into a queue and the moderators – called “list-owners,” an unfortunate terminology – decide which posts get put though to the list, and which are deleted.
This technological set-up makes the moderation process opaque and centralized. The many list members cannot see which posts have not been approved by the few moderators. Understandably, in the case of nettime, this has led to a great deal of discussion about censorship and “power grabbing” moderators. The discussion was particularly acrimonious in the case of traffic-heavy ASCII-art and spam-art that can either be seen as creative experimentation with the medium, or as destructive flooding of a discursive space. Deleting commercial spam, however, was universally favored.
In order to make the process of moderation more transparent, an additional list was introduced in February 2000, nettime-bold. This channel has been carrying all posts that go into the queue prior to moderators’ evaluation. Because this list is also archived on the Web, members can view for themselves the difference between what was sent to the list and what was approved by the moderators.
In addition to increasing the list’s transparency, having access to the entire feed of posts created the option for members to implement parallel but alternative moderation criteria. In practice, however, this has not yet occurred. Nevertheless, giving members this option has transformed the status of the moderators from being the exclusive decision makers to “trusted filters.” It has also provided the possibility for forking (i.e. the list splitting into two differently moderated forums).
Nettime is entirely run by volunteers. Time and resources are donated. The products of nettime are freely available to members and non-members alike. Even the paper publications are available in their entirety in the nettime archives [12]. Reflecting its history and also the diversity of its contributors and submissions, nettime has maintained the rule that “you own your own words.” Authors decide how to handle redistribution of their own texts, though to be frank, it is hard to have control over a text’s after-life once it has been distributed to 2,500 addresses and archived on the Web.
Despite its many advantages – ease of use, low technical requirements for participating, direct delivery of the messages into members’ inboxes – the format of the e-mail list is clearly limited when it comes to collaborative knowledge creation.
Moderation is essential once a list reaches a certain diversity and recognition, but the options for how to effect this moderation are highly constrained. Nettime’s solution – establishing an additional unmoderated channel – has not essentially changed the fact that there is a very strict hierarchy between moderators and subscribers. While involvement is flexible (ranging from lurkers to frequent contributors) the responsibility is inflexibly restricted to the two fixed social roles enabled by the software (subscriber and moderator). The additional channel has also not changed the binary moderation options: approval or deletion. The social capacities built into the e-mail list software remain relatively primitive, and so are the options for OS-INT projects using this platform.
Wikipedia is a spin-off of Nupedia. Nupedia – the name is a combination of GNU and encyclopedia – is a project to create an authoritative encyclopedia inspired, and morally supported, by Richard Stallman’s GNU project [13]. However, apart from being published under an open license, Nupedia’s structure is similar to the traditional editorial process. Experts write articles that are reviewed by a board of expert editors (with some public input via the “article in progress” section) before being finalized, approved, and published. Once published, the articles are finished. Given the extensive process, it’s not surprising that the project has been developing at a glacial pace.
Wikipedia was started in early 2001 as an attempt to create something similar – a free encyclopedia that would ultimately be able to compete with the Encyclopaedia Britannica – but it was developed via a very different, much more open process. The two projects are related but independent – Nupedia links to articles on Wikipedia if it has no entries for a keyword, and some people contribute to both projects, but most don’t.
The project’s technological platform is called Wikiweb, named after the Hawaiian word wikiwiki, which means fast [14]. The original software was written in 1994 but recently rewritten to better handle the rapidly growing size and volume of Wikipedia.
The Wiki platform incorporates one of Berners-Lee’s original concepts for the Web: to let people not only see the source code, but also freely edit the content of pages they view. In the footer of most Wikipages is the option to “Edit this page,” which gives the user access to a simple form that allows them to change the displayed page’s content. The changes become effective immediately, without being reviewed by a board or even the original author. Each page also has a “history” function that allows users to review the changes and, if necessary, revert to an older version of the page.
In this system, writing and editing are collective and cumulative. A reader who sees a mistake or omission in an article can immediately correct it or add the missing information. Following the open source peer-review maxim, formulated by Eric Raymond as “given enough eyeballs, all bugs are shallow,” this allows the project to grow not only in number of articles, but also in terms of the articles’ depth, which should improve over time through the collective input of knowledgeable readers.
Since the review and improvement process is public and ongoing, there is no difference between beta and release versions of the information (as there is in Nupedia). Texts continuously change. Peer-review becomes peer-editing, resulting in what Larry Sanger, one of the original project leaders, hailed as the “most promiscuous form of publishing.”
At least as far as its growth is concerned, the project has been very successful. It passed 1,000 pages around February 12, 2001, and 10,000 articles around September 7, 2001. In its first year of existence, over 20,000 encyclopedia entries were created – that’s a rate of over 1,500 articles per month. By the end of March 2002, the number of articles had grown to over 27,000.
The quality of the articles is a different matter and difficult to judge in a general manner. Casual searching brings up some articles that are in very good shape and many that aren’t. Of course, this is not surprising given the given the fact that the project is still very young. Many of the articles function more as invitations for input than as useful reference sources. For the moment, many texts have an “undergraduate” feel to them, which may be appropriate, since the project just finished its “first year.” However, it remains to be seen if the project will ever graduate.
Both Nupedia and Wikipedia have been supported by Jimbo Wales, CEO of the San Diego-based search engine company Bomis, who has donated server space and bandwidth to the project. The code-base was rewritten by a student at the University of Cologne, Germany, and for a bit more than one year, Larry Sanger held a full-time position (via Bomis) as editor-in-chief of Nupedia and chief organizer at Wikipedia. In January 2002, funding ran out and Larry resigned. He now contributes as a volunteer. There are currently close to 1,200 registered users, but since it’s possible to contribute anonymously, and quite a few people do, the actual number of contributors is most likely higher.
Wikipedia has not suffered from the resignation of its only paid contributor. It seems that it has reached, at least for the moment, the critical mass necessary to remain vibrant. Since anyone can read and write, the paid editor did not have any special status. His contributions were primarily cognitive, because he had more time than anyone else did to edit articles and write initial editing rules and FAQ files. His influence was entirely reputation-based. He could, and did, motivate people, but he could not force anyone to do anything against their will.
The products of this encyclopedia are freely available to anyone. The texts are published under the GNU Free Document license [15]. This states that the texts can be copied and modified for any purpose, as long as the original source is credited and the resulting text is published under the same license. Not only the individual texts are available, the entire project – including its platform – can be downloaded as a single file for mirroring, viewing offline, or any other use. Effectively, not even the system administrator can control the project.
The scale of people’s involvement in the project is highly flexible, ranging from the simple reader who corrects a minor mistake, to the author who maintains a lengthy entry, to the editor who continuously improves other people’s entries. These roles depend entirely on each contributor’s commitment, and are not pre-configured in the software. Everyone has the same editing capabilities.
So far, the project has suffered little from the kind of vandalism that one might expect to occur given its open editing capabilities. There are several reasons for this. On the one hand, authors and contributors who have put effort into creating an entry have a vested interest in maintaining and improving the resource, and due to the “change history” function, individual pages can be restored relatively easily. The latest version of the platform has an added feature that can send out alerts to people who request them whenever a specific page has been changed.
The other reason is that the project still has a “community” character to it, so there seems to be a certain shared feeling that it is a valuable resource and needs to be maintained properly. Finally, in case of read differences over content, it’s often easier to create a new entry rather than to fight over an existing one. This is one of the great advantages of having infinite space.
So far, self-regulation works quite well. It remains to be seen how long the current rate of growth can be sustained, and if it really translates into an improvement over the quality of the individual encyclopedia entries. So far, the prospects look good, but there are very few examples of the long-term dynamics of such open projects. Given the fact that its stated competitor, the Encyclopaedia Britannica, has been publishing since 1768, long term development is clearly essential to such a project.
NoLogo.org is perhaps the most prominent second-generation slash site. This makes it a good example of how the OS-INT experience, embodied by a specific code, is now at a stage where it can be replicated across different contexts with relative ease. NoLogo.org is based on the current, stable release of Slashcode, an open source software platform released under the GPL, and developed for and by the Slashdot community. Slashdot is the most well-known and obvious example of OS-INT, since it is one of the main news and discussion sites for the open source movement.
Of particular importance for OS-INT is the collaborative moderation process supported by the code. Users who contribute good stories or comments on stories are rewarded with “karma,” which is essentially a point system that enables people to build up their reputation. Once a user has accumulated a certain number of points, she can assume more responsibilities, and is even trusted to moderate other people’s comments.
Points do have a half-life however. If a user stops contributing, their privileges expire. Each comment can be assigned points by several different moderators, and the final grade (from -1 to +5) is an average of all the moderators’ judgments. A good contribution is one that receives high grades from multiple moderators. This creates a kind of double peer-review process. The first is the content of the discussion itself where people respond to one another, and the second is the unique ranking of each contribution.
This approach to moderation addresses very elegantly several problems that bedevil e-mail lists. First, the moderation process is collaborative. No individual moderator can impose his or her preferences. Second, moderation means ranking, rather than deleting. Even comments ranked -1 can still be read. Third, users set their preferences individually, rather than allowing a moderator to set them for everyone. Some might enjoy the strange worlds of -1 comments, whereas others might only want to read the select few that garnered +5 rankings. Finally, involvement is reputation- (i.e. karma-) based and flexible. Since moderation is collaborative, it’s possible to give out moderation privileges automatically. Moderators have very limited control over the system. As an additional layer of feedback, moderators who have accumulated even more points through consistently good work can “meta-moderate,” or rank the other moderators.
The social potential embodied in Slashcode was available when Naomi Klein’s January 2000 book No Logo: Taking Aim at the Brand Bullies became a sudden international best-seller. In the wake of the anti-globalization protests in Seattle in November 1999, and after, the book began to sell in the 10,000s and later 100,000s. She found herself caught in a clash of old and new media and facing a peculiar problem.
A book is a highly hierarchical and centralized form of communication – there is only one single author, and a very large number of readers. It is centralized because users form a relationship with the author, while typically remaining isolated from one another. This imbalance of the broadcast model is usually not a problem, since readers lack efficient feedback channels.
However, today many readers have e-mail and began to find Naomi’s e-mail address on the Web. She started receiving e-mails en masse, asking for comments, advice, and information. There was no way she could take all these e-mails seriously and respond to them properly. The imbalance between the needs of the audience and the capacities of the author were just too great, particularly since Naomi had no interest in styling herself as the leader or guru of the anti-globalization movement. (Of course that didn’t stop the mass media from doing so without her consent.) As she explains the idea behind the Nologo.org:
“Mostly, we wanted a place where readers and researchers interested in these issues could talk directly to one another, rather than going through me. We also wanted to challenge the absurd media perception that I am “the voice of the movement,” and instead provide a small glimpse of the range of campaigns, issues and organizations that make up this powerful activist network – powerful precisely because it insistently repels all attempts to force it into a traditional hierarchy” [16].
The book, which touched a nerve for many people, created a global, distributed y”communityy” of isolated readers. The book provided a focus, but nowhere to go except to the author. The Slashcode-based Web site provided a readily available platform for the readers to become visible to one another and break through the isolation created by the book.
The book and the OS-INT platform are complementary. The book is a momentary and personal solidification of a very fluid and heterogeneous movement. The coherent analysis that the traditional author can produce still has a lot of value. The OS-INT platform, on the other hand, is a reflection of the dynamic multiplicity of the movement, a way to give back something to the readers (and others) and a connective learning process. More than the book, Nologo.org fuses action with reflection.
Of course, all the problems that are traditionally associated with public forums are still there, dissent – at times vitriolic and destructive – is voiced, but the moderating system allows members of the group to deal with differences in opinion in ways that do not impede the vitality of the forum. The learning process of Slashdot, in terms of to how to deal with these issues, benefited NoLogo significantly. Within the first year, 3,000 users registered on the site which serves requests of some 1,500 individual visitors per day.
As a distinct practice, Open Source Intelligence is still quite young and faces a few challenges.
First, there is the issue of scale. Compared to traditional broadcast media, OS-INT projects are still very small (with the exception of slashdot, which has about half a million registered users) [17]. Since scale and exposure significantly affect the social dynamics, growth might not come easily for many projects.
Second, there is an issue of economics. Most OSI-INT projects are pure volunteer projects. Resources are donated. Wikipedia, for example, depends on Bomis Inc. for hardware and bandwidth. NoLogo.org is financed through royalties from book sales. Most OS-INT project have not yet produced any revenue to cover some of the inevitable costs. So far, they have quite successfully relied on donations (from sympathetic individuals, corporations or foundations), but prolonged crisis of the Internet economy does not necessarily make it easier to raise funds, which becomes more important as the projects grow in size and the infrastructure/bandwidth needs increase.
Compared to traditional production and publishing models, OS-INT projects take part to a large degree outside the traditional monetary economy. Contributors, by and large, are not motivated by immediate financial gain. However, not all resources can be secured without money, so new and creative models of financing such projects need to be found.
Slashdot, for example, which could rely for a long time on advertisement as a main revenue source, recently had to increase the size of banners in order to keep up with costs. However, it gave users the possibility to access the site without advertisement – in exchange for a small subscription fee.
It is likely that OSI-INT projects, from an economic point of view, will develop into a hybrid involving direct revenues (e.g. subscription, advertisement), goodwill donations and volunteer efforts. How these different elements will relate to one another will change from project to project. There is a lot of room – and need – for creative experiments.
Despite these challenges, there are good reasons to be optimistic about its future. First, the socio-technological learning process is deepening. The platforms and practices of OS-INT are becoming better understood, and consequently the hurdles for users as well as providers are getting lower.
On the users’ side, the experience of learning how to deal with participatory, rather than broadcast media is growing. Their distinct character is being developed, mastered and appreciated.
For providers, the learning experience of OS-INT is embedded in sophisticated, freely available GPL software. The start-up costs for new projects are minimal, and possibilities for adapting the platform to the idiosyncratic needs of each project are maximized. The resulting diversity, in turn, enriches the connective learning process.
Second, as the mass media converges into an ever smaller number of (cross-industrial) conglomerates, which relentlessly promote and control their multitude of media products, the need for alternative information channels rises, at least among people who invest time and cognitive energy into being critically informed.
Given the economics of advertisement-driven mass media, it is clear that the possibilities of an “alternative newspaper” is rather limited. OS-INT platforms, by distributing labor throughout the community, offer the possibility of reaching a wider audience without being subject to the same economic pressures that broadcast and print media face to deliver those audiences to advertisers, particularly considering the fact that paid subscriptions allow access to advertisement-free content.
The more homogenous the mainstream media becomes, the more room opens up for alternatives. And if these alternatives are to be viable, then they must not be limited to alternative content, but must also explore the structure of their production. This is the promise and potential of OS-INT.
The range of technologies are as wide as the range of communities, and a close relationship exists between the two. Technologies open and close possibilities in the same sense that social communities do. As Lawrence Lessig pointed out, what code is to the online world, architecture is to the physical world [18]. The way we live and the structures in which we live are deeply related. The culture of technology increasingly becomes the culture of our society. 
The authors are associated with some of the projects analyzed in this article. Felix Stalder is currently one of the moderators of the nettime mailing list (nettime-l). Jesse Hirsh is closely involved with Nologo.org.
An earlier version of this paper was presented at the conference “Critical Upgrade: Reality Check for Cyber Utopias” (Zagreb, 4-5 May 2002).
1. We use the term Open Source for its deliberate openness. Contrary to the more narrow term Free Software, Open Source seems better suited to label a general collaborative approach not limited to code. We acknowledge the historical and ideological differences between the two concepts, but we believe that they are of limited relevance in the context of the present argument.
2. http://www.zakon.org/robert/internet/timeline/#1970s, accessed 25 March 2002.
3. Tim Berners-Lee with Mark Fischetti, 1999. Weaving the Web: The Original Design and the Ultimate Destiny of the World Wide Web by its Inventor. New York: HarperCollins
4. Lawrence Lessig, 2001. The Future of Ideas: The Fate of the Commons in a Connected World. New York: Random House.
5. Max Weber, 1954. Max Weber on Law in Economy and Society. Translated by Talcott Parsons. Cambridge, Mass.: Harvard University Press
6. Eric Raymond, 2000. “Homesteading the Noosphere,” at http://www.tuxedo.org/~esr/writings/cathedral-bazaar/homesteading/x349.html.
8. Often, but not always, these principles are supported by licenses setting the legal parameters for what can, or cannot, be done with the informational products governed by them. For an overview of the different licenses, see the Open Source initiative’s list of more than 30 “approved licenses” at http://www.opensource.org/licenses.
10. http://amsterdam.nettime.org/Lists-Archives/nettime-l-0203/msg00080.html.
11. J. Bosma, P. Van Mourik Broekman, T. Byfield, M. Fuller, G. Lovink, D. McCarty, P. Schultz, F. Stalder, M. Wark, and F. Wilding (editors), 1999. Readme! Ascii Culture and the Revenge of Knowledge. New York: Autonomedia.
12. http://www.nettime.org/pub.html.
13. http://www.gnu.org/encyclopedia/free-encyclopedia.html.
14. http://www.wiki.org.
15. http://www.wikipedia.com/wiki/GNU+Free+Documentation+License.
16. http://www.nologo.org/letter.shtml.
17. OS-INT projects take place on the Internet hence they still cannot have the broad reach of traditional broadcast media.
18. Lawrence Lessig, 1999. Code and Other Laws of Cyberspace. New York: Basic Books.
Paper received 15 May 2002; revised version received 20 May 2002; accepted 20 May 2002.
Posted in Intelligence, OSINT | Leave a Comment »
Posted by blosint on August 9, 2008
Pittsburgh, PA— Vivisimo (www.vivisimo.com), the leading provider of intelligent clustering, search and metasearch software for organizing search results, today introduced a new service for Clusty to search and cluster political news, government information sites such as FirstGov, think tanks and large, publicly released government documents such as the 2006 Federal Budget and the 9/11 Report from a single site. Read the rest of this entry »
Posted in Collection, OSINT | Leave a Comment »
