Comments Please! The Future Of Blog Comments

11 Nov

Often times the comments sections of blogging sites suffer from a multiplicity of problems – they are overrun by spam or by repeated entries of the same or similar point, continue endlessly, and are generally overcrowded with grammatical and spelling mistakes. Comments sections that were once seen as an unmitigated good are now seen as something irrelevant at best, and a substantial distraction at worst. Here, I discuss a few ways we can re-engineer commenting systems to mitigate some of the problems in the extant models, and possibly add value to them.

Comments are generally displayed in a chronological or reverse chronological order, which implies that, firstly, the comments are not arranged in any particular order of relevance and, secondly, that users just need to repost their comments to position them in the most favorable spot – the top or the bottom of the comment heap.

One way to “fix” this problem is by having a user based rating system for comments. A variety of sites have implemented this feature to varying levels of success. The downside of using a rating system is that people don’t have to explain their vote for, or against, the comment. This occasionally leads to rating “spam”. The BBC circumvents this problem on its news forums by allowing users to browse comments either in a chronological order or in the order of reader’s recommendations.

Another way we can make comments more useful is by creating message board like commenting systems that separate comments under mini-sections or “topics”. One can envision topics like “factual problems in XYZ” or “readers suggested additional resources and links” that users can file their comments under. This kind of a system can help in two ways – by collating wisdom (analysis and information) around specific topical issues raised within the article, and by making it easier for users to navigate to the topic, or informational blurb, of their choice. This system can also be alternatively implemented by allowing users to tag portions of the article in place – much like a bibliographic system that adds a hyperlink to relevant portions of the story in comments.

The above two ways deal with ordering the comments but do nothing to address the problem of small irrelevant repetitive comments. These are often posted by the same user under one or multiple aliases. One way to address this issue would be to set a minimum word limit for comments. This will encourage users to put in a more considered response. Obviously, there is a danger of angering the user, leading to him/her adding a longer, more pointless comment or just giving up. On average, I believe that it will lead to an improvement in the quality of the comments. We may also want to consider developing algorithms that disallow repeated postings of same comments by a user.

The best way to realize the value of comments is to ask somebody – preferably the author of the article – to write a follow-up article that incorporates relevant comments. Ideally, the author will use this opportunity to acknowledge factual errors and analyze points raised in the comments. Hopefully, this follow-up piece will be able to solicit more comments, and the process would repeat again, helping to take discussion and analysis forward.

Another way to go about incorporating comments is to use a wiki-like system of comments to create a “counter article” or critique for each article. In fact, it would be wonderful to see a communally edited opinion piece that grows in stature as multiple views get presented, qualified, and edited. Wikipedia does implement something like this in the realm of information but to bring it to the realm of opinions would be interesting.

One key limitation of most current commenting systems on news and blog sites is that they only allow users to post textual responses. As blog and news publishing increasingly leverages multimedia capabilities of the web, commenting systems would need to be developed that allow users to post their response in any media. This will once again present a challenge in categorizing and analyzing relevant comments but I am sure novel methods, aside from tagging and rating, will eventually be developed to help with the same.

The few ideas that I have mentioned above are meant to be seen as a beginning to the discussion on this topic and yes, comments would be really appreciated!

Making Comments More Useful

10 Nov

Often times comments sections of blogging sites suffer from a multiplicity of problems – they are overrun by spam or by repeated entries of the same or similar point; continue endlessly and generally overcrowded with grammatical and spelling mistakes. Comments sections that were once seen as an unmitigated good are now seen as something irrelevant at best and a substantial distraction at worst. Here below I discuss a few ways we can re-engineer commenting systems so to mitigate some of the problems in the extant models, and possibly add value to them.

Comments are generally displayed in a chronological or reverse chronological order, which implies that firstly the comments are not arranged in any particular order of relevance and secondly that users just need to repost their comments to position them in the most favorable spot – the top or the bottom of the comment heap. One way to “fix” this problem is by using a user based rating system for comments. A variety of sites have implemented this feature to varying levels of success. The downside of using a rating system is that people don’t have to explain their vote ( Phillip Winn) for or against the comment leading occasionally to rating “spam”. BBC circumvents this problem on its news forums by allowing users to browse comments either in a chronological order or in the order of reader’s recommendations.

Another way we can make comments more useful is by creating message board like commenting systems that separate comments under mini-sections or “topics”. One can envision topics like “factual problems in XYZ” or “readers suggested additional resources and links” that users can file their comments under. This kind of a system can help in two ways – by collating wisdom (analysis and information) around specific topical issues raised within the article and by making it easier for users to navigate to the topic or informational blurb of their choice. This system can also be alternatively implemented by allowing users to tag portions of the article in place – much like a bibliographic system that hyperlinks relevant portions of the story to comments.

The above two ways deal with ordering the comments but do nothing to address the problem of small irrelevant repetitive comments, often times posted by the same user under one or multiple aliases. One way to address this issue would be to set a minimum word limit for comments. This will prod users to put in a more considered response. Obviously, there is a danger of angering the user leading to him/her adding a longer more pointless comment or just giving up but on an average, I believe that it will lead to an improvement in the quality of the comments. We may also want to consider coding in algorithms that disallow repeated postings of same comments by a user.

The best way to realize the value of comments is to ask somebody – preferably the author of the article- to write a follow-up article that incorporates relevant comments. Ideally, the author will use this opportunity to acknowledge factual errors and analyze points raised in the comments. Hopefully, then this follow up piece will be able to solicit more comments and the process repeated again helping take discussion and analysis forward.

Another way to go about incorporating comments is to use a wiki-like system of comments to create a “counter article” or critique for each article. In fact, it would be wonderful to see a communally edited opinion piece that grows in stature as multiple views get presented, qualified, and edited. Wikipedia does implement something like this in the realm of information but to bring it to the realm of opinions would be interesting.

One key limitation of most current commenting systems on news and blog sites is that they only allow users to post textual responses. As blog and news publishing increasingly leverages multimedia capabilities of the web, commenting systems would need to be developed that allow users to post their response in any media. This will once again present a challenge in categorizing and analyzing relevant comments but I am sure novel methods, aside from tagging and rating, will eventually be developed to help with the same.

The few ideas that I have mentioned above are meant to be seen as a beginning to the discussion on this topic and yes, comments would be really appreciated.

Playing with Numbers: Coming up with Objective Ratings of a Subjective Reality

1 Nov

Statistics are only meaningful to the extent that people can identify the phenomenon being measured, come up with a sensible measurement scales to measure primary or secondary observable phenomena and then interpret the results and display them in a lucid fashion. Often times that’s too much to ask and our world is now crumbling under the load of heaps of pointless incomprehensible statistics.

Increasingly, we are trying to understand the world around us via numbers. To this end, a host of research centers and organizations now annually release rankings on issues ranging from corruption to democracy to freedom of press. These rankings are then featured on prime real estate across media and used in homilies, laudatory notes and everything in-between; to buttress indefensible claims; and to bring a sense of “objectivity” to a media-saturated with rants of crazed morons.

“Lost in translation” are subtleties of data, methods of data collection and of analysis, and the caveats. What remains, often times, are savaged numbers that peddle whatever theory that you want them to hawk.

Understanding with numbers

The field of social science has been revolutionized in the recent decades with “positivist” approaches using statistics dominating the field. The rise in importance of “numbers” in research is not incidental for numbers provide powerful new ways, particularly statistics, to analyze concepts. Today numbers are used to understand everything from democracy to emotions. But how do we go about measuring things and assigning number to thing which we haven’t yet even been able to define, much less explain?

Let me narrow my focus to creation, interpretation and usage of rankings to substantiate the problems with using statistics.

More Specifically, Rankings

Reporters Sans Frontiers (Reporters without Borders and henceforth called RSF) came out with its annual “Worldwide Press Freedom Rankings”. The latest rankings place USA at 53, along with Botswana and Tonga, India at 105 while Jamaica and Liberia are ranked 26 and 83 respectively. The top-ranked South Asian country in the rankings is Bhutan at 98. Intuitively, the rankings don’t make any sense and a little digging into RSF’s methodology for compiling these rankings explains why.

Media’s fascination with rankings

The rankings received wide attention and made it to the front pages of countless newspapers. There is a reason why rankings are the choice nourishment of media starved of any “real information”. Numbers capture, or so is thought, a piece of “objective” information about the “reality”. Their usage is buoyed by the fact that rankings are seductively simple and easy to interpret. Everyone seems to intuitively know the difference between first and second. All that needs to be done is present the fluff, the requisite shock and horror and the article is written.

On to the problems with rankings or the “rank smell”

How can you measure objectively when you need a subjective criterion to come up with a scale?

This is something I raise earlier when I talk about how we can understand concepts like democracy or say emotions using numbers. Researchers do it by assigning number or related phenomenon – in the case of emotions it may be checking the heart rate or doing a brain scan or counting the number of times you use certain words, while in the case of democracy it can be how frequently the elites change, or how many people vote in the elections. But still numerous problems remain especially when we try to order these relatively hazily defined concepts. Say for example the elite turnover in US Congress has of late been fairly close to 2% and that doesn’t seem fairly democratic to me and how does it compare with somewhere like India, where elite turnover may be higher but where members of one family have held key positions in India politics since inception.

Relativity
To rank something means to determine the relative position of something. Rankings NEVER tell one about absolute position of something unless of course they are an incidental result of a score on a shared scale. For instance – RSF’s ranking of USA at 53 in the worldwide press freedom rankings doesn’t tell one whether USA’s press enjoys freedom over say a particular bare threshold below which a functioning press can’t be legitimately said to exist. A lot of people have misinterpreted India’s slide from 80, in 2002, to 105. They believe that it is a slide in absolute terms but the rankings only tell us of a slide in relative terms. There may be an argument to made that India is doing better than it is doing in 2002 in absolute terms but not in relative terms to say other countries. In other words, the press freedom in India may have improved since 2002 but as compared to other countries, India’s press is less free today.

The scale of things

To rank something, one has to use a common scale. Generally a scale, especially one measuring a complex concept like democracy, would be a composite scale of a variety of variables. One now needs to think of a couple of things. How does one weight the variables in the scale between time periods and between countries? For example how do you account for higher usage rates of media in one country (and possibly associated higher level of censorship) to say a country with low media usage and possibly lower total censorship? One may also argue that the media penetration is lower by deliberate action (as in limitation for foreign content owners to broadcast) or other factors (poverty). One must also tackle the problem of assigning “weight” to each facet.

Methodology

RSF ranking are based on a non-representative survey of pre-chosen experts. Hence it is more of a poorly conducted opinion poll rather than a scientific survey. Statistics gets its power of generalizability from the concept of randomization. RSF methodology is more akin to conducting a poll of television pundits on who will win the elections and I am fairly sure that the results would be more often wrong than right.

Secondly, questionnaire includes questions about topics like Internet censorship. No explicit mechanism has been detailed where we know that these scores are weighted based on say Internet penetration in each country. If no cases of Internet censorship was reported in Ghana, and it consequently gets a higher ranking as compared to a country Y whose press is freer but did report one case of Internet censorship – it implies the system is flawed. Let me give you another example. India has the largest number of newspapers in the world and there is a good chance that the total number of journalists harassed may well exceed that of Eritrea. It doesn’t automatically flow that Eritrean press is freer. One may need to account for not only the number of journalists (for more journalists per capita may mean a freer press) but also crime against journalists per capita. In the same vein we may need to account for countries which in general have a high crime rate and where journalists by pure chance, rather than say a government witch hunt, may have a higher chance of dying.

One also need to account for the fact that statistics on these crimes are hard to come by especially in poor countries with barebones media and there is a good chance that they are under-reported there.

On the positive side
Rankings do give one some estimate about the relative freedom of a country. Proximity to Saudi Arabia in the ranking does give us an idea about the relative media freedom.

A lot of the criticism lobbed against India’s low placement in the RSF rankings has been prompted by people’s perception of India as a functioning democracy with a relatively free press. What goes unmentioned are episodes like Tehelka and the one faced by Rajdeep Sardesai of recently. India’s press, especially in small towns, is constantly under pressure from the local politicians who monitor aggressively.

What can RSF do?
I would like to see a more detailed report on each country especially marking areas where India is lagging behind. Release more data. Aside from protecting sources, there should be no concerns regarding release of more data. Release it to the world so policymakers and citizens can better understand where improvements need be made.

Release a composite score index that is comparable across time rather than countries. There are far too many problems comparing countries. Controlling for major variables like economic growth etc., we can get a fairly good estimate of how things changed in the course of a set of years.

Conclusion

Whenever we do use numbers to understand concepts, we sacrifice something in what we understand or our conceptual understanding. Some numbers like demographics are relatively non-debatable. Even there debates have arisen in defining who are say Caucasians etc? More debatable are how numbers are used in say the realm of content analysis. What does it mean when a person says a particular word in a sentence? Does it mean that somebody who uses the word “evil” twice in describing Bush hates him twice as much as the person who only uses it once? The understanding and “counting” of words has largely been limited to simple linear additions. We haven’t yet tried understanding strength of words as an equation of countless variables or more importantly learned how to work with that much data so we use shortcuts in our understanding.

Numbers can give one a sense of false objectivity. The ways numbers are trimmed and chopped to support a particular point of view leave them meaningless, yet powerful.

The problems that I describe above are twin fold – errors in coming up with rankings and errors in reporting the rankings. In all, we need to be careful about the numbers we see and use. It doesn’t mean that we need to distrust all the statistics that we see and burrow our nose but we can do well by being careful and honest.

Closing Thought

According to UNECA, Ethiopia “counted 75 000 computers in 2001 and 367 000 television sets in 2000. Only 2.8 % of the total number of households in the country had access to television and approximately 18.4 % of people had a radio station in 1999 and 2000.” These numbers do inform. They talk about poverty. For the West, obsessed with issues of liberty and running from its own increasingly authoritarian regimes, press freedom is “the” issue. In the hustle, they miss some of the more important numbers coming from other countries that tell different stories.

Strategic perspectives on dealing with a nuclear North Korea and Iran

22 Oct

The following commentary is Chaste and can be seen as a follow-up to my article on the topic.

“I agree with most of your points. And I do understand the strategic value of combating naivete with naivete. After all, there is no strategic value in calling your opponent out on his underlying reasoning. He will deny the underlying motive and simply brand you a conspiracy theorist. On the other hand, one could argue that your opponent adopts naivete self-consciously is simply a front, and as a delaying tactic. Once a particular naive position is exposed, he will adopt another slightly less naive one and thus prolong his contention endlessly. For this, it may be useful to clearly address all underlying reasons up front. As mentioned, I am unable to evaluate the relative strategic merits of the two positions.

So, with that caveat clearly stated, here is what I believe to be the crux of the issue. I do not think that anyone seriously believes that the leaders of DPR Korea or of Iran will use nuclear weapons against another country. What the west objects to, is precisely the acquisition of deterrents by these target regimes. Such deterrents could embolden these target regimes to engage in non-nuclear anti-western activities. Recall for example that Pakistan started openly funding and training groups dedicated to fomenting violence in Indian Kashmir around the same time that it is thought to have acquired nuclear capability. This surely was no coincidence. Such Pakistani activity in the past was followed by Indian invasions. But with a nuclear Pakistan, India has been forced to accept the deaths of thousands of its security forces, an actual invasion in Kargil, and the deaths and displacement of tens of thousands of Kashmiri civilians with little more than a lot of sound and fury. Thus, what the west worries about is not that Iran may bomb Israel, but that it might be emboldened to be a more active supporter of groups like Hezbollah or Hamas.

What the west worries about is the removal of the threat of overwhelming force as a factor in their dealing with target nations like Iran or DPR Korea, and the possibility of having to rely primarily on diplomacy. Successful diplomatic outcomes generally require either diplomatic pressure which can deliver total victory in a zero-sum game, or normal diplomacy which delivers a compromise settlement. Diplomatic pressure requires the building of a multi-national near consensus, which in turn can dramatically alter the stakes consequent on the different choices made by the target nation.

The west faces problems in exerting diplomatic pressure on both counts. The first problem is the building up of a multi-national near consensus. No one outside the area cares much about DPR Korea (this may actually make a consensus more possible), and a majority of nations oppose the western agenda in the middle-east. The second problem lies in the difficulty of dramatically altering the stakes for the target nations. This will be inherently difficult in the case of an isolationist country like DPR Korea. In the case of the broader middle-east, this is difficult because of the middle east’s reserves of precious oil.

The West’s best shot with DPR Korea is that of an aggressive investment in the “sunshine” policy. This will make DPR Korea less isolationist, and allow multi-national players to dramatically alter the stakes consequent on its different choices. There is no appetite for such a policy in large part because DPR Korea does not appear to be particularly interested in anything beyond self-preservation. Therefore, the west can substantively ignore DPR Korea’s nuclear deterrent, beyond its opportunities for political posturing.

Iran is a different case since it does have interests beyond mere self-preservation: subversion of Israeli policy at least concerning Israeli occupation of the West Bank and Gaza, and its aspirations to being a regional power in an oil-rich region. With the difficulty in forging a multi-national consensus against Iran on these issues and the difficulty of dramatically altering the stakes for an oil-rich nation, the west will have no choice but to use normal diplomacy, which can only deliver a compromise settlement. But the west has no interest in compromising either on Israeli policies or in arriving at an agreement that respects Iran’s aspirations to be a regional power. Because of the West’s refusal of any diplomatic compromises and the difficulty of building up diplomatic pressure, the west is very keen on retaining overwhelming force as a factor in its dealings with Iran. As I have mentioned before, the last option will be neutralized if Iran acquires a nuclear deterrent.”

Why North Korea wants nukes? and How to live in a nuclear world?

21 Oct

The testing of a nuclear device by North Korea has drawn the ire of US, South Korea, and Japan, among others. Countless penny-a-quote pundits have come forth with their opinions as to why North Korea developed nuclear weapons, with most “analysis” limited to understanding North Korea’s development of nukes as an act of villainy by the autocratic “thug” ruling the “hermetic” kingdom. That the puerile minds of non-analysts bloated on clichéd Hollywood fare will offer such trash is expected but the relative lack of other explanations is stunning.

Why does North Korea want nuclear weapons? I argue that North Korea wants nuclear weapons for the same reason India and Pakistan wanted them, and that is as a deterrent against hostile action from other states. Walter Pincus, of The Washington Post, traces North Korea’s initial interest in nuclear weapons to the threats made by US presidents Harry S. Truman and Dwight D. Eisenhower to use nuclear weapons against North Korea during the Korean War.

"In 1950, when a reporter asked Truman whether he would use atomic bombs at a time when the war was going badly, the president said, " That includes every weapon we have."

Three years later, Eisenhower made a veiled threat, saying he would "remove all restraints in our use of weapons" if the North Korean government did not negotiate in good faith an ending to that bloody war.

In 1957, the United States placed nuclear-tipped Matador missiles in South Korea, to be followed in later years, under both Republican and Democratic administrations, by nuclear artillery, most of which was placed within miles of the demilitarized zone." N. Korean Nuclear Conflict Has Deep Roots (N. Korean Nuclear Conflict Has Deep Roots (WP) )

Aside from the initial nuclear threats, today over forty thousand American troops man the Korean peninsula and another thirty thousand stay on a base in Japan. Stack on to this the fact that Japan is widely acknowledged to have the capability to produce nuclear weapons at a short notice, and we can begin to understand North Korea’s motivations for developing nuclear weapons as a response to its threat perception.

One may argue that understanding the motivations behind North Korea’s acquisition of nuclear capability does not fundamentally change anything for either U.S.; South Korea or Japan, all of whom still see a nuclear-tipped North Korea as a threat. I believe differently – understanding North Korea’s actions in terms of its threat perception can inform our policy in multiple ways. Firstly, if you look at North Korea’s actions as a primarily defensive measure then one may argue that North Korea will probably only use nuclear weapons if attacked. This posit is most likely to hold true because U.S. owns an arsenal of over 10,000 nukes and any usage of nuclear weapons by North Korea will evoke a swift, debilitating response.

Secondly, the lessons learned should inform US diplomacy in the future – especially towards Iran, Cuba, and Iran. Threats from the US will only hasten these countries attempts to develop a nuclear arsenal.

Lastly, we all need to adjust to the idea of a nuclear-capable world. Nuclear weapons, as recent past has shown, are not particularly hard to develop or acquire – this is I say given three third-world countries, namely Pakistan, India, and North Korea, have been able to develop them. Aside from this, a slew of countries, including Israel and Japan already have nuclear weapons or can easily make them. In short, nuclear weapons technology will continue to proliferate, and there is very little we can do to stop this process.

This brings us to question of the repercussions of such a world. The fact remains that the probability that anyone will use a nuclear weapon is remote given that it will bring universal international castigation and a swift response from other powers. Secondly, given the rapid rise in ability of non-nuclear weapons like say MOAB or cluster bombs to afflict harm and destruction, and the comparatively less vocal condemnation on their use will bias countries towards using these “conventional” weapons. Thirdly, possession of nuclear weapons doesn’t equate to the capacity of reliably delivering them and even if one possesses the technology for delivery, the threat of universal condemnation and a swift response limits the probability of their use to nearly zero.

There are a few legitimate concerns about a nuclear-tipped world, and they have been dealt with below. Possession of nuclear weapons by a nation does limit U.S. choices against that nation, but the concern is largely theoretical for any use of nuclear weapons will result in a very strong response from the US. The second concern is about the ability of nuclear weapons to annihilate civilization. This concern stems from our understanding of the severity of the nuclear threat from cold war days when a nuclear war between the U.S. and the Soviet Union could have produced complete annihilation.

The scenario today is a bit different, and war between the U.S. and Russia, the only other power capable of delivering a similar nuclear response, is remote. Of course, conditions can change, but it still seems unlikely that we will reach such a scenario. Another facet that has garnered a lot of attention is the threat of terrorists using dirty nuclear bombs. There are two parts to the issue – one is state-sponsored terrorism which will be dealt in much the same way as response to conventional attack, and the second is threat of terrorists acquiring nuclear weapons from stockpiles of nations. This second threat must be dealt with the US trying to provide infrastructure and monetary assistance to countries to help them secure their stockpiles of nuclear material.

In all, we can take two things away from this discussion – the threat emanating from nuclear proliferation is greatly exaggerated, and that clichéd panic button responses of putting blanket sanctions against nations are unlikely to work.

Muslim Issues, Humanitarian Issues

4 Aug

The latest Lebanese crisis—I cringe at using the word crisis for it seems news organizations use it all too frequently to condense all human suffering and all other news into this pointless pithy—has been covered in the Arab media as a predominantly Muslim affair where a Jewish state is attacking Muslims. While the thrust of the statement remains true, the fact of the matter is that what is happening in Lebanon is a humanitarian crisis, a human tragedy if you will and has little or nothing to do with people there being Muslims or non-Muslims. The portrayal is all the more bankrupt given the fact that Lebanon has about 40% Christian population. Kashmir, Chechnya, Palestine, Lebanon or Bosnia are and should be treated as a humanitarian crisis and not as Muslim crisis by the Arab media. There is a subtext in all the coverage in the Arab media that a Saudi resident or an Arab should feel more about the Lebanese than say someone sitting in EU. There is subtle and not too subtle racism that accentuates the us vs. them schism that has opened up between the world and Islam as a whole. There are mitigating reasons that are offered including the fact that Arab press is deliberately framing it as a Muslim issue to demand action from their ostensibly Muslim governments but then again I think it is giving too much credit to the Arab media for this deep-rooted problem that finds its face in all major Muslim media from Indonesia to Pakistan.

Of course, the Western media can’t go scot-free either. Western media outlets eager to portray Hezbollah as a Shiite militia backed by Iran and eager to portray Lebanese as a bunch of ‘enemy terrorists’ have overlooked the fact that “Hezbollah is principally neither a political party nor an Islamist militia. It is a broad movement that evolved in reaction to Israel’s invasion of Lebanon in June 1982” NY Times

Roger Pape, in his NY Times op-ed piece, adds,

“Evidence of the broad nature of Hezbollah’s resistance to Israeli occupation can be seen in the identity of its suicide attackers. Hezbollah conducted a broad campaign of suicide bombings against American, French and Israeli targets from 1982 to 1986. Altogether, these attacks, which included the infamous bombing of the Marine barracks in 1983, involved 41 suicide terrorists.

In writing my book on suicide attackers, I had researchers scour Lebanese sources to collect martyr videos, pictures, and testimonials and the biographies of the Hezbollah bombers. Of the 41, we identified the names, birthplaces and other personal data for 38. Shockingly, only eight were Islamic fundamentalists. Twenty-seven were from leftist political groups like the Lebanese Communist Party and the Arab Socialist Union. Three were Christians, including a female high-school teacher with a college degree. All were born in Lebanon.”

From Satellites to Streets

11 Jul

There has been a tremendous growth in satellite-guided navigation systems and secondary applications relying on GIS systems, like finding shops near the place you are, etc. However, it remains opaque to me why we are using satellites to beam this information when we can easily embed RFID/or similar chips on road signs for pennies. Road signs need to move from the ‘dumb’ painted visual boards era to electronic tag era, where signs beam out information on a set frequency (or answer when queried) to which a variety of devices may be tuned in.

Indeed it would be wonderful to have “rich” devices, connected to the Internet, where people can leave our comments, just like message boards, or blogs. This will remove the need for expensive satellite signal reception boxes or the cost of maintaining satellites. The concept is not limited to road signs and can include any and everything from shops to homes to chips informing the car where the curbs are so that it stays in the lane.

Possibilities are endless. And we must start now.

In India, Reforms Don’t Reach the Rural Areas

6 Jul

Pankaj Mishra, writing for the New York Times, takes on the myth of “New India,”

“Recent accounts of the alleged rise of India barely mention the fact that the country’s $728 per capita gross domestic product is just slightly higher than that of sub-Saharan Africa and that, as the 2005 United Nations Human Development Report puts it, even if it sustains its current high growth rates, India will not catch up with high-income countries until 2106.

Nor is India rising very fast on the report’s Human Development Index, where it ranks 127, just two rungs above Myanmar and more than 70 below Cuba and Mexico. Despite a recent reduction in poverty levels, nearly 380 million Indians still live on less than a dollar a day.

Malnutrition affects half of all children in India, and there is little sign that they are being helped by the country’s market reforms, which have focused on creating private wealth rather than expanding access to health care and education. Despite the country’s growing economy, 2.5 million Indian children die annually, accounting for one out of every five child deaths worldwide; and facilities for primary education have collapsed in large parts of the country (the official literacy rate of 61 percent includes many who can barely write their names). In the countryside, where 70 percent of India’s population lives, the government has reported that about 100,000 farmers committed suicide between 1993 and 2003. ”

A related article in the BBC talks about how the recent economic growth in India and China has meant little reprieve for those living in the rural areas.

What Does it Mean to be Literate in India?

3 Jul

A country hailed internationally for its engineers is also home to about a third of world’s illiterates (UNESCO, 2000).

Indian government defines literacy as the ability to read and write, which is similar to UNICEF’s definition. The 2001 census put India’s literacy rate at 65.4%, leaving over 250 million (counting only people older than 7) people who can’t read and write. The female literacy levels were worse. “In 1991, less than 40 percent of the 330 million women aged 7 and over were literate, which means [then] there are over 200 million illiterate women in India.”

While these figures are bad enough, the picture looks distinctly worse when one surveys the literacy attainment of people classified as literate.

“A recent study by ORG-CSR (2003) conducted in rural villages across five states – Uttar Pradesh, Madhya Pradesh, Bihar, Rajasthan, and Gujarat – confirms the low skill attainment levels of many literates in India. To share some key findings on reading, print awareness, writing, and functional aspects of ability with the written word in Hindi: 68.2% perceived themselves to be literate.

  1. Based on their reading of an extremely simple paragraph from textbooks at 2nd to 3rd grade level, the field surveyors classified the sample as: 12% who can read with ease, 36.3% who made mistakes or read with a range of reading difficulties, and 51.7% who could not read at all.
  2. Faced with a square block of Hindi text printed centered on a square piece of paper with no other graphical indicators of beginning, ending, or page orientation, 37.4% could not hold the printed matter in the proper orientation for reading. After this was shown (or known), 42.5% could not point to the end of the text. Half the sample could not move their finger to delineate the left to right direction of print and a nearly equal proportion could not move from the end of one line to the beginning
    of the next line immediately below.
  3. Only 37.5% could write their full name correctly, 15.1% could write it partially or with mistakes, and 47.4% could not write it at all.
  4. Reading the bus board, one of the most common encounters with print in village life, was, by their own admission, not possible for 51.9%. Self-reports on other functional aspects inform us that 56% could not read a newspaper, 54.8% could not read letters, and 56.7% could not write a letter themselves.

….

“A nation’s literacy rate is determined, to a great degree, by the definition of literacy and the method used to measure it. Countries struggling to achieve higher rates often tend to lower definitional bars, which then makes progress that much easier. India is no exception, and this raises simple but unanswered questions. How many of India’s literate people—literate according to the Census—can read the headlines of a newspaper?

If a demonstrated “ability to decode the simplest of passages were operationalized” as the definition of literacy, not necessarily with understanding, then only 10–15% would be fully literate.”

Source: Brij Kothari and others

In short, there are near half a billion people who cannot decode simple passages. Given the importance of literacy in improving important things like health and access to jobs, it is critical that India invest more money in literacy programs. But where to invest? Brij Kothari believes that literacy can be increased by providing Same Language Subtitling (SLS)—subtitling the lyrics of popular song-based television programs like Chitramala in the same language as the audio. Initial tests of SLS in Gujarat have been successful. The novel approach to increasing literacy leverages the fact that a lot of Indians have access to television and watch a bunch of it.

Counterpoint
Voice technologies such as speech recognition, text to speech, and auto-correction make the ability to write and read the written word increasingly optional. Thus, one way to move forward is to make such technologies more widely available. Another way would be to make essential information more broadly available in a non-textual format. Until now, these ideas have been considered as fallback options. But to continue to think of them as that would be a mistake. They may simply reflect the future of humanity.

p.s. An article on inequality in achievement in maths in India: India Shining, Bharat Drowning: Comparing Two Indian States to the Worldwide Distribution in Mathematics Achievement (pdf) by Jishnu Das et al.

Piracy—the Real Mantra Behind Indian Success

9 May

The Wall Street Journal (WSJ) ran an opinion piece yesterday arguing that India is “rapidly evolving into Asia’s innovation center” and “leaving China in the dust” because of its famed intellectual property regime.

Not only is the claim patently bogus, but the opposite—that the Indian innovation is surviving primarily through piracy—is very likely true. It is, of course, clear that the authors of the article have never ventured to Palika Bazaar or Nehru Place in Delhi or the countless other entities that use and sell pirated materials in India. Nor have the authors ever read an Indian science book, for all they will find are hasty copies of works by foreign authors on poor paper. Nor have they ever been to an Indian store in the US. For they won’t be able to spot a rightfully purchased Bollywood movie at these stores.

Yes, India has a wonderful copyright law. At least so the gentlemen would like us to believe. However, it is rarely enforced. The article points out that in 1994 the copyright act was amended to explain the rights of holder and penalties for infringement. “In 1994, the Indian Copyright Act was amended to clearly explain the rights of a copyright holder and the penalties for infringement of copyrighted software.” Nowhere does the article mention that the act itself was rewritten to make it tougher. The only effect of the law, which the article mentions has been called one of the “toughest in the world” (without quoting sources), was to create this handbook.

Since the implementation of a copyright law that was “one of the toughest in the world,” a government study on copyright piracy in India done in 1999 concludes, “The total value of pirated copyright products sold in India during 1996-97 was about Rs. 1,833 crores which formed 20% of the legal market. Segment wise, the piracy rate is found to be the highest in computer software (44%) and lowest in cinematographic works (5%).”

Let me finish by focusing on how piracy has helped India innovate. Without the countless street level computer training centers which mostly rely upon pirated software, there wouldn’t have been an IT revolution in the country. Without the lax patent laws on Pharmaceuticals, which patented only the way in which a medication is produced and not the mix of ingredients itself, there would have been no Indian success story in Pharmaceuticals. Without the cheap knock-off science books that are abundant for poor Indian students, there wouldn’t have been the countless educated Indians with a high level of understanding of fundamentals of science.

Moving on to the authors’ contention that India is leaving behind China in the dust. The authors use the following line to support such an exaggerated claim, “The number of Indian patent applications filed has increased 400% over the past 15 years.”

Aah, the wonders of statistics.

Let’s put the numbers in perspective. “According to the World Intellectual Property Organization’s (WIPO), the number of international patent applications from Japan, Republic of Korea, and China, has risen by 162%, 200%, and 212%, respectively, since 2000. These growth rates reflect the rapidly growing technological strength in northeast Asia.” More instructively, China in 2004 filed for 1,705 patents while India filed for 689. [PDF – WIPO statistics]. One more comment about China. Chinese economy (and innovation) with annualized growth rates of upwards of 9% and with high tech stalwarts like Lenovo is flourishing. Any comments as to leaving China in the dust aren’t just wrong, but also dumb or dishonest.

Lastly, I would like to address the question of why this poorly researched article trumpeting fake achievements and rationale for India’s success has made it to the Wall Street Journal. My guess is that this is a deliberate piece, produced after much deliberation with the businesses. It comes as no surprise that one of the authors of the article, Mr. Wilder is a lawyer representing the euphemistically named IP lobbying Association called the “Association for Competitive Technology.”

Ditty for Bush

6 Dec

Seldom has a country reached such levels of obsequiousness that Pakistan reached when officials chose to include a rhyming poem titled, The Leader, praising George W Bush in its English-language course book for 16 year-olds. The poem spells out George W Bush in addition to coming up with lines like – “Strong in his faith, refreshingly real” and “Bracing for war, but praying for peace”.

Patient and steady with all he must bear,
Ready to meet every challenge with care,
Easy in manner, yet solid as steel,
Strong in his faith, refreshingly real
Isn’t afraid to propose what is bold,
Doesn’t conform to the usual mould,
Eyes that have foresight, for hindsight won’t do,
Never backs down when he sees what is true,
Tells it all straight, and means it all too.

Going forward and knowing he’s right,
Even when doubted for why he would fight,
Over and over he makes his case clear,
Reaching to touch the ones who won’t hear.
Growing in strength he won’t be unnerved,
Ever assuring he’ll stand by his word.

Wanting the world to join his firm stand,

Bracing for war, but praying for peace,
Using his power so evil will cease,
So much a leader and worthy of trust,
Here stands a man who will do what he must.

Highway to India

5 Dec

Amy Waldman recently wrote a series of articles about the socio-cultural impact of highways and the burgeoning number of cars.

Here are the links:

The articles include interactive features with audio commentary and slideshows. While the photographs are well shot, Waldman’s hurried narration leaves much to be desired.

Brief comments and caveats: Waldman puts far too much emphasis on the posited transformative cultural power of both the highways and the increasing number of cars. And she repeatedly paints “old India” in stereotypical terms like, “the land of dharma”, etc.

The Art of Reading

27 Nov

The value of reading is constrained by how one chooses to read aside from what one chooses to read.

Reading is anti-evolutionary. Neither our brains nor our eyes were designed to excitedly decipher small symbols printed on a paper. But then reading is much more than deciphering symbols. Words provide wonderful abstract worlds in which we can embody the characters that are described in the book. But to live with them, in them and empathize with them, we need to spend time with them and nurture them carefully in our minds. A character in a novel is truly subjective (it is often left deliberately open to manipulation). The emotions, the pitch of the scream, rationality of action and the sinister atmosphere are all amplified or mellowed, tampered with or abandoned in our minds. The true pleasure of reading lies in reading slowly to go over the nuances and the phraseology. Of course, not all novelists and all passages invite this cohabitation. In fact, some novelists will go out of their way to create atmospheric dread that pushes you away from the analysis but then you are living through the temporary paralysis of emotions that comes when environment overwhelms you. But then you need to pause and introspect for that is when you can empathize with the character.

Reading slowly can help one introspect and come to a better understanding of oneself and the world around us. If one chooses to look at a novel merely as a teleological progression towards the resolution of some quibble, then it merely becomes a tool for entertainment.

Perhaps a more important virtue, as compared to reading slowly, is reading critically. A novelist imposes his or her worldview on you and you need to be able to critically think through the points that s/he makes, and separate out the chaff from the wheat.

The Lost Art

Today, reading slowly is a lost art. Leisurely reading a passage and then mulling over its contents seems archaic. Doubtless, pointless drivel camouflaged as writing has taken much away from the pleasure (and motivation) for reading slowly. The other obvious villain is the television with its increasingly crazed editing. Once upon a time, a shot lasted 90 seconds. Now it lasts for less than 6 seconds on average. The reader today needs a more action-packed story that relentlessly moves across scenes, countries, and emotions—all in a hurried progression to the end. So not only are novelists concocting stories that encourage hurried reading, readers are actually reading books the same way as they watch telenovelas or sitcoms—mindlessly.

Let me end with a caveat. I am not saying that speed reading is necessarily bad. In fact, there is good reason to believe that it is a very important tool for academics and few other people who need to consume a lot of information in a very limited amount of time.

End of Information Hierarchy

11 Nov

Today, people have a variety of ways to explore a collection via the Internet as opposed to carefully orchestrated explorations in a brick and mortar museum with a curated exhibition.

A curator comes up with a story along with other contextual information about the exhibit and arranges the exhibition so that the person exploring it has only a few chosen entry points and few ways of exploring the collection. Some of the impediments are put in deliberately while others are a result of hosting an exhibition in the real world where the design of building etc. still matter.

Cut to the online world and the user is untethered from most of the curated connivances. This, in turn, may be a result of the fact that people haven’t really understood how best to present a virtual museum but that is not the point I want to get into. The result of the untethered experience is that these cultural objects are seen in a twice removed setting -e.g. a pot taken from an archaeological site and then photographed and put on the Internet. So what is the result of all this? It is hard to give an objective listing but one can see that some of the “meaning” is lost in this journey of an artifact from the ground to the Internet.

What happens when information that was once tethered in a context or a story is made available virtually free of context over say Google. Is storing information in hierarchical networks or associations obsolete? How do you maintain the integrity of information when context-free snippets of information are freely available?

Say, for example, once upon a time people learned about history via a scholar who chose carefully the specific issues about history. Today, a teen gets his/her history by searching on the web often encountering a lot of miscellaneous information. I would argue that the person then can come away, from such a scattered exploration, with a bunch of miscellaneous trivia and no real understanding of the major issue at hand. The key idea here is that for transmission of “knowledge”, the integrity of information is of prime value.

War and Sex

11 Nov

War is deadly for both sexes. A missile doesn’t differentiate between a man and a woman. Then, what is the role of gender in war?

Nearly all active militaries in the world have substantially more male soldiers than female soldiers and far more men die on the battlefields than women. But the impact of wars is never limited to artificial battlefields. War enters civilian life through hunger, inadequate health care, the decline in availability of potable water, rape, pillage, and many other ways, reducing life expectancy drastically for both men and women. For example, life expectancy in Afghanistan is 46 years (men), 46 years (women) according to UN figures. The figures hide an important fact that on average women will live longer than men. These figure mean that more women are dying as a result of war than men. These figures still don’t take into account the large number of crimes like rape that are committed predominantly against women.

Facts About South Asia

19 Aug

South Asia is home to one-fifth of the world’s population and about 40% of the world’s absolute poor—people living on less than $1/day. Imagine the lifestyle of an American earning $1/day and you will get a window into the poverty described by these figures.

India is home to nearly half of the illiterate population in the world. The adult literacy rate in South Asia (49%) is behind sub-Saharan Africa (57%) as well as that of Arab states (59%). To make matters worse, South Asia’s current annual expenditure on education is 1.9% of GNP. In contrast, military spending in the region is 3.8% of GNP and is as high as 7% in Pakistan which has 50% more soldiers than teachers. A brief zoom in on Pakistan’s education system…. what indeed are people fortunate enough to afford an education are taught? According to a report by an independent government agency, SDPI ( Sustainable Development Policy Institute. See Link at Bottom), ‘facts’ like “Hindu has always been an enemy of Islam.” and “The religion of the Hindus did not teach them good things — Hindus did not respect women…” have been included with the general objective of inculcating “Love and aspiration for Jehad, Tableegh (Prosyletization), Jehad, Shahadat (martyrdom), sacrifice, ghazi (the victor in holy wars), shaheed (martyr)”

Due process of law is often quoted as a key ingredient for a free society. With over 20 million court cases pending at the end of 2002, India doesn’t even pretend. More stark crime statistics in India include—over 1 million people in jail waiting for trial, and a conviction rate of about 1%.

Links
SDPI report on Pakistan Education System (pdf)
World Literacy of Canada

Merchants of Art

26 May

“Shakespeare Wallah” was my introduction to the magic of Merchant Ivory Productions. An elegy to a lost era, a bitter-sweet tale of a traveling English theatre troupe in India right after the Indian independence, it is still vivid in my memory. The debonair Shashi Kapoor and Satyajit Ray’s beautiful score are the two other things that I remember from the film. Since then, I have seen many other Merchant Ivory productions. And their films have always left me simultaneously reassured and disturbed.

A constant in all their movies has been the excellent production values, largely a product of Mr. Merchant’s vision, and his acknowledged genius for creating beautiful, authentic sets on a shoestring budget. With the demise of Ismail, we no longer have a producer who fussed over each detail.

The troika of James Ivory, Ruth Prawer Jhabwala, and Ismail Merchant over the past four decades virtually invented a new genre of films. They showed that you don’t need to compromise on art to be successful. And to me, that is the legacy of Merchant Ivory.

I will forever be indebted to their patience, art, and virtue.

Further Reading:
BBC article on Ismail Merchant

The Misspelled Universe

18 Apr

How many times have you typed something in Google to be asked “Did you mean: ….” Next time you reach this page, stay a little longer and take a look at the pages that Google did find. This is your gateway to the parallel universe of misspelled words. Well let me correct myself — these “misspelled” words can belong to a different language altogether or they even might be rarely used genuine English words with close resemblance to the heavily used ones.

An entire gamut of information is being denied to us due to mere errors in spelling. To deride these spelling mistakes as “mere errors in spelling” is to ignore a small minority of people who deliberately misspell words so as to make their pages less publicly accessible. This works as an effective low-tech solution for every underground society has demands obscurity.

Then there are people who exploit misspellings to make their living e.g. People searching auction sites like eBay for misspelled (or mislabeled) items, and hence hopefully underbid items. (* eBay now offers a spell-check utility but surprisingly few people still refuse to use it.)

Excepting eBay entrepreneurs, one thing that is clear is that we are “losing”‘ this increasingly vast pool of information containing misspelled “keywords” (words we type in a search engine). There is an argument to be made that the quality of information source with misspelled words may itself be poor and hence we needn’t worry about the “lost” information. Arguably, the frequency of misspelled words in a peer-reviewed journal is much lower than say my blog. ;) The normative question is, Does that rightly consign my blog to obscurity?

Internet search is a classic case of finding a needle in a haystack, and search algorithms are built of dispense with as much “clutter” (hay) as fast as possible, leaving a very small minority of websites that are given genuine value. What we are seeing are two trends implicit in Google’s search algorithm — most of our search needs are about “popular”‘ items (given a higher rank by Google), and it is progressively harder to find “unpopular” sources. On the face of it the trend is innocuous and even sensible but the wider ramifications include information hegemony.

Let us turn the discussion around to sites that use “syntactically correct but meaningless verbiage including commons search terms” (a sentence like “Indeed, a blind crenelation blasphemously a player inside the stictomys. For example, a whopper behind a ferrocyanide indicates that the saccharinity behind a casino tropez another euphausiacea from another modem.”) People also “Google bomb” (mass posting on blogs/lists associating a search phrase with online address). Some sites have in fact automated this by writing programs that automatically go to different blogs/lists and post entries/comments like “poker chips poker – [web address].” This problem is much worse as it is making it progressively harder for us to find “genuine” (or most popular/reliable) information.

So will there be too much seemingly reliable unreliable information or will we miss a lot of seemingly unreliable reliable information? Chances are that both will happen.

Some Die Young

26 Dec

There are 34 countries in Africa where life expectancy at birth of both men and women is equal to or less than 51.
Data are from the 2003 UN estimates.
Note that life expectancy at birth is strongly impacted by infant mortality.

Name of country Av. Age of Men  Av. Age of Women

Angola      39      41
Benin       48      51
Botswana    39      40
Burkina Faso    45      46
Burundi     40      41
Cameroon    45      47
Central African 38      40
Republic
Chad        44      46
Rep of Congo    47      50
DR Congo    41      43
Djibouti    45      47
Equatorial  48      50
Guinea
Ethiopia    45          46
Guinea      49      49
Guinea Bissau   44      47
Ivory Coast 41          41
Kenya       43          46
Lesotho     32          38
Liberia     41          42
Malawi      37          38
Mali        48          49
Mozambique  37      40
Namibia     43      46
Niger       46      46
Rwanda      39      40
Sierra Leone    33      35
Somalia     45      48
South Africa    45      51
Swaziland   33      35
Tanzania    42      44
Togo        48      51
Uganda      45      47
Zambia      33      32
Zimbabwe    34      33