Showing posts with label newspaper. Show all posts
Showing posts with label newspaper. Show all posts

Tuesday, December 20, 2011

Trading and predictive analytics

I attended today's class in the course Trading Strategies and Systems offered by Prof Vasant Dhar from NYU Stern School of Business. Luckily, Vasant is offering the elective course here at the Indian School of Business, so no need for transatlantic travel.

The topic of this class was the use of news in trading. I won't disclose any trade secrets (you'll have to attend the class for that), but here's my point: Trading is a striking example of the distinction between explanation and prediction. Generally, techniques are based on correlations and on "blackbox" predictive models such as neural nets. In particular, text mining and sentiment analysis are used for extracting information from (often unstructured) news articles for the purpose of prediction.

Vasant mentioned the practical advantage of a machine-learning approach for extracting useful content from text over linguistics know-how. This reminded me of a famous comment by Frederick Jelinek, a prominent
Natural Language Processing researcher who passed away recently:
"Whenever I fire a linguist our system performance improves" (Jelinek, 1998)
This comment was based on Jelinek's experience at IBM Research, while working on computer speech recognition and machine translation.

Jelinek's comment did not make linguists happy. He later defended this claim in a paper entitled "Some of My Best Friends are Linguists" by commenting,
"We all hoped that linguists would provide us with needed help. We were never reluctant to include linguistic knowledge or intuition into our systems; if we didn't succeed it was because we didn't fi nd an effi cient way to include it."
Note: there are some disputes regarding the exact wording of the quote ("Anytime a linguist leaves the group the recognition rate goes up") and its timing -- see note #1 in the Wikipedia entry.

Sunday, November 14, 2010

Data visualization in the media: Interesting video

A colleague who knows my fascination with data visualization pointed me to a recent interesting video created by Geoff McGhee on Journalism in the Age of Data. In this 8-part video, he interviews media people who create visualizations for their websites at the New York Times, Washington Post, CNBC, and more. It is interesting to see their view of why interactive visualization might be useful to their audience, and how it is linked to "good journalism".

Also interviewed are a few visualization interface developers (e.g., IBM's Many Eyes designers) as well as Infographics experts and participants at the major Inforgraphics conference in Pamplona, Spain. The line between beautiful visualizations (art) and effective ones is discussed in Part IV ("too sexy for its own good" - Gert Nielsen) - see also John Grimwade's article.


Journalism in the Age of Data from Geoff McGhee on Vimeo.

The videos can be downloaded as a series of 8 podcasts, for those with narrower bandwidth.

Saturday, January 10, 2009

Beer and ... crime

I often glimpse the local newspapers while visiting a foreign country (as long as it is in a language I can read). Yesterday, the Australian Herald Sun had the article "Drop in light beer sales blamed for surge in street violence".

The facts presented: "Light beer sales have fallen 15% in seven years, while street crime has soared 43%". More specifically: "Police statistics show street assaults rose from 6400 in 2000-01 to more than 9000 in 2007-08. At the same time, Victorians' thirst for light beer dried up."

The interpretation by health officials: "there was a definite connection between the move away from light beer and the rise in drunken violence."

The action: There is now a suggestion to drastically reduce tax on light beer to encourage people to switch back from full-strength beer.

I am far from being an expert on drinking problems or crime in Australia (although they are both very visible here in Melbourne), but let's look at the title of this article and the data-interpretation-action sequence more carefully. The title Drop in light beer sales blamed for surge in street violence implies that the drop in light beer sales is the cause of increase in violence. Obviously such a direct causal relationship cannot be true unless perhaps retailers of light beer have become frustrated and violent... So, the first causal argument (I suppose) is that the decline in drinking light beer reflects a move to full-strength alcohol, which in turn leads to more violence. If there indeed is a shift of this sort, then the decline in light beer sales is merely a proxy for violent behavior trends*.

The second causal hypothesis, implied by the proposed action, is that beer drinkers in Victoria will switch from full-strength to light beer if the latter is sufficiently cheap.

To establish such causal arguments I'd like to see a bit more research (which might already exist and not mentioned in the article):
  • Have people in Victoria indeed shifted from drinking light beer to full-strength beer? (perhaps via a survey or from transactional data at "bottle" stores) -- there might just be an overall decline in beer consumption, as well as an overall increase in violent behavior like in other places in the world
  • Has violence increased also by non beer drinkers? What about drugs, violent movies, shift of populations, economic trends, global violence levels?
  • If such a shift exists, what are its reasons? (e.g., better quality of full strength beer, social trends, price)
  • What segment of beer drinkers in Victoria becomes violent? (age, gender, employment, income, where they buy beer, etc.)
  • Is today's beer drinking population different from the population 7 years ago in some other important ways that relate to violence?
  • Has violence been treated differently over the years? (police presence, social norms, etc.)
  • Determine how price-sensitive today's drinkers-become-aggressors are.
Only after answering the above questions, and perhaps others, would I be comfortable with seeing a causal relationship between the price of light beer (compared to full-strength beer) and the levels of aggression.

And if beer drinking is indeed a cause of violence in Victoria, how about adopting behavioral and educational ideas from other countries like France? Or maybe alcohol is simply loosening the inhibitions on the growing aggressive 21st century society.

------------------------------------------
*Note: Even if light beer sales are merely a proxy for crime levels, they can be used for predictive purposes. For example, police stations can use light beer sale levels for staffing decision.