This site uses cookies to help make it more useful and reliable. Our cookies page explains what they are, which ones we use, and how you can manage or remove them.

Science & Technology > Managing information & knowledge >

Data Mining for Effective Decision Making

Paper ID: 517 Last updated: 31/01/2012 09:08:31
Criteria: bullet Impact:  Likelihood:  Controversy:  Where: Global When: 3-10yrs How Fast: Years
0 people thought this paper expanded their thinking bullet
Keywords: bullet Information and Knowledge - mathematics, economics

Summary bullet

Discussion bullet

Mathematical tools for discovering patterns in large databases, along with stochastic modeling, could contribute to better decision making in a range of fields.

Technologies are increasingly available for collecting and storing massive amounts of data, whether about the movement of the equity markets, drug treatment outcomes, or changing air quality. Mathematical techniques will increasingly enable mining of this data to provide a basis for making effective decisions.

In the financial world, for example, large quantities of high-frequency data need to be processed effectively and efficiently so that opportunities can be quickly identified and acted upon 24 hours a day, 7 days a week. Rapid implementation of new mathematical models and algorithms, along with an expanding computing infrastructure, has the potential to meet this need. Statistical pattern recognition could be used to identify arbitrage opportunities and improve speculative trading. Integration of techniques from mathematical finance and insurance could lead to new products: catastrophe bonds, futures, and options; equity-linked life insurance; credit risk derivatives; and mortgage-backed securities. More financial decision making could be automated and made more effective by superior models, analytical tools, and processing systems.

Advances in data mining are also likely to be made in the areas such as national security and crime detection. Advances will possibly come from the development of human-machine-synergistic methods, designed to maximize the different abilities of people and machines. This will require collaborative work between mathematicians and computer scientists and psychologists, in many ways parallel to the existing collaborations between mathematicians and computers scientists and biologists and chemists on projects such as genome mapping and pharmacological research.

The highly visible collapses of Enron and LTCM have brought mathematics-based business into disrepute. In the case of Enron, however, the problems seem to have been associated with people and culture rather than with the underlying equations.

Implications bullet

More effective decision making based on analysis of large quantities of data
Expansion of the array of financial products on the market

Early indicators bullet

Current application of transaction analysis for fraud detection and product recommendation systems over large data sets

What To Watch:
Data-mining techniques are applied to massive collections of environmental sensor data to find causal relationships between industrial pollutants and global warming.
Data mining is applied increasingly to the massive amount of data associated with health care, concerning patients, procedures, and drug treatment outcomes.
Mathematicians and physicists are hired at an increasing rate by investment banks and traders.

Leaders:
Institutions:

Department of Homeland Security (the proposed Analysis, Dissemination, Visualization, Insight, and Semantic Enhancement system)
National Security Agency
KDNuggets, a leading site and newsletter on Data Mining and Knowledge Discovery [<http://www.kdnuggets.com/>link]
Autonomy, world-leading data mining software spun out from Cambridge University
University of Namur, Belgium
University of Technology, Sydney, Australia
Wessex Institute of Technology
European Bioinformatics Institute
University of Kent
University of Manchester, School of Informatics
UK e-Science Data Mining Special Interest Group

Drivers & Inhibitors bullet

Effective testing of mathematical models in real-world environments
Growth of grid computing
Development of massive distributed storage devices
Continued development of stochastic algorithms, data-mining and knowledge discovery algorithms, and modeling languages

Parallels & Precedents bullet

Enron's and Long Term Capital Management's attempts to combine academic modeling of financial markets with trading expertise, despite the ignominious end of both enterprises.[1][2][3][4][5][6][7][8][9][10]

Social Bookmarking bullet

Comments bullet

CommentsDate/TimeUser
Good example of a weak signal that may be amplified in the near future. For some experts algorithms are a general purpose technology that will revolutionise our lives.06/02/2012 12:59:17mjreilly71

People who viewed this paper also viewed bullet

IDHeadline
297Information Management and Handling: Controlling the Rising Flood of Data that Surrounds Us
507Computing on the Human Platform
500City Limits? The Future of London
300Understanding Complexity: How to Answer the Big Questions
541The Rise of Proactive and Context-Aware Computing

Sources bullet

Ref.PublisherDateTitleCategory
1OtherThe Data Mine Visit siteTech
2OtherCase Study: LTCM - Long-Term Capital Management." ERisk.com. [<http://www.erisk.com/Learning/CaseStudies/ref_case_ltcm.asp>link]"Tech
3OtherNOVA: Trillion Dollar Bet." PBS. Aired 8 February 2000. [<http://www.pbs.org/wgbh/nova/transcripts/2704stockmarket.html>link]"Visit siteTech
4OtherData Mining Techniques (statistical explanation) Tech
5OtherBasterfield, David. Directions in Computational Finance." OGI. [<http://www.cse.ogi.edu/2020vision/Basterfield.pdf>link]"Tech
6OtherMathematics: Giving Industry the Edge." Faraday Partnership for Industrial Mathematics (April 2002). [<http://www.smithinst.ac.uk/News/Roadmap/Roadmap/>link]"Visit siteTech
7OtherMcLean, B. and P. Elkind. 2003. Partners in Crime." <i>Fortune.com</i> 27 Oct. [<http://www.fortune.com/fortune/investing/articles/0,15114,517415,00.html>link]"Tech
8OtherData Mining and Discovery." American Association for Artificial Intelligence. [<http://www.aaai.org/AITopics/html/mining.html>link]"Visit siteTech
9OtherKargupta, H., A. Joshi, K. Sivakumar, and Y. Yesha. Abstract of <i>Data Mining: Next Generation Challenges and Future Directions</i>" UMBC [<http://www.cs.umbc.edu/~hillol/Kargupta/ngdmbook.html>link]"Visit siteTech
10OtherAgosta, Lou. The Future of Data Mining -- Predictive Analysis." <i>DMReview magazine</i> (August 2004). [<http://www.dmreview.com/article_sub.cfm?articleId=1007209>link]"Visit siteTech
close
The contents of this paper were supplied by the Institute for the Future, and have been reviewed by the Outsights-Ipsos MORI Partnership. Any views expressed are independent of government and do not constitute government policy.