Data mining may be characterized as the process of extracting intelligent information from large amounts of raw data, and day-by-day becoming a pervasive technology in activities as diverse as using historical data to predict the success of a awareness raising campaign by looking into pattern sequence formations, or a promotional operation by looking into pattern sequence transformations, or a monitoring tool by looking into pattern sequence repetitions, or a analysis tool by looking into pattern sequence formations.
Theories and concepts on data mining have recently been added to the arena of database and researches in this aspect do not go beyond more than a decade. Very minor research and development activities have been observed in the nineties, along the immense prospect of information and communication technologies (ICTs). Organized and coordinated researches on data mining started since 2001, with the advent of various workshops, seminars, promotional campaigns, and funded researches. International Conferences on Data Mining organized by Institute of Electrical and Electronics Engineers, Inc. (since 2001), Wessex Institute of Technology (since 1999), Society for Industrial and Applied Mathematics (since 2001), Institute of Computer Vision and applied Computer Sciences (since 1999), and World Academy of Science are among the leaders in creating awareness on advanced research activities on data mining and its effective applications. Furthermore, these events reveal that the theme of research has been shifting from fundamental data mining to information engineering and/or information management along these years.
Data mining is a promising and relatively new area of research and development, which can provide important advantages to the users. It can yield substantial knowledge from data primarily gathered through a wide range of applications. Various institutions have derived considerable benefits from its application and many other industries and disciplines are now applying the methodology in increasing effect for their benefit.
Subsequently, collective efforts in machine learning, artificial intelligence, statistics and database communities have been reinforcing technologies of knowledge discovery in databases to extract valuable information from massive amounts of data in support of intelligent decision making. Data mining aims to develop algorithms for extracting new patterns from the facts recorded in a database, and up till now, data mining tools adopted techniques from statistics, network modeling and visualization to classify data and identify patterns. Ultimately, knowledge recovery aims to enable an information system to transform information to knowledge through hypothesis, testing and theory formation. It sets new challenges for database technology: new concepts and methods are needed for basic operations, query languages, and query processing strategies (Yuan, Buttenfield, Gehagen & Miller, 2004; Witten & Frank, 2005).
However, data mining does not provide any straightforward analysis, nor does it necessarily equate with machine learning, especially in a situation of relatively larger databases. Furthermore, an exhaustive statistical analysis is not possible, though many data mining methods contain a degree of non-determinism to enable them to scale massive datasets.
At the same time, successful applications of data mining are not common, despite the vast literature now accumulating on the subject. The reason is that, although it is relatively straightforward to find pattern or structure in data, but establishing its relevance and explaining its cause are both very difficult tasks. In addition, much of what that has been discovered so far may well be known to the expert. Therefore, addressing these problematic issues requires the synthesis of underlying theory from the databases, statistics, algorithms, machine learning and visualization (Hastie, Tibshirani & Friedman, 2001; Giudici, 2003; Yuan, Buttenfield, Gehagen & Miller, 2004).
Along these perspectives, to enable practitioners in improving their researches and participate actively in solving practical problems related to data explosion, optimum searching, qualitative content management, improved decision making, and intelligent data mining a complete guide is the need of the hour. A book featuring all these aspects can fill an extremely demanding knowledge gap in the contemporary world.
Furthermore, data mining is not an independently existed research subject anymore. To understand its essential insights, and effective implementations one must open the knowledge periphery in multi-dimensional aspects. Therefore, in this era of information revolution data mining should be treated as a cross-cutting and cross-sectoral feature. At the same time, data mining is becoming an interdisciplinary field of research driven by a variety of multi-dimensional applications. On one hand it entails techniques for machine learning, pattern recognition, statistics, algorithm, database, linguistic and visualization. On the other hand, one finds applications to understand human behavior, such as that of the end user of an enterprise. It also helps entrepreneurs to perceive the type of transactions involved, including those needed to evaluate risks or detect scams.
The reality of data explosion in multi-dimensional databases is a surprising and widely misunderstood phenomenon. For those about to use an OLAP (online analytical processing) product, it is critically important to understand what data explosion is, what causes it, and how it can be avoided, because the consequences of ignoring data explosion can be very costly, and, in most cases, result in project failure (Applix, 2003), while enterprise data requirements grow at 50-100 percent a year, creating a constant storage infrastructure management challenge (Intransa, 2005).
Concurrently, the database community draws much of its motivation from the vast digital datasets now available online and the computational problems involved in analyzing them. Almost without exception, current databases and database management systems are designed without to knowledge or content, so the access methods and query languages they provide are often inefficient or unsuitable for mining tasks. The functionality of some existing methods can be approximated either by sampling the data or re-expressing the data in a simpler form. However, algorithms attempt to encapsulate all the important structure contained in the original data, so that information loss is minimal and mining algorithms can function more efficiently. Therefore, sampling strategies must try to avoid bias, which is difficult if the target and its explanation are unknown.
These are related to the core technology aspects of data mining. Apart from the intricate technology context, the applications of data mining methods lag in the development context. Lack of data has been found to inhibit the ability of organizations to fully assist clients, and lack of knowledge made the government vulnerable to the influence of outsiders who did have access to data from countries overseas. Furthermore, disparity in data collection demands a coordinated data archiving and data sharing, as it is extremely crucial for developing countries.
The technique of data mining enables governments, enterprises and private organizations to carry out mass surveillance and personalized profiling, in most cases without any controls or right of access to examine this data. However, to raise the human capacity and establish effective knowledge systems from the applications of data mining, the main focus should be on sustainable use of resources and the associated systems under specific context (ecological, climatic, social and economic conditions) of developing countries. Research activities should also focus on sustainable management of vulnerable resources and apply integrated management techniques, with a view to support the implementation of the provisions related to research and sustainable use of existing resources (EC, 2005).
To obtain advantages of data mining applications, the scientific issues and aspects of archiving scientific and technology data can include the discipline specific needs and practices of scientific communities as well as interdisciplinary assessments and methods. In this context, data archiving can be seen primarily as a program of practices and procedures that support the collection, long-term preservation, and low cost access to, and dissemination of scientific and technology data. The tasks of the data archiving include: digitizing data, gathering digitized data into archive collections, describing the collected data to support long term preservation, decreasing the risks of losing data, and providing easy ways to make the data accessible. Hence, data archiving and the associated data centers need to be part of the day-to-day practice of science. This is particularly important now that much new data is collected and generated digitally, and regularly (Codata, 2002; Mohammadian, 2004).
So far, data mining has existed in the form of discrete technologies. Recently, its integration into many other formats of ICTs has become attractive as various organizations possessing huge databases began to realize the potential of information hidden there (Hernandez, Göhring & Hopmann, 2004). Thereby, the Internet can be a tremendous tool for the collection and exchange of information, best practices, success cases and vast quantities of data. But it is also becoming increasingly congested and its popular use raises issues about authentication and evaluation of information and data. Interoperability is another issue, which provides significant challenges. The growing number and volume of data sources, together with the high-speed connectivity of the Internet and the increasing number and complexity of data sources, are making interoperability and data integration an important research and industry focus. Moreover, incompatibilities between data formats, software systems, methodologies and analytical models are creating barriers to easy flow and creation of data, information and knowledge (Carty, 2002). All these demand, not only technology revolution, but also tremendous uplift of human capacity as a whole.
Therefore, the challenge of human development taking into account the social and economic background while protecting the environment confronts decision makers like national governments, local communities and development organizations. A question arises, as how can new technology for information and communication be applied to fulfill this task (Hernandez, Göhring & Hopmann, 2004)? This book gives a review of data mining and decision support techniques and their requirement to achieve sustainable outcomes. It looks into authenticated global approaches on data mining and shows its capabilities as an effective instrument on the base of its application as real projects in the developing countries. The applications are on development of algorithms, computer security, open and distance learning, online analytical processing, scientific modeling, simple warehousing, and social and economic development process.
Applying data mining techniques in various aspects of social development processes could thereby empower the society with proper knowledge, and would produce economic products by raising their economic capabilities.
On the other hand, coupled to linguistic techniques data mining has produced a new field of text mining. This has considerably increased the applications of data mining to extract ideas and sentiment from a wide range of sources, and opened up new possibilities for data mining that can act as a bridge between the technology and physical sciences and those related to social sciences. Furthermore, data mining today is recognized as an important tool to analyze and understand the information collected by governments, businesses and scientific centres. In the context of novel data, text and web-mining application areas are emerging fast and these developments call for new perspectives and approaches in the form of inclusive researches.
Similarly, info-miners in the distance learning community are using one or more info-mining tools. They offer a high quality open and distance learning (ODL) information retrieval and search services. Thus, ICT based info-mining services will likely be producing huge digital libraries such as e-books, journals, reports and databases on DVD and similar high-density information storage media. Most of these offline formats are PC-accessible, and can store considerably more information per unit than a CD-ROM (COL, 2003). Hence, knowledge enhancement processes can be significantly improved through proper use of data mining techniques.
Thus, data mining techniques are gradually becoming essential components of corporate intelligence systems and are progressively evolving into a pervasive technology within activities that range from the utilization of historical data to predicting the success of an awareness campaign, or a promotional operation in search of succession patterns used as monitoring tools, or in the analysis of genome chains or formation of knowledge banks. In reality, data mining is becoming an interdisciplinary field driven by various multi-dimensional applications. On one hand it involves schemes for machine learning, pattern recognition, statistics, algorithm, database, linguistic and visualization. On the other hand, one finds its applications to understand human behavior, or to understand the type of transactions involved, or to evaluate risks or detect frauds in an enterprise. Data mining can yield substantial knowledge from raw data that are primarily gathered for a wide range of applications. Various institutions have derived significant benefits from its application, and many other industries and disciplines are now applying the modus operandi in increasing effect for their overall management development.
This book has tried to examine the meaning and role of data mining in terms of social development initiatives and its outcomes in developing economies in terms of upholding knowledge dimensions. At the same time, it gave an in-depth look into the critical management of information in developed countries with a similar point of view. Furthermore, this book has tried to provide an overview on the main issues of data mining (including its classification, regression, clustering, association rules, trend detection, feature selection, intelligent search, data cleaning, privacy and security issues, etc.) and knowledge enhancing processes as well as a wide spectrum of data mining applications such as computational natural science, e-commerce, environmental study, business intelligence, network monitoring, social service analysis, etc. to empower the knowledge society.
Where the Book Stands
In the global context, a combination of continual technological innovation and increasing competitiveness makes the management of information a huge challenge and requires decision-making processes built on reliable and opportune information, gathered from available internal and external sources. Although the volume of acquired information is immensely increasing, this does not mean that people are able to derive appropriate value from it (Maira, & Marlei, 2003). This deserves authenticated investigation on information archival strategies and demands years of continuous investments in order to put in place a technological platform that supports all development processes and strengthens the efficiency of the operational structure. Most organizations are supposed to have reached at a certain level where the implementation of IT solutions for strategic levels becomes achievable and essential. This context explains the emergence of the domain generally known as “intelligent data mining”, seen as an answer to the current demands in terms of data/information for decision-making with the intensive utilization of information technology.
The objective of the book is to examine the meaning and role of data mining in a particular context (i.e., in terms of development initiatives and its outcomes), especially in developing countries and transitional economies. If the management of information is a challenge even to enterprises in developed countries, what can be said about organizations struggling in unstable contexts such as developing ones? The book has tried to focus on data mining application in developed countries’ context, too.
With the unprecedented rate at which data is being collected today in almost all fields of human endeavor, there is an emerging demand to extract useful information from it for economic and scientific benefit of the society. Intelligent data mining enables the community to take advantages out of the gathered data and information by taking intelligent decisions. This increase the knowledge content of each member of the community, if can be applied to practical usage areas. Eventually, a knowledge base is being created and a knowledge based society will be established.
However, data mining involves the process of automatic discovery of patterns, sequences, transformations, associations and anomalies in massive databases, and is a enormously inter-disciplinary field representing the confluence of several disciplines, including database systems, data warehousing, machine learning, statistics, algorithms, data visualization, and high-performance computing (LCPS, 2001; UN, 2004). A book of this nature, encompassing such omnipotent subject area has been missing in the contemporary global market, and this book intends to fill in this knowledge gap.
In this context, this book has provided a overview on the main issues of data mining (including its classification, regression, clustering, association rules, trend detection, feature selection, intelligent search, data cleaning, privacy and security issues, and etc.) and knowledge enhancing processes as well as a wide spectrum of data mining applications such as computational natural science, e-commerce, environmental study, financial market study, machine learning, web mining, nanotechnology, e-tourism, and social service analysis.
Apart from providing insight into the advanced context of data mining, this book has emphasized on:
Development and availability of shared data, metadata, and products commonly required across diverse societal benefit areas;
Promoting research efforts that are necessary for the development of tools required in all societal benefit areas;
Encouraging and facilitating the transition from research to operations of appropriate systems and techniques;
Facilitating partnerships between operational groups and research groups;
Developing recommended priorities for new or augmented efforts in human capacity building;
Contributing to, access, and retrieve data from global data systems and networks;
Encouraging the adoption of existing and new standards to support broader data and information usability;
Data management approaches that encompass a broad perspective on the observation of data life cycle, from input through processing, archiving, and dissemination, including reprocessing, analysis and visualization of large volumes and diverse types of data;
Facilitating recording and storage of data in clearly defined formats, with metadata and quality indications to enable search, retrieval, and archiving as easily accessible data sets;
Facilitating user involvement and conducting outreach at global, regional, national and local levels;
Complete and open exchange of data, metadata, and products within relevant agencies and national policies and legislations.
Organization of Chapters
This book has altogether fifteen chapters and they been divided into three sections; Education and Research; Tools, Techniques, Methods; and Applications of Data Mining. Section one has three chapters and they discusses on policy and decision making approaches of data mining for socio-development aspects in technical and semi-technical contexts. Section two comprises of five chapters and they illustrate tools, techniques and methods of data mining applications for various human development processes and scientific research. The third section has seven chapters and those chapters’ show various case studies, practical applications and research activities on data mining applications that are being used in the social development processes for empowering the knowledge societies.
Chapter 1 provides an overview of a series of multiple criteria optimization-based data mining methods that utilize multiple criteria programming (MCP) to solve various data mining problems. Authors state that data mining is being established on the basis of many disciplines, such as machine learning, databases, statistics, computer science and operation research and each field comprehends data mining from its own perspectives by making distinct contributions. They further state that due to the difficulty of accessing the accuracy of hidden data and increasing the predicting rate in a complex large-scale database, researchers and practitioners have always desired to seek new or alternative data mining techniques. Therefore, this chapter outlines a few research challenges and opportunities at the end.
Chapter 2 identifies some important barriers to the successful application of computational intelligence (CI) techniques in a commercial environment and suggests various ways in which they may be overcome. It states that CI offers new opportunities to a business that wishes to improve the efficiency of their operations. In this context, this chapter further identifies a few key conceptual, cultural and technical barriers and describes different ways in which they affect the business users and the CI practitioners. This chapter aims to provide knowledgeable insight for its readers through outcome of a successful computational intelligence project and expects that by enabling both parties to understand each other’s perspectives, the true potential of CI may be realized.
Chapter 3 describes two data mining techniques that are used to explore frequent large itemsets in the database. In the first technique called Closed Directed Graph Approach the algorithm scans the database once making a count on possible 2-itemsets from which only the 2-itemsets with a minimum support are used to form the closed directed graph and explores possible frequent large itemsets in the database. In the second technique, Dynamic Hashing Algorithm large 3-itemsets are generated at an earlier stage that reduces the size of the transaction database after trimming and thereby cost of later iterations will be reduced. Furthermore, this chapter predicts that the above techniques may help researchers not only to understand about generating frequent large itemsets, but also finding association rules among transactions within relational databases, and make knowledgeable decisions.
It is observed that daily, different satellites capture data of distinct contexts, and among which images are processed and stored by many institutions. In Chapter 4 authors present relevant definitions on remote sensing and image mining domain, by referring to related work in this field and indicating about the importance of appropriate tools and techniques to analyze satellite images and extract knowledge from this kind of data. As a case study, the Amazonia deforestation problem is being discussed; as well INPE's effort to develop and spread technology to deal with challenges involving Earth observation resources. The purpose is to present relevant technologies, new approaches and research directions on remote sensing image mining, and demonstrating how to increase the analysis potential of such huge strategic data for the benefit of the researchers.
Chapter 5 reviews contemporary research on machine learning and web mining methods that are related to areas of social benefit. It demonstrates that machine learning and web mining methods may provide intelligent web services of social interest. The chapter also reveals a growing interest for using advanced computational methods, such as machine learning and web mining, for better services to the public, as most research identified in the literature has been conducted during recent years. The chapter tries to assist researchers and academics from different disciplines to understand how web mining and machine learning methods are applied to web data. Furthermore, it aims to provide the latest developments on research in this field that is related to societal benefit areas.
In recent times, Customer Relationship Management (CRM) can be related to sales, marketing and even services automation. Additionally, the concept of CRM is increasingly associated with cost savings and streamline processes as well as with the engendering, nurturing and tracking of relationships with customers. Chapter 6 seeks to illustrate how, although the product and service elements as well as organizational structure and strategies are central to CRM, data is the pivotal dimension around which the concept revolves in contemporary terms, and subsequently tried to demonstrate how these processes are associated with data management, namely: data collection, data collation, data storage and data mining, which are becoming essential components of CRM in both theoretical and practical aspects.
In Chapter 7, authors have introduced the concept of “one-sum” Weighted Association Rules (WARs) and named such WARs as Allocating Patterns (ALPs). An algorithm is also being proposed to extract hidden and interesting ALPs from data. The chapter further point out that ALPs can be applied in portfolio management. Modeling a collection of investment portfolios as a one-sum weighted transaction-database that contains hidden ALPs can do this, and eventually those ALPs, mined from the given portfolio-data, can be applied to guide future investment activities.
Chapter 8 is focused to data mining applications and their utilizations in formulating performance-measuring tools for social development activities. In this context, this chapter has provided justifications to include data mining algorithm to establish specifically derived monitoring and evaluation tools for various social development applications. In particular, this chapter gave in-depth analytical observations to establish knowledge centers with a range of approaches and finally it put forward a few research issues and challenges to transform the contemporary human society into a knowledge society.
Chapter 9 highlighted a few areas of development aspects and hints application of data mining tools, through which decision-making would be easier. Subsequently, this chapter has put forward potential areas of society development initiatives, where data mining applications can be introduced. The focus area may vary from basic education, health care, general commodities, tourism, and ecosystem management to advanced uses, like database tomography. This chapter also provided some future challenges and recommendations in terms of using data mining applications for empowering knowledge society.
Chapter 10 focuses on business data warehouse and discusses about the retailing giant Wal-Mart. In this chapter, the planning and implementation of the Wal-Mart data warehouse is being described and its integration with the operational systems is discussed. It also highlighted some of the problems that have been encountered during the development process of the data warehouse, including providing some future recommendations.
In Chapter 11 medical applications literature associated with nanoscience and nanotechnology research was examined. Authors retrieved about 65000 nanotechnology records in 2005 from the Science Citation Index/ Social Science Citation Index (SCI/SSCI) using a comprehensive 300+ term query. This chapter is intended to facilitate the nanotechnology transition process by identifying the significant application areas. It also identified the main nanotechnology health applications from today’s vantage point, as well as the related science and infrastructure. The medical applications were identified through a fuzzy clustering process, and metrics were generated using text mining to extract technical intelligence for specific medical applications/ applications groups.
Chapter 12 introduces an early warning system for SMEs (SEWS) as a financial risk detector that is based on data mining. Through a study this chapter composes a system in which qualitative and quantitative data about the requirements of enterprises are taken into consideration, during the development of an early warning system. Moreover, during the formation of this system; an easy to understand, easy to interpret and easy to apply utilitarian model is targeted by discovering the implicit relationships between the data and the identification of effect level of every factor related to the system. This chapter also shows the way of empowering knowledge society from SME’s point of view by designing an early warning system based on data mining. Using this system, SME managers could easily reach financial management, risk management knowledge without any prior knowledge and expertise.
Chapter 13 looks at various business intelligence (BI) projects in developing countries, and specifically focuses on Brazilian BI projects. Authors poised this question that, if the management of IT is a challenge for companies in developed countries, what can be said about organizations struggling in unstable contexts such as those often prevailing in developing countries. Within this broad enquiry about the role of BI playing in developing countries, two specific research questions were explored in this chapter. The purpose of the first one is to determine whether those approaches, models or frameworks are tailored for particularities and the contextually situated business strategy of each company, or if they are “standard” and imported from “developed” contexts. The purpose of the second one is to analyze: what type of information is being considered for incorporation by BI systems; whether they are formal or informal in nature; whether they are gathered from internal or external sources; whether there is a trend that favors some areas, like finance or marketing, over others, or if there is a concern with maintaining multiple perspectives; who in the firms is using BI systems, etc.
Technologies such as geographic information systems (GIS) enable geo-spatial information to be gathered, modified, integrated, and mapped easily and cost effectively. However, these technologies generate both opportunities and challenges for achieving wider and more effective use of geo-spatial information in stimulating and sustaining sustainable development through elegant policy making. In Chapter 14, the author proposes a simple and accessible conceptual knowledge discovery interface that can be used as a tool. Moreover, the chapter addresses some issues that might make this knowledge infrastructure stimulate sustainable development, especially emphasizing sub-Saharan African region.
Finally, Chapter 15 discusses about the application of data mining to develop drought monitoring tools that enable monitoring and prediction of drought’s impact on vegetation conditions. The chapter also summarizes current research using data mining approaches (e.g., association rules and decision-tree methods) to develop various types of drought monitoring tools and briefly explains how they are being integrated with decision support systems. This chapter also introduced how data mining can be used to enhance drought monitoring and prediction in the United States and at the same time, assist others to understand how similar tools might be developed in other parts of the world.
Conclusions
Data mining is becoming an essential tool in science, engineering, industrial processes, healthcare, and medicine. The datasets in these fields are large, complex, and often noisy. However, extracting knowledge from raw datasets requires the use of sophisticated, high-performance and principled analysis techniques and algorithms, based on sound statistical foundations. In turn, these techniques require powerful visualization technologies; implementations that must be carefully tuned for enhanced performance; software systems that are usable by scientists, engineers, and physicians as well as researchers.
Data mining, as stated earlier, is denoted as the extraction of hidden predictive information from large databases, and it is a powerful new technology with great potential to help enterprises focus on the most important information in their data warehouses. Data mining tools predict future trends and behaviors, allowing entrepreneurs to make proactive, knowledge-driven decisions. The automated, prospective analyses offered by data mining move beyond the analyses of past events provided by retrospective constituents typical of decision support systems. Data mining tools can answer business questions that traditionally were too time consuming to resolve. They scour databases for hidden patterns, finding predictive information that experts may miss because it lies outside their expectations.
In effect, data mining techniques are the result of a long process of research and product development. This evolution began when business data was first stored on computers, continued with improvements in data access, and more recently, generated technologies that allow users to navigate through their data in real time. Thus, data mining takes this evolutionary progression beyond retrospective data access and navigation to prospective and proactive information delivery. Furthermore, data mining algorithms allow researchers to device unique decision making tools from emancipated data varying in nature. Foremost, applying data mining techniques extremely valuable utilities can be devised that could raise the knowledge content at each tier of society segments.
But, in terms of accumulated literature and research contexts, not many publications are available in the field of data mining applications in social development phenomenon, especially in the form of a book. By taking this as a base line, a compiled literature is seemed to be extremely valuable in the context of utilizing data mining and other information techniques for the improvement of skills development, knowledge management and societal benefits. Similarly, search around gray literature, including Internet search engines do not fetch sufficient bibliographies in the field of data mining for development perspective. Due to the high demand from researchers’ in the aspect of ICTD, a book of this format stands to be unique. Moreover, utilization of new ICTs in the form of data mining deserves appropriate intervention for their diffusion at local, national, regional and global levels.
It is assumed that numerous individuals, academics, researchers, engineers, professionals from government and non-government security and development organizations will be interested in this increasingly important topic for carrying out implementation strategies towards their national development. This book will assist its readers to understand the key practical and research issues related to applying data mining in development data analysis, cyber acclamations, digital deftness, contemporary CRM, investment portfolios, early warning system in SMEs, business intelligence, and intrinsic nature in the context of society uplift as a whole and the use of data and information for empowering knowledge societies.
Most books of data mining deal with mere technology aspects, despite the diversified nature of its various applications along many tiers of human endeavor. However, there are a few activities in recent years that are producing high quality proceedings, but it is felt that compilation of contents of this nature from advanced research outcomes that have been carried out globally may produce a demanding book among the researchers.