Big Data...BIG Problem2 Jun, 2015 By: Niraj Tenany, Netwoven, Inc.
Most Big Data problems are typically contained within a data collection process. This process has grown so big and with the analysis of such a large volume of data, it has become difficult to manage utilizing conventional Relational Data Management.
The Big Data universe includes data that is voluminous, complex and very fast moving. IDC underscored this situation when the company estimated in 2010, that we generated enough digital information, worldwide, to fill a stack of DVDs reaching from the earth to the moon and back. That's about 1.2 zettabytes, or more than one trillion gigabytes — a 50% increase over 2009. The company further estimates that from 2011 on, the amount of data produced globally will double every two years.
In addition to the sheer volume of data being created, IDC estimates that in five years the majority of analytic data will come from unstructured sources. This only exacerbates the problem of managing such a mammoth amount of web pages, social network content, emails and multimedia files.
All this is changing the way organizations need to process and analyze data as well as turn it into productive information. To address this situation, enterprise organizations are engaging in Business Intelligence (BI) applications.
Although search technologies and BI applications have been in existence for many years, they often live in separate silos of operation. The BI community has not yet adopted search technology, nor have search technology creators seriously considered integrating with the BI market. The answer to this conundrum resides within better search-based applications. But first, let’s explore the definition of search-based application.
Search-Based Business Intelligence Application
Search-based Applications are applications that utilize the search technology to retrieve information from both structured and unstructured data sources. Search-based BI applications correlate and find patterns in the data by utilizing search techniques. In addition, search technologies scale easily to large volumes of data, which is a great fit for big data needs.
With a search-based application, the end-user enters any information known about a customer or product. This can be a keyword or words that are part of a corporate taxonomy. The search engine locates the information from various disparate repositories (both structured and unstructured) and provides a return result that integrates information from these various repositories.
Search interfaces have been known to be one of the most user-friendly interfaces. Additional features include:
- Enablement of integration and the search of any data source
- Search capabilities to be performed across multiple sources
- Easy integration of structured and unstructured data sources
- Real-time indexing
- Assisted navigation for filtering search results, to reduce processing time
- Ability to display results in a highly visual and interactive form
For additional information on how to create search-based applications using fast searchers, use Resources found on Netwoven’s website, and download conference material presented by Netwoven consultants.
Niraj Tenany is President & CEO of Netwoven Inc., a leading Enterprise Information Management organization helping companies design & deploy solutions for business collaboration, intelligence, workflow, and relationship management. For detailed company info and links to Resources, visit http://www.netwoven.com