Big Data

What is Big Data?

Big Data usually includes data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target, as of 2012 ranging from a few dozen terabytes to many petabytes of data in a single data set.

Big data Every Where !

Lots of data is being collected and warehoused                           

• Web data, e-commerce                        
• purchases at department/grocery stores
• Bank/Credit Card transactions
• Social Network
• Walmart handles more than 1 million customer transactions every hour, which is imported into databases estimated to contain more than 2.5 petabytes (2560 terabytes) of data – the equivalent of 167 times the information contained in all the books in the US Library of Congress.
• Facebook handles 50 billion photos from its user base.
• eBay.com uses two data warehouses at 7.5 petabytes and 40PB as well as a 40PB Hadoop cluster for search, consumer recommendations, and merchandising. Inside eBay’s 90PB data warehouse.

HACE Theorem:

Big Data starts with large-volume, Heterogeneous, Autonomous sources with distributed and decentralized control, and seeks to explore Complex and Evolving relationships among data. These characteristics make it an extreme challenge for discovering useful knowledge from the Big Data.

Some makes it 4V's:

Need for Big Data:

• What is the maximum file size you have dealt so far?
• Movies/Files/Streaming video that you have used?
• What have you observed?
• What is the maximum download speed you get?
• Simple computation                           
• How much time to just transf  

Memory unit	Size	Binary size
Kilobyte(KB)	103	210
Megabyte(MB)	106	220
Gigabyte(GB)	109	230
Terabyte(TB)	1012	240
Petabyte(PB)	1015	250
Exabyte(EB)	1018	260
Zettabyte(ZB)	1021	270
Yottabyte(YB)	1024	280

     Big Data Now: 2013 Edition

     Current Perspectives from O'Reilly Media

     Publisher: O'Reilly
     Released: February 2014
   








Description

In Big Data Now: 2013 Edition, we pulled together our top posts from the O'Reilly Data blog from late fall 2012 through late fall 2013. In 2013,“big data” became more than just a technical term for scientists, engineers,and other technologists—the term entered the mainstream on a myriad of fronts, becoming a household word in news, business, health care, and people’s personal lives

Posts have been divided into four main chapters:

• Evolving Tools and Techniques
• Changing Definitions
• Real Data
• Health Care

    

    The Culture of Big Data

     By Mike Barlow

     Publisher: O'Reilly

     Released: October 2013

Description

Technology does not exist in a vacuum. In the same way that a plant needs water and nourishment to grow, technology needs people and process to thrive and succeed. Culture (i.e., people and process) is integral and critical to the success of any new technology deployment or implementation.

Big data is not just a technology phenomenon. It has a cultural dimension. It's vitally important to remember that most people have not considered the immense difference between a world seen through the lens of a traditional relational database system and a world seen through the lens of a Hadoop Distributed File System. This paper broadly describes the cultural challenges that accompany efforts to create and sustain big data initiatives in an evolving world whose data management processes are rooted firmly in traditional data warehouse architectures.