William Thomas Cain, Getty Images
Industryweek 9958 073015bigdatamagimage

The Big Problem With Big Data

Aug. 1, 2015
Why hoard your data like decades of newspapers and soup cans? Do something with it, and make it work for you and your company.

Inside a conference room off the lobby of one of the many buildings on the Microsoft campus in suburban Seattle, Dave Brown holds court on (relatively) big data. Brown is a senior research software development engineer for Microsoft Research, which is a long and official way to say he has been working with data and systems for years, and is here to show off some of that work.

Standing next to a screen large enough to mount on most living room walls, he sifts through what he describes as 40,000 lines of code about American tornadoes. He has already plugged them in, searched for patterns and, more important, worked to figure out what those patterns represent. The result is a 3-D spatiotemporal interactive data visualization — in this instance, of the contiguous United States, littered with data points, more in the Midwest than anywhere else.

“This,” he says, pointing toward Kansas, “is a small set.”

Small because big data is a relative term. Some companies consider a couple hundred gigabytes in the system to be big data, while others hardly blink at thousands of terabytes. The collection and storage of data isn’t a problem — “The amount of computation and storage you can buy per dollar keeps getting greater,” says Doug Cutting, chief architect of Cloudera and the founder of numerous open source projects that have supported big data, including Lucene, Hadoop and Avro — nor is sifting through it. But actually analyzing it and finding meaning can provide a problem.

Start with the DIKW Pyramid, a business and data staple for decades. DIKW is an acronym, short for data, information, knowledge and wisdom. Data is the base of the pyramid, and given context, it becomes information. In turn, information given context becomes knowledge, and knowledge given context becomes wisdom, which is the top of the pyramid. Most of us are far better at collecting data, the base of the pyramid, than we are at turning it into knowledge. That, in brief, is the biggest problem with big data.

“We can leverage more data than ever before,” Cutting says, “and we have to, because we can use it to see at a very high resolution what we’re doing — and we can improve it.”

For instance, that tornado data, without context, is just 40,000 lines of dates, numbers, locations, stories and lives captured in a digital nugget. But when properly analyzed, they show trends, patterns even. Where will tornadoes land? When will they land? Is there any indication about how strong the next one might be? Brown’s set indicates there is an indication, along with a forecast for more twisters in the late afternoon in the central Great Plains states.

Every business can do the same with its own data if analyzed properly. You make your money work for you in stocks, bonds, equipment, property and capital. Why not make your data work for you, too? Turn your big data into big analytics.

Popular Sponsored Recommendations

Monitoring IT, OT and IIoT: Use Cases and Dashboards

Aug. 20, 2023
The convergence of IT and OT makes it crucial to have a unified view of the entire infrastructure. From machines on the factory floor, PLCs, and IIoT devices, to industrial gateways...

Gain a competitive edge with real-world lessons on private 5G networks

Nov. 16, 2023
The use of private networks in manufacturing applications is rapidly growing. In this paper, we present valuable insights and lessons learned from the field with the goal of enhancing...

Digital Production Tracking: How Connected Platforms with No-Code Deliver Value

June 20, 2023
Manual tracking lacks the real-time visibility manufacturers need to identify root causes and remain competitive. Digital, connected production tracking is crucial for your operations...

How to Build Zero-Cost On-Site Solar and Storage Projects

Nov. 25, 2023
The Inflation Reduction Act offers tax credits, incentives, and financing that enable no-cost projects. In Enel’s eBook, discover the critical role that incentives play in your...

Voice your opinion!

To join the conversation, and become an exclusive member of IndustryWeek, create an account today!