Wall Street Scratches Its Head Over Big Data Challenge

By Melanie Rodier of Wall Street & Technology Reports

"The data challenge is a juggernaut, and Wall Street firms whose budgets are virtually flat these days, are scrambling to figure out how to deal with it. 

An effective strategy will recognize the importance of big data and include an investigation of the requirements to ingest, index and integrate structured and unstructured, streaming and static data from a variety of sources, Robert Desautels, CEO, President and Founder, Harvard Research Group, said at yesterday’s High Performance Computing Linux for Wall Street conference. 

Here are a few figures that were bandied around at the event : U.S. securities message rates have been growing 30% every 6 months over the last decade, according to Larry Tabb from TABB Group. 

Data volume is set to expand 800% in the next five years. Unstructured data grows 10 to 50 times faster than structured data, Jean Staten Healy, director, WW Cross-IBM Linux and Open Virtualization, IBM Corp., noted. 

In the meantime, data is getting more social. There is a 42% growth per year in internet-connected devices, Healy said. A staggering 1.2 zetabytes of data (1.2 trillion gigabytes) exists in the digital universe. 

The big problem for Wall Street is that IT budgets are growing less than 1%. As a result, organizations are scrambling to get to grips with the data challenge as cost-effectively as possible." 

Read more here