search menu icon-carat-right cmu-wordmark

Posts by Ian Gorton

Principles of Big Data Systems: You Can't Manage What You Don't Monitor

Principles of Big Data Systems: You Can't Manage What You Don't Monitor

• Blog
Ian Gorton

The term big data is a subject of much hype in both government and business today. Big data is variously the cause of all existing system problems and, simultaneously, the savior that will lead us to the innovative solutions and business insights of tomorrow. All this hype fuels predictions such as the one from IDC that the market for big data will reach $16.1 billion in 2014, growing six times faster than the overall information...

Read More
Four Principles of Engineering Scalable, Big Data Software Systems

Four Principles of Engineering Scalable, Big Data Software Systems

• Blog
Ian Gorton

In earlier posts on big data, I have written about how long-held design approaches for software systems simply don't work as we build larger, scalable big data systems. Examples of design factors that must be addressed for success at scale include the need to handle the ever-present failures that occur at scale, assure the necessary levels of availability and responsiveness, and devise optimizations that drive down costs. Of course, the required application functionality and engineering...

Read More
The Importance of Software Architecture in Big Data Systems

The Importance of Software Architecture in Big Data Systems

• Blog
Ian Gorton

Many types of software systems, including big data applications, lend them themselves to highly incremental and iterative development approaches. In essence, system requirements are addressed in small batches, enabling the delivery of functional releases of the system at the end of every increment, typically once a month. The advantages of this approach are many and varied. Perhaps foremost is the fact that it constantly forces the validation of requirements and designs before too much progress...

Read More
Addressing the Software Engineering Challenges of Big Data

Addressing the Software Engineering Challenges of Big Data

• Blog
Ian Gorton

New data sources, ranging from diverse business transactions to social media, high-resolution sensors, and the Internet of Things, are creating a digital tidal wave of big data that must be captured, processed, integrated, analyzed, and archived. Big data systems storing and analyzing petabytes of data are becoming increasingly common in many application areas. These systems represent major, long-term investments requiring considerable financial commitments and massive scale software and system deployments....

Read More

Contact the Author

Ian Gorton

SEI Digital Library

SEI Publications

Visit the SEI Digital Library to see other publications by Ian Gorton

View publications