SEI Insights

SEI Blog

The Latest Research in Software Engineering and Cybersecurity

Four Principles of Engineering Scalable, Big Data Software Systems

Posted on by in

In earlier posts on big data, I have written about how long-held design approaches for software systems simply don't work as we build larger, scalable big data systems. Examples of design factors that must be addressed for success at scale include the need to handle the ever-present failures that occur at scale, assure the necessary levels of availability and responsiveness, and devise optimizations that drive down costs. Of course, the required application functionality and engineering constraints, such as schedule and budgets, directly impact the manner in which these factors manifest themselves in any specific big data system. In this post, the latest in my ongoing series on big data, I step back from specifics and describe four general principles that hold for any scalable, big data system. These principles can help architects continually validate major design decisions across development iterations, and hence provide a guide through the complex collection of design trade-offs all big data systems require.

More from Ian Gorton


View other blog posts by Ian Gorton.

Other Publications

Visit the SEI Digital Library for other publications by Ian.



We welcome comments with a wide range of opinions and views. To keep the conversation focused on topic, we reserve the right to moderate comments.

Add a Comment


Type the characters you see in the picture above.