search menu icon-carat-right cmu-wordmark

Subject: Big Data

Reference Architectures for Big Data Systems

Reference Architectures for Big Data Systems

• SEI Blog
John Klein

Have you ever been developing or acquiring a system and said to yourself, I can't be the first architect to design this type of system. How can I tap into the architecture knowledge that already exists in this domain? If so, you might be looking for a reference architecture. A reference architecture describes a family of similar systems and standardizes nomenclature, defines key solution elements and relationships among them, collects relevant solution patterns, and provides...

Read More
Prototyping for Developing Big Data Systems

Prototyping for Developing Big Data Systems

• SEI Blog
Rick Kazman

There are several risks specific to big data system development. Software architects developing any system--big data or otherwise--must address risks associated with cost, schedule, and quality. All of these risks are amplified in the context of big data. Architecting big data systems is challenging because the technology landscape is new and rapidly changing, and the quality attribute challenges, particularly for performance, are substantial. Some software architects manage these risks with architecture analysis, while others use...

Read More
Top 10 SEI Blog Posts of 2016

Top 10 SEI Blog Posts of 2016

• SEI Blog
Douglas C. Schmidt

The crop of Top 10 SEI blog posts published in the first half of 2016 (judged by the number of visits by our readers) represents a cross section of the type of cutting-edge work that we do at the SEI: at-risk emerging technologies, cyber intelligence, big data, vehicle cybersecurity, and what ant colonies can teach us about securing the internet. In all, readers visited the SEI blog more than 52,000 times for the first six...

Read More
Big Data Technology Selection: A Case Study

Big Data Technology Selection: A Case Study

• SEI Blog
John Klein

A recent IDC forecast predicts that the big data technology and services market will realize "a 26.4 percent compound annual growth rate to $41.5 billion through 2018, or about six times the growth rate of the overall information technology market." In previous posts highlighting the SEI's research in big data, we explored some of the challenges related to the rapidly growing field, which include the need to make technology selections early in the architecture design...

Read More
The SEI Technical Strategic Plan

The SEI Technical Strategic Plan

• SEI Blog
Kevin Fall

By Kevin FallDeputy Director, Research, and CTO This is the second installment in a series on the SEI's technical strategic plan. Department of Defense (DoD) systems are becoming increasingly software reliant, at a time when concerns about cybersecurity are at an all-time high. Consequently, the DoD, and the government more broadly, is expending significantly more time, effort, and money in creating, securing, and maintaining software-reliant systems and networks. Our first post in this series provided...

Read More
A Five-Year Technical Strategic Plan for the SEI

A Five-Year Technical Strategic Plan for the SEI

• SEI Blog
Kevin Fall

The Department of Defense (DoD) and other government agencies increasingly rely on software and networked software systems. As one of over 40 federally funded research and development centers sponsored by the United States government, Carnegie Mellon University's Software Engineering Institute (SEI) is working to help the government acquire, design, produce, and evolve software-reliant systems in an affordable and secure manner. The quality, safety, reliability, and security of software and the cyberspace it creates are major...

Read More
The 2014 Year in Review: Top 10 Blog Posts

The 2014 Year in Review: Top 10 Blog Posts

• SEI Blog
Douglas C. Schmidt

In 2014, the SEI blog has experienced unprecedented growth, with visitors in record numbers learning more about our work in big data, secure coding for Android, malware analysis, Heartbleed, and V Models for Testing. In 2014 (through December 21), the SEI blog logged 129,000 visits, nearly double the entire 2013 yearly total of 66,757 visits....

Read More
Principles of Big Data Systems: You Can't Manage What You Don't Monitor

Principles of Big Data Systems: You Can't Manage What You Don't Monitor

• SEI Blog
Ian Gorton

The term big data is a subject of much hype in both government and business today. Big data is variously the cause of all existing system problems and, simultaneously, the savior that will lead us to the innovative solutions and business insights of tomorrow. All this hype fuels predictions such as the one from IDC that the market for big data will reach $16.1 billion in 2014, growing six times faster than the overall information...

Read More
Four Principles of Engineering Scalable, Big Data Software Systems

Four Principles of Engineering Scalable, Big Data Software Systems

• SEI Blog
Ian Gorton

In earlier posts on big data, I have written about how long-held design approaches for software systems simply don't work as we build larger, scalable big data systems. Examples of design factors that must be addressed for success at scale include the need to handle the ever-present failures that occur at scale, assure the necessary levels of availability and responsiveness, and devise optimizations that drive down costs. Of course, the required application functionality and engineering...

Read More
Android, Heartbleed, Testing, and DevOps: An SEI Blog Mid-Year Review

Android, Heartbleed, Testing, and DevOps: An SEI Blog Mid-Year Review

• SEI Blog
Douglas C. Schmidt

In the first half of this year, the SEI blog has experienced unprecedented growth, with visitors in record numbers learning more about our work in big data, secure coding for Android, malware analysis, Heartbleed, and V Models for Testing. In the first six months of 2014 (through June 20), the SEI blog has logged 60,240 visits, which is nearly comparable with the entire 2013 yearly total of 66,757 visits. As we reach the mid-year point,...

Read More
The Importance of Software Architecture in Big Data Systems

The Importance of Software Architecture in Big Data Systems

• SEI Blog
Ian Gorton

Many types of software systems, including big data applications, lend them themselves to highly incremental and iterative development approaches. In essence, system requirements are addressed in small batches, enabling the delivery of functional releases of the system at the end of every increment, typically once a month. The advantages of this approach are many and varied. Perhaps foremost is the fact that it constantly forces the validation of requirements and designs before too much progress...

Read More
Addressing the Software Engineering Challenges of Big Data

Addressing the Software Engineering Challenges of Big Data

• SEI Blog
Ian Gorton

New data sources, ranging from diverse business transactions to social media, high-resolution sensors, and the Internet of Things, are creating a digital tidal wave of big data that must be captured, processed, integrated, analyzed, and archived. Big data systems storing and analyzing petabytes of data are becoming increasingly common in many application areas. These systems represent major, long-term investments requiring considerable financial commitments and massive scale software and system deployments....

Read More