Societal-scale software systems, such as today’s commercial social media platforms, are among the most widely used software systems in the world, with some platforms reporting billions of daily active users. These systems have created new mechanisms for global communication and connect people with unprecedented speed. Despite the numerous benefits of societal-scale systems, these systems are designed to optimize user engagement and scale by using psychology (such as gaming and reward mechanisms) to influence users. Individual users struggle with privacy of their data and bias in these systems, while governments face new threats of misinformation.

As a federally funded research and development center, the SEI seeks to find answers to the most complex challenges faced by the Department of Defense (DoD) in the areas of artificial intelligence engineering, cybersecurity, and software engineering. As discussed in the following blog posts Architecting the Future of Software Engineering: A Research and Development Roadmap and Software Engineering as a Strategic Advantage: A National Roadmap for the Future, the SEI—in close collaboration with our advisory board and other leaders in the software engineering research community—developed a roadmap for the decade and identified six research focus areas. This blog post explores one of those six software engineering challenges: engineering societal-scale software systems.

Software engineering for societal-scale systems focuses on predicting the full range of impacts that arise when humans are integral components of a system, including unintended consequences and the potential for misuse and manipulation. A key challenge in this area is predicting outcomes of socially inspired quality attributes, such as trust, bias, or privacy. The goal is to leverage insights from the social sciences to build and evolve societal-scale software systems that factor in these attributes.

What Are Societal-Scale Systems?

Societal-scale systems consist of more than conventional social media platforms. The essential characteristic of these systems is that they are information and communications channels that foster desired outcomes (such as engagement and action) as a primary source of revenue. Some examples include the following:

• social media platforms, such as Facebook, Twitter, and Instagram
• search platforms, such as Google Search and YouTube, that help people find desired content on the Internet or hosted by the service and that also provide individualized recommendations based on data
• virtual reality or metaverse systems
• systems that attempt to predict events (such as school shootings, terrorist attacks, flu outbreaks, and super spreaders) based on search or other data
• gamification to increase engagement in areas such as personal health (e.g. Fitbit) or financial activities (e.g. Robinhood)

Software system developers have long accounted for people as system users. However, the unprecedented scale, complexity, and speed of societal-scale systems have expanded the role of people, who are now de facto components. This evolution of the role of humans in societal-scale systems has created new software engineering challenges.

Limitations of Current Practice

Societal-scale software systems connect people and provide new communication mechanisms, but there is limited understanding of how these systems influence the behaviors of individuals over time or how to predict the collective behaviors of millions of people. The limitations of engineering societal-scale systems under the existing state of the practice, which lacks the understanding and safeguards required to mitigate these issues, create serious risks to both individuals and society. In particular, bias and misinformation create unforeseen and unrestrained consequences.

Although societal-scale systems are among the most frequently used software systems around the world, there are significant limitations with current practices for engineering these systems, including the following:

• Societal-scale software systems incorporate artificial intelligence (AI), create new information flows, and reshape societal knowledge, but there is little understanding of how a resulting socio-technical system will behave at scale.
• Societal-scale systems today are developed primarily to maximize engagement through individual interactions that collectively create networks used to influence behavior or perception. Current engineering practices have limited means of assessing individual engagement algorithms or create safeguards against uncontrolled warping of social knowledge.
• Societal-scale systems provide daily direct connections to a global audience, dramatically increasing the scale for potential social manipulation. The manipulation can be unintentional, due to a lack of understanding of these systems, or intentional, as nation-states or rogue actors create online campaigns to manipulate populations.

New Design Principles Are Needed for Societal-Scale Systems

The SEI’s recent study on Architecting the Future of Software Engineering: A National Agenda for Software Engineering Research & Development provided a research roadmap that includes advanced architectural paradigms needed to create systems with new types of components while maintaining predictable aspects of system behavior. In particular, introducing AI components into systems, considering humans as elements of a system, and effectively exploiting quantum computing pose particularly important challenges for future systems.

Engineering societal-scale systems involves making subtle judgments (for example, the appropriate interaction between software systems and free-speech principles) and predicting aggregate behaviors across many people. These systems should provide information or communication channels that can predictably lead to bounded or desired outcomes (such as engagement, accuracy, and so forth) for a given range of interactions from their human components. As these systems proliferate, more research is needed to enable such prediction and control of system behavior.

5 Research Topics for Societal-Scale Systems

Addressing the current limitations in constructing societal-scale systems requires thinking architecturally about the various activities involved in continuous evolution of these large-scale software systems. A common theme across the following research activities proposed in Architecting the Future of Software Engineering is the critical need for cross-disciplinary research, including computer science with various social sciences, including psychology, sociology, and economics.

New Quality Attributes and Architectures

Societal scale socio-technical systems have new quality attributes that are not well understood. Most software architectures today are created to support tradeoffs across well-understood quality attributes, such as performance, reliability, and safety. A key goal of defining new quality attributes includes defining the metrics of merit (e.g., what items are important to measure in societal-scale systems) and how to measure them.

The focus on quality attributes is to understand the relationship of design decisions in societal-scale systems and the behaviors of the systems in use. New approaches are needed to deal with the many dimensions of human behavior. Engineering systems with predictable impact on humans will help avoid surreptitious ideological influence. Trust, privacy, and bias are not totally new, but what is new is the challenge of predicting and monitoring these attributes.

We want to apply social sciences as a basis for understanding these quality attributes, similar to the way we apply physical sciences as a basis for designing for other quality attributes, such as predictability in cyber-physical systems. One approach is to separate the measurable manifestation of a quality attribute from the interpretation of it. For example, using votes as a measure of merit (like the best product or solution) seems useful but also has challenges, including bias and understanding the biases in samples of voters.

Developing a Theory of Socio-Technical Knowledge Creation

Future software developers must consider multiple dimensions of individual interactions and understand how these individual interactions enable or create risks to society’s general knowledge. Addressing this challenge requires understanding how knowledge is propagated and identifying mechanisms to provide constraints that limit how much information is warped or influenced by speculation.

The communication mechanisms in socio-technical systems provide new pathways of information flow and impact social structures, norms, and public perceptions and understanding. Information has historically flowed from a few sources to large audiences through video, audio, or paper media (e.g., newspapers, books, magazines) organizations that generally included information about sources. While these media organizations were sometimes biased, they generally undertook serious efforts for verification or vetting. Information today flows quickly through social networks,massive online open courses (MOOCs) frequently through multiple sources, with sporadic vetting of the information, and little understanding of bias.

Figure 1 below illustrates a conceptual funnel, where the data going into the funnel follows multiple steps before being considered fact or knowledge. Different fields (e.g., journalism, intelligence, and academia) have different versions of the funnel. However, knowledge-based professions all have ways of testing and selecting information to identify what is and is not well founded.

Current computational methods alone are insufficient to make the judgments needed to move data through the funnel. Specific areas of interest include the following:

• protecting social epistemology by identifying the knowledge-funnel mechanisms
• identifying the essential characteristics of effective online moderation systems for vetting data and rating data sources (e.g., karma on Reddit)
• understanding and mitigating unintended consequences at scale in social media
• identifying and mitigating risks of social epistemology influencers that negatively impact knowledge creation

Adherence to Policy

Governments around the world are creating new policies and rules to govern socio-technical systems. These rules can vary at national, state, or even local government boundaries even though the socio-technical systems exist at a global scale. New mechanisms are needed to engineer these systems when policy rules vary across locations and to provide the measurement and audit capability required for government regulation.

To establish a technically grounded and adaptable governance framework for societal-scale systems, software engineering research is needed to help formulate and validate data-driven and openly understood techniques for industry, government, and society. This research is needed because engineers face many tradeoffs as governments address the following considerations:

• requiring transparency when influence mechanisms operate in the software
• monitoring policy adherence while preserving the privacy of individuals
• protecting against companies, governments, or other actors misusing the ability to filter content
• building automated monitoring and reporting for policy adherence

Experimentation and Testing

Socio-technical systems operate at such large scale and with such diversity of human interaction that typical experimentation and testing approaches are ineffective. Modeling is one option, but this requires confirmation that the system and the model are consistent. It also requires a deeper understanding of individual and aggregate behaviors. Another solution is experimentation and testing as part of the operational system, such as the A/B testing performed by many tech companies. As a general strategy, however, this approach has technical challenges in bounding the experimentation. It also has potential legal and moral concerns related to human-subject testing (i.e., testing the system without the knowledge of the people using it), potentially increasing the risk of manipulation.

A deeper understanding is needed about what testing really means for societal-scale systems and how experimentation and testing should be conducted. Some additional questions include the following:

• Should people affected by a system have a role in its design (i.e., participatory design)?
• How do we create an experimentation environment to explore these areas and test new approaches? This question involves understanding the threshold of what is acceptable to test on live users versus what must be revealed to users to let them know they are part of a test.
• Since many people depend on socio-technical systems, should there be reliability or safety expectations (as in telecommunication and air-traffic management systems) to test when a main hub fails or to determine how a system can fail safely?

Software Development Using Socio-Technical Systems

An interesting use case occurs when software engineers use socio-technical systems to build software because of the influence that these systems have on engineers and the practice of engineering. These systems and open-source environments allow software developers to access huge amounts of code and create a type of digital infrastructure that both influences and is influenced by software engineers. Examples of such software include

• Stack Overflow, where people may use solutions without considering their origin or effects
• GitHub, where mining code could lead to assumptions about software qualities and possibly influence developers
• social coding environments, where popularity, number of users or followers, links, and a host of other types of social information are used in the technical decision-making process, such as the adoption of libraries and frameworks
• new ways of learning, from tutorials for developers to massive online open courses (MOOCs) that enable recognized experts to teach to anyone

Researching how societal-scale systems influence software development is important to better understand how to improve software quality and to consider how these influences could be focused to help address workforce challenges. For example, if socio-technical systems influence software developers to make better decisions these systems could augment software-developer activities and potentially increase software-developer productivity.

The Future of Architecting Societal-Scale Systems

Building new software engineering approaches for societal-scale systems requires a cross-disciplinary approach that blends together social sciences, humanities, education, journalism, and computer science. The growing recognition of software’s impact is generating new quality attribute requirements for which software engineers must develop better design approaches. In addition to traditional quality attributes (such as modifiability, reliability, performance, etc.), there is a need to add a roster of new ones, such as transparency and influence. These and other technical foundations are important to inform the discussions with individuals and government regulators looking for new solutions to understand or predict the behavior of societal-scale systems.