Hi, this is Chad Dougherty of the Vulnerability Analysis team. One of the important roles that our team plays is coordinating vulnerability information among a broad range of vendors. Over the years, we have gained a considerable amount of experience communicating with vendors of all shapes and sizes. Based on this experience, we can offer some guidance to vendors about communicating product security issues.
Just to be clear, we're talking about product security as opposed to security products. Product security involves vulnerabilities caused by programming or design defects, insecure default or recommended deployment configurations, and other similar issues. These issues potentially affect any product offered by a vendor, including security products such as firewalls, antivirus, and authentication services. This distinction is likely obvious to most readers, but we've seen confusion about it before so it makes sense to clarify.
First, let's address the topic of receiving information about product security. Vendors should provide a persistent and reliable mechanism for researchers to report vulnerabilities. We agree with statements that, for most vendors, it's a matter of "when" you'll receive a report of a vulnerability affecting your products, not a matter of "if." Here are some specific recommendations:
Provide an easily identifiable role email address specifically for product security issues In our experience, it's extremely beneficial for the vendor to provide a role email address (e.g., a shared mailbox or an alias) for receiving information. Pick an intuitive and memorable name for the role address, such as "product-security@", "security-team@", "security-response@", or "secure@". Note that "security@" may not be a viable option because IETF RFC 2142 suggests using that mailbox name for reporting or inquiring about network operational security issues. Sites that have implemented the recommendations in this RFC may have already allocated that name, and external sites that are familiar with the recommendation may resist sending other types of messages to that address.
Even if this address only has a small number of recipients, having a fixed point of contact is valuable to external parties. Avoid using individuals as direct points of contact because of the potential problems that can be caused by job changes. Vulnerability reporters get frustrated when messages to individuals bounce or when they learn that the individual is no longer connected with the product(s) in question. While there is some additional management overhead in maintaining a role address, our experience indicates that it's a worthwhile investment. Giving members of the product security group the ability to respond from this role address provides additional consistency to external parties.
We discourage the use of standard email addresses such as "info@", "support@", and "webmaster@" for the security point of contact because security reports sent to these addresses can easily be overlooked or mishandled.
Provide a web-based reporting form A web-based reporting form allows vendors to collect reports in a well-structured manner so that the reports can potentially be searched and managed more efficiently. The form also provides a slightly easier mechanism for vulnerability reporters who may wish to remain anonymous. You may want to consider our vulnerability reporting form for ideas about how to construct your own page.
Use cryptography if possible The vulnerability reports you will receive may contain sensitive information that could be damaging to users or customers if it were exposed to an unauthorized source. Publishing an encryption key corresponding to the role address mentioned above allows researchers, especially those that consider confidential reporting part of the responsible disclosure process, to securely communicate vulnerability reports. If you think the importance of crypto in the vulnerability reporting process is exaggerated, consider this example: We once sent an encrypted mail containing sensitive information to an external researcher. We received a response from the individual telling us how relieved he was that we encrypted the message because his ISP's POP server was discovered to be compromised not long after we sent the message. Disclosure of the information would've had negative implications for a considerable number of parties.
We use PGP in our processes, but vendors may choose to offer PGP, S/MIME, or both. Maintaining crypto keys also lets vendors sign patches, support bulletins, or advisories. Users can then verify the authenticity of these publications before taking any of the actions they prescribe. Signing can be done with the same key used to receive reports or with a separate signing-only key, depending on the key management practices a vendor chooses to follow. As with the email alias mentioned above, there is some additional overhead in maintaining keys; however, it's been our experience with this, too, that the benefits can outweigh the costs if managed appropriately. The U.S. National Institute of Standards and Technology (NIST) maintains some good resources for key management policies.
Note that we haven't provided any recommendations here about what vendors should actually DO when they receive product security reports. That is a separate topic that we'll try to cover in a future post.
Because communication should be a bidirectional process, let's next consider the publication of product security information. Here are some specific recommendations:
Provide a "slash security" web page One of the first places users and potential vulnerability reporters will visit to learn about a vendor's product security is the "/security" page on the vendor's main website (e.g., "http://www.example.com/security"). Ideally, this page will be the primary focal point for everything related to product security. We recommend that it contain or refer to, as appropriate, the following content:
product security bulletins and advisories
product security contact information, including all of the resources described above
security patches and updates
documents (e.g., whitepapers, FAQs, support bulletins, electronic books) describing best practices for secure deployment or operation
documents describing secure development practices involving relevant products
In some cases, vendors have already devoted the "/security" page to their security product offerings (recall the distinction we described above). Vendors often have a lot of time and money invested in their site layout, so it's unreasonable to expect them to redesign. In these cases, we encourage vendors to include a reference to product security somewhere on the "/security" page, possibly redirecting to a "/support/security" or "/product-security" page, for example. Having "Security" as a dropdown option from a "Support" menu is also helpful.
We have received positive feedback about vendors that provide good product security pages, and, from a public perception standpoint, it's perhaps the single most valuable security-related resource you can provide.
Offer security mailing list subscription Sending copies of security advisories (or notifications of their availability on the web) to a mailing list is an effective way to get your users and customers to take action in mitigating a vulnerability. We regularly tell users to subscribe to this type of mailing list for the products they use whenever the lists are available. As with other publications mentioned above, email messages sent to these lists should be cryptographically signed to let users verify the authenticity of the message before taking action. A common technique attackers use is to send spoofed "security advisory" emails that contain malicious code in the form of "security patches." To confirm authenticity, signing is crucial if you intend to email information to customers.
We recognize that it may be counterproductive for vendors of specialized products or those that are used exclusively in critical deployments to publish product security information so broadly. In these cases, we encourage vendors to contact known customers directly or provide information in a way that is restricted to existing customers.
One of the underlying themes in this post is the recommendation to centralize points of contact as much as possible. For vendors that have a small or closely related product lines, this centralization can be a straightforward process. However, we understand that many vendors are conglomerates that have a large number of independent divisions. We've dealt with instances where one affected product division had no idea another affected product division even existed. Mergers and acquisitions present another challenge to centralization. The general recommendation we can provide in these cases is to still create points of contact at the coarsest level possible. For example, a company with a network equipment division based in the U.S. and an automation systems division based in Germany might choose to set up "www.bigcorp-networks.example/security/" and "www.bigcorp-automation.de.example/security/" and ideally have links to both of these pages at "www.bigcorp.example/security/". The same redundancy should apply to email contacts.
Vendors' attention to product security is receiving increased scrutiny in security and IT communities. Presenting organized methods for communicating product security information is an important element to demonstrating to customers (both current and potential), security researchers, the media, and the general public that you have at least some awareness of and commitment to security.
The term "software security" often evokes negative feelings among software developers because it is associated with additional programming effort, uncertainty, and road blocks on fast development and release cycle. To secure software, developers must follow numerous guidelines that, while intended to satisfy some regulation or other, can be very restrictive and hard to understand. As a result, a lot of fear, uncertainty, and doubt can surround software security. This blog post, the first in a series, is based on a keynote I recently delivered at the International Conference on Availability, Reliability, and Security (ARES). In this talk I describe how the SecureDevOps movement attempts to combat the toxic environment surrounding software security by shifting the paradigm from following rules and guidelines to creatively determining solutions for tough security problems.