index.rst (27060B)
1 .. _mozilla_projects_nss_an_overview_of_nss_internals: 2 3 An overview of NSS Internals 4 ============================ 5 6 .. container:: 7 8 | A High-Level Overview to the Internals of `Network Security Services 9 (NSS) <https://developer.mozilla.org/en-US/docs/NSS>`__ 10 | Software developed by the Mozilla.org projects traditionally used its own implementation of 11 security protocols and cryptographic algorithms, originally called Netscape Security Services, 12 nowadays called Network Security Services (NSS). NSS is a library written in the C programming 13 language. It's free and open source software, and many other software projects have decided to 14 use it. In order to support multiple operating systems (OS), it is based on a cross platform 15 portability layer, called the Netscape Portable Runtime (NSPR), which provides cross platform 16 application programming interfaces (APIs) for OS specific APIs like file system access, memory 17 management, network communication, and multithreaded programming. 18 | NSS offers lots of functionality; we'll walk through the list of modules, design principles, 19 and important relevant standards. 20 | In order to allow interoperability between software and devices that perform cryptographic 21 operations, NSS conforms to a standard called PKCS#11. (Note that it's important to look at the 22 number 11, as there are other PKCS standards with different numbers that define quite different 23 topics.) 24 | A software or hardware module conforming to the PKCS#11 standard implements an interface of C 25 calls, which allow querying the characteristics and offered services of the module. Multiple 26 elements of NSS's own modules have been implemented with this interface, and NSS makes use of 27 this interface when talking to those modules. This strategy allows NSS to work with many 28 hardware devices (e.g., to speed up the calculations required for cryptographic operations, or 29 to access smartcards that securely protect a secret key) and software modules (e.g., to allow 30 to load such modules as a plugin that provides additional algorithms or stores key or trust 31 information) that implement the PKCS#11 interface. 32 | A core element of NSS is FreeBL, a base library providing hash functions, big number 33 calculations, and cryptographic algorithms. 34 | Softoken is an NSS module that exposes most FreeBL functionality as a PKCS#11 module. 35 | Some cryptography uses the same secret key for both encrypting and decrypting, for example 36 password based encryption (PBE). This is often sufficient if you encrypt data for yourself, but 37 as soon as you need to exchange signed/encrypted data with communication partners, using public 38 key encryption simplifies the key management. The environment that describes how to use public 39 key encryption is called Public Key Infrastructure (PKI). The public keys that are exchanged 40 between parties are transported using a container; the container is called a certificate, 41 following standard X.509 version 3. A certificate contains lots of other details; for example, 42 it contains a signature by a third party that expresses trust in the ownership relationship for 43 the certificate. The trust assigned by the third party might be restricted to certain uses, 44 which are listed in certificate extensions that are contained in the certificate. 45 | Many (if not most) of the operations performed by NSS involve the use of X.509 certificates 46 (often abbreviated as “cert”, unfortunately making it easy to confuse with the term “computer 47 emergency response team“). 48 | When checking whether a certificate is trusted or not, it's necessary to find a relevant trust 49 anchor (root certificate) that represents the signing capability of a trusted third party, 50 usually called a Certificate Authority (CA). A trust anchor is just another X.509 certificate 51 that is already known and has been deliberately marked as trusted by a software vendor, 52 administrators inside an organizational infrastructure, or the software user. NSS ships a 53 predefined set of CA certificates. This set, including their trust assignments, is provided by 54 NSS as a software module, called CKBI (“built-in root certificates”), which also implements the 55 PKCS#11 interface. On an organizational level the contents of the set are managed according to 56 the Mozilla CA policy. On a technical level the set is a binary software module. 57 | A cryptographic transaction, such as encryption or decryption related to a data exchange, 58 usually involves working with the X.509 certs of your communication partners (peer). It's also 59 required that you safely keep your own secret keys that belong to your own certificates. You 60 might want to protect the storage of your secret keys with PBE. You might decide to modify the 61 default trust provided by NSS. All of this requires storing, looking up, and retrieving data. 62 NSS simplifies performing these operations by offering storage and management APIs. NSS doesn't 63 require the programmer to manage individual files containing individual certificates or keys. 64 Instead, NSS offers to use its own database(s). Once you have imported certificates and keys 65 into the NSS database, you can easily look them up and use them again. 66 | Because of NSS's expectation to operate with an NSS database, it's mandatory that you perform 67 an initialization call, where you tell NSS which database you will be using. In the most simple 68 scenario, the programmer will provide a directory on your filesystem as a parameter to the init 69 function, and NSS is designed to do the rest. It will detect and open an existing database, or 70 it can create a new one. Alternatively, should you decide that you don't want to work with any 71 persistent recording of certificates, you may initialize NSS in a no-database mode. Usually, 72 NSS will flush all data to disk as soon as new data has been added to permanent storage. 73 Storage consists of multiple files: a key database file, which contains your secret keys, and a 74 certificate database file which contains the public portion of your own certificates, the 75 certificates of peers or CAs, and a list of trust decisions (such as to not trust a built-in 76 CA, or to explicitly trust other CAs). Examples for the database files are key3.db and 77 cert8.db, where the numbers are file version numbers. A third file contains the list of 78 external PKCS#11 modules that have been registered to be used by NSS. The file could be named 79 secmod.db, but in newer database generations a file named pkcs11.txt is used. 80 | Only NSS is allowed to access and manipulate these database files directly; a programmer using 81 NSS must go through the APIs offered by NSS to manipulate the data stored in these files. The 82 programmer's task is to initialize NSS with the required parameters (such as a database), and 83 NSS will then transparently manage the database files. 84 | Most of the time certificates and keys are supposed to be stored in the NSS database. 85 Therefore, after initial import or creation, the programmer usually doesn't deal with their raw 86 bytes. Instead, the programmer will use lookup functions, and NSS will provide an access handle 87 that will be subsequently used by the application's code. Those handles are reference counted. 88 NSS will usually create an in-memory (RAM) presentation of certificates, once a certificate has 89 been received from the network, read from disk, or looked up from the database, and prepare 90 in-memory data structures that contain the certificate's properties, as well as providing a 91 handle for the programmer to use. Once the application is done with a handle, it should be 92 released, allowing NSS to free the associated resources. When working with handles to private 93 keys it's usually difficult (and undesired) that an application gets access to the raw key 94 data; therefore it may be difficult to extract such data from NSS. The usual minimum 95 requirement is that private keys must be wrapped using a protective layer (such as 96 password-based encryption). The intention is to make it easier to review code for security. The 97 less code that has access to raw secret keys, the less code that must be reviewed. 98 | NSS has only limited functionality to look up raw keys. The preferred approach is to use 99 certificates, and to look up certificates by properties such as the contained subject name 100 (information that describes the owner of the certificate). For example, while NSS supports 101 random calculation (creation) of a new public/private key pair, it's difficult to work with 102 such a raw key pair. The usual approach is to create a certificate signing request (CSR) as 103 soon as an application is done with the creation step, which will have created a handle to the 104 key pair, and which can be used for the necessary related operations, like producing a 105 proof-of-ownership of the private key, which is usually required when submitting the public key 106 with a CSR to a CA. The usual follow up action is receiving a signed certificate from a CA. 107 (However, it's also possible to use NSS functionality to create a self-signed certificate, 108 which, however, usually won't be trusted by other parties.) Once received, it's sufficient to 109 tell NSS to import such a new certificate into the NSS database, and NSS will automatically 110 perform a lookup of the embedded public key, be able to find the associated private key, and 111 subsequently be able to treat it as a personal certificate. (A personal certificate is a 112 certificate for which the private key is in possession, and which could be used for signing 113 data or for decrypting data.) A unique nickname can/should be assigned to the certificate at 114 the time of import, which can later be used to easily identify and retrieve it. 115 | It's important to note that NSS requires strict cleanup for all handles returned by NSS. The 116 application should always call the appropriate dereference (destroy) functions once a handle is 117 no longer needed. This is particularly important for applications that might need to close a 118 database and reinitialize NSS using a different one, without restarting. Such an operation 119 might fail at runtime if data elements are still being referenced. 120 | In addition to the FreeBL, Softoken, and CKBI modules, there is an utility library for general 121 operations (e.g., encoding/decoding between data formats, a list of standardized object 122 identifiers (OID)). NSS has an SSL/TLS module that implements the Secure Sockets 123 Layer/Transport Layer Security network protocols, an S/MIME module that implements CMS 124 messaging used by secure email and some instant messaging implementations, a DBM library that 125 implements the classic database storage, and finally a core NSS library for the big set of 126 “everything else”. Newer generations of the database use the SQLite database to allow 127 concurrent access by multiple applications. 128 | All of the above are provided as shared libraries. The CRMF library, which is used to produce 129 certain kinds of certificate requests, is available as a library for static linking only. 130 | When dealing with certificates (X.509), file formats such as PKCS#12 (certificates and keys), 131 PKCS#7 (signed data), and message formats as CMS, we should mention ASN.1, which is a syntax 132 for storing structured data in a very efficient (small sized) presentation. It was originally 133 developed for telecommunication systems at times where it was critical to minimize data as much 134 as possible (although it still makes sense to use that principle today for good performance). 135 In order to process data available in the ASN.1 format, the usual approach is to parse it and 136 transfer it to a presentation that requires more space but is easier to work with, such as 137 (nested) C data structures. Over the time NSS has received three different ASN.1 parser 138 implementations, each having their own specific properties, advantages and disadvantages, which 139 is why all of them are still being used (nobody has yet dared to replace the older with the 140 newer ones because of risks for side effects). When using the ASN.1 parser(s), a template 141 definition is passed to the parser, which will analyze the ASN.1 data stream accordingly. The 142 templates are usually closely aligned to definitions found in RFC documents. 143 | A data block described as DER is usually in ASN.1 format. You must know which data you are 144 expecting, and use the correct template for parsing, based on the context of your software's 145 interaction. Data described as PEM is a base64 encoded presentation of DER, usually wrapped 146 between human readable BEGIN/END lines. NSS prefers the binary presentation, but is often 147 capable to use base64 or ASCII presentations, especially when importing data from files. A 148 recent development adds support for loading external PEM files that contain private keys, in a 149 software library called nss-pem, which is separately available, but should eventually become a 150 core part of NSS. 151 | Looking at the code level, NSS deals with blocks of raw data all the time. The common structure 152 to store such an untyped block is SECItem, which contains a size and an untyped C pointer 153 variable. 154 | When dealing with memory, NSS makes use of arenas, which are an attempt to simplify management 155 with the limited offerings of C (because there are no destructors). The idea is to group 156 multiple memory allocations in order to simplify cleanup. Performing an operation often 157 involves allocating many individual data items, and the code might be required to abort a task 158 at many positions in the logic. An arena is requested once processing of a task starts, and all 159 memory allocations that are logically associated to that task are requested from the associated 160 arena. The implementation of arenas makes sure that all individual memory blocks are tracked. 161 Once a task is done, regardless whether it completed or was aborted, the programmer simply 162 needs to release the arena, and all individually allocated blocks will be released 163 automatically. Often freeing is combined with immediately erasing (zeroing, zfree) the memory 164 associated to the arena, in order to make it more difficult for attackers to extract keys from 165 a memory dump. 166 | NSS uses many C data structures. Often NSS has multiple implementations for the same or similar 167 concepts. For example, there are multiple presentations of certificates, and the NSS internals 168 (and sometimes even the application using NSS) might have to convert between them. 169 | Key responsibilites of NSS are verification of signatures and certificates. In order to verify 170 a digital signature, we have to look at the application data (e.g., a document that was 171 signed), the signature data block (the digital signature), and a public key (as found in a 172 certificate that is believed to be the signer, e.g., identified by metadata received together 173 with the signature). The signature is verified if it can be shown that the signature data block 174 must have been produced by the owner of the public key (because only that owner has the 175 associated private key). 176 | Verifying a certificate (A) requires some additional steps. First, you must identify the 177 potential signer (B) of a certificate (A). This is done by reading the “issuer name” attribute 178 of a certificate (A), and trying to find that issuer certificate (B) (by looking for a 179 certificate that uses that name as its “subject name”). Then you attempt to verify the 180 signature found in (A) using the public key found in (B). It might be necessary to try multiple 181 certificates (B1, B2, ...) each having the same subject name. 182 | After succeeding, it might be necessary to repeat this procedure recursively. The goal is to 183 eventually find a certificate B (or C or ...) that has an appropriate trust assigned (e.g., 184 because it can be found in the CKBI module and the user hasn't made any overriding trust 185 decisions, or it can be found in a NSS database file managed by the user or by the local 186 environment). 187 | After having successfully verified the signatures in a (chain of) issuer certificate(s), we're 188 still not done with verifying the certificate A. In a PKI it's suggested/required to perform 189 additional checks. For example: Certificates were valid at the time the signature was made, 190 name in certificates matches the expected signer (check subject name, common name, email, based 191 on application), the trust restrictions recorded inside the certificate (extensions) permit the 192 use (e.g., encryption might be allowed, but not signing), and based on environment/application 193 policy it might be required to perform a revocation check (OCSP or CRL), that asks the 194 issuer(s) of the certificates whether there have been events that made it necessary to revoke 195 the trust (revoke the validity of the cert). 196 | Trust anchors contained in the CKBI module are usually self signed, which is defined as having 197 identical subject name and issuer name fields. If a self-signed certificate is marked as 198 explicitly trusted, NSS will skip checking the self-signature for validity. 199 | NSS has multiple APIs to perform verification of certificates. There is a classic engine that 200 is very stable and works fine in all simple scenarios, for example if all (B) candidate issuer 201 certificates have the same subject and issuer names and differ by validity period; however, it 202 works only in a limited amount of more advanced scenarios. Unfortunately, the world of 203 certificates has become more complex in the recent past. New Certificate Authorities enter the 204 global PKI market, and in order to get started with their business, they might make deals with 205 established CAs and receive so-called cross-signing-certificates. As a result, when searching 206 for a trust path from (A) to a trusted anchor (root) certificate (Z), the set of candidate 207 issuer certificates might have different issuer names (referring to the second or higher issuer 208 level). As a consequence, it will be necessary to try multiple different alternative routes 209 while searching for (Z), in a recursive manner. Only the newer verification engine (internally 210 named libPKIX) is capable of doing that properly. 211 | It's worth mentioning the Extended Validation (EV) principle, which is an effort by software 212 vendors and CAs to define a stricter set of rules for issuing certificates for web site 213 certificates. Instead of simply verifying that the requester of a certificate is in control of 214 an administrative email address at the desired web site's domain, it's required that the CA 215 performs a verification of real world identity documents (such as a company registration 216 document with the country's authority), and it's also required that a browser software performs 217 a revocation check with the CA, prior to granting validity to the certificate. In order to 218 distinguish an EV certificate, CAs will embed a policy OID in the certificate, and the browser 219 is expected to verify that a trust chain permits the end entity (EE) certificate to make use of 220 the policy. Only the APIs of the newer libPKIX engine are capable of performing a policy 221 verification. 222 | That's a good opportunity to talk about SSL/TLS connections to servers in general (not just EV, 223 not just websites). Whenever this document mentions SSL, it refers to either SSL or TLS. (TLS 224 is a newer version of SSL with enhanced features.) 225 | When establishing an SSL connection to a server, (at least) a server certificate (and its trust 226 chain) is exchanged from the server to the client (e.g., the browser), and the client verifies 227 that the certificate can be verified (including matching the name of the expected destination 228 server). Another part of the handshake between both parties is a key exchange. Because public 229 key encryption is more expensive (more calculations required) than symmetric encryption (where 230 both parties use the same key), a key agreement protocol will be executed, where the public and 231 private keys are used to proof and verify the exchanged initial information. Once the key 232 agreement is done, a symmetric encryption will be used (until a potential re-handshake on an 233 existing channel). The combination of the hash and encryption algorithms used for a SSL 234 connection is called a cipher suite. 235 | NSS ships with a set of cipher suites that it supports at a technical level. In addition, NSS 236 ships with a default policy that defines which cipher suites are enabled by default. An 237 application is able to modify the policy used at program runtime, by using function calls to 238 modify the set of enabled cipher suites. 239 | If a programmer wants to influence how NSS verifies certificates or how NSS verifies the data 240 presented in a SSL connection handshake, it is possible to register application-defined 241 callback functions which will be called by NSS at the appropriate point of time, and which can 242 be used to override the decisions made by NSS. 243 | If you would like to use NSS as a toolkit that implements SSL, remember that you must init NSS 244 first. But if you don't care about modifying the default trust permanently (recorded on disk), 245 you can use the no-database init calls. When creating the network socket for data exchange, 246 note that you must use the operating system independent APIs provided by NSPR and NSS. It might 247 be interesting to mention a property of the NSPR file descriptors, which are stacked in layers. 248 This means you can define multiple layers that are involved in data processing. A file 249 descriptor has a pointer to the first layer handling the data. That layer has a pointer to a 250 potential second layer, which might have another pointer to a third layer, etc. Each layer 251 defines its own functions for the open/close/read/write/poll/select (etc.) functions. When 252 using an SSL network connection, you'll already have two layers, the basic NSPR layer and an 253 SSL library layer. The Mozilla applications define a third layer where application specific 254 processing is performed. You can find more details in the NSPR reference documents. 255 | NSS occassionally has to create outbound network connections, in addition to the connections 256 requested by the application. Examples are retrieving OCSP (Online Certificate Status Protocol) 257 information or downloading a CRL (Certificate Revocation List). However, NSS doesn't have an 258 implementation to work with network proxies. If you must support proxies in your application, 259 you are able to register your own implementation of an http request callback interface, and NSS 260 can use your application code that supports proxies. 261 | When using hashing, encryption, and decryption functions, it is possible to stream data (as 262 opposed to operating on a large buffer). Create a context handle while providing all the 263 parameters required for the operation, then call an “update” function multiple times to pass 264 subsets of the input to NSS. The data will be processed and either returned directly or sent to 265 a callback function registered in the context. When done, you call a finalization function that 266 will flush out any pending data and free the resources. 267 | This line is a placeholder for future sections that should explain how libpkix works and is 268 designed. 269 | If you want to work with NSS, it's often helpful to use the command line utilities that are 270 provided by the NSS developers. There are tools for managing NSS databases, for dumping or 271 verifying certificates, for registering PKCS#11 modules with a database, for processing CMS 272 encrypted/signed messages, etc. 273 | For example, if you wanted to create your own pair of keys and request a new certificate from a 274 CA, you could use certutil to create an empty database, then use certutil to operate on your 275 database and create a certificate request (which involves creating the desired key pair) and 276 export it to a file, submit the request file to the CA, receive the file from the CA, and 277 import the certificate into your database. You should assign a good nickname to a certificate 278 when importing it, making it easier for you to refer to it later. 279 | It should be noted that the first database format that can be accessed simultaneously by 280 multiple applications is key4.db/cert9.db – database files with lower numbers will most likely 281 experience unrecoverable corruption if you access them with multiple applications at the same 282 time. In other words, if your browser or your server operates on an older NSS database format, 283 don't use the NSS tools to operate on it while the other software is executing. At the time of 284 writing NSS and the Mozilla applications still use the older database file format by default, 285 where each application has its own NSS database. 286 | If you require a copy of a certificate stored in an NSS database, including its private key, 287 you can use pk12util to export it to the PKCS#12 file format. If you require it in PEM format, 288 you could use the openssl pkcs12 command (that's not NSS) to convert the PKCS#12 file to PEM. 289 | This line is a placeholder for how to prepare a database, how to dump a cert, and how to 290 convert data. 291 | You might have been motivated to work with NSS because it is used by the Mozilla applications 292 such as Firefox, Thunderbird, etc. If you build the Mozilla application, it will automatically 293 build the NSS library, too. However, if you want to work with the NSS command line tools, you 294 will have to follow the standalone NSS build instructions, and build NSS outside of the Mozilla 295 application sources. 296 | The key database file will contain at least one symmetric key, which NSS will automatically 297 create on demand, and which will be used to protect your secret (private) keys. The symmetric 298 key can be protected with PBE by setting a master password on the database. As soon as you set 299 a master password, an attacker stealing your key database will no longer be able to get access 300 to your private key, unless the attacker would also succeed in stealing the master password. 301 | Now you might be interest in how to get the 302 :ref:`mozilla_projects_nss_nss_sources_building_testing`