Scheduling.h (43972B)
1 /* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*- 2 * vim: set ts=8 sts=2 et sw=2 tw=80: 3 * This Source Code Form is subject to the terms of the Mozilla Public 4 * License, v. 2.0. If a copy of the MPL was not distributed with this 5 * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ 6 7 /* 8 * [SMDOC] GC Scheduling 9 * 10 * GC Scheduling Overview 11 * ====================== 12 * 13 * See also GC scheduling from Firefox's perspective here: 14 * https://searchfox.org/mozilla-central/source/dom/base/CCGCScheduler.cpp 15 * 16 * Scheduling GC's in SpiderMonkey/Firefox is tremendously complicated because 17 * of the large number of subtle, cross-cutting, and widely dispersed factors 18 * that must be taken into account. A summary of some of the more important 19 * factors follows. 20 * 21 * Cost factors: 22 * 23 * * GC too soon and we'll revisit an object graph almost identical to the 24 * one we just visited; since we are unlikely to find new garbage, the 25 * traversal will be largely overhead. We rely heavily on external factors 26 * to signal us that we are likely to find lots of garbage: e.g. "a tab 27 * just got closed". 28 * 29 * * GC too late and we'll run out of memory to allocate (e.g. Out-Of-Memory, 30 * hereafter simply abbreviated to OOM). If this happens inside 31 * SpiderMonkey we may be able to recover, but most embedder allocations 32 * will simply crash on OOM, even if the GC has plenty of free memory it 33 * could surrender. 34 * 35 * * Memory fragmentation: if we fill the process with GC allocations, a 36 * request for a large block of contiguous memory may fail because no 37 * contiguous block is free, despite having enough memory available to 38 * service the request. 39 * 40 * * Management overhead: if our GC heap becomes large, we create extra 41 * overhead when managing the GC's structures, even if the allocations are 42 * mostly unused. 43 * 44 * Heap Management Factors: 45 * 46 * * GC memory: The GC has its own allocator that it uses to make fixed size 47 * allocations for GC managed things. In cases where the GC thing requires 48 * larger or variable sized memory to implement itself, it is responsible 49 * for using the system heap. 50 * 51 * * C Heap Memory: Rather than allowing for large or variable allocations, 52 * the SpiderMonkey GC allows GC things to hold pointers to C heap memory. 53 * It is the responsibility of the thing to free this memory with a custom 54 * finalizer (with the sole exception of NativeObject, which knows about 55 * slots and elements for performance reasons). C heap memory has different 56 * performance and overhead tradeoffs than GC internal memory, which need 57 * to be considered with scheduling a GC. 58 * 59 * Application Factors: 60 * 61 * * Most applications allocate heavily at startup, then enter a processing 62 * stage where memory utilization remains roughly fixed with a slower 63 * allocation rate. This is not always the case, however, so while we may 64 * optimize for this pattern, we must be able to handle arbitrary 65 * allocation patterns. 66 * 67 * Other factors: 68 * 69 * * Other memory: This is memory allocated outside the purview of the GC. 70 * Data mapped by the system for code libraries, data allocated by those 71 * libraries, data in the JSRuntime that is used to manage the engine, 72 * memory used by the embedding that is not attached to a GC thing, memory 73 * used by unrelated processes running on the hardware that use space we 74 * could otherwise use for allocation, etc. While we don't have to manage 75 * it, we do have to take it into account when scheduling since it affects 76 * when we will OOM. 77 * 78 * * Physical Reality: All real machines have limits on the number of bits 79 * that they are physically able to store. While modern operating systems 80 * can generally make additional space available with swapping, at some 81 * point there are simply no more bits to allocate. There is also the 82 * factor of address space limitations, particularly on 32bit machines. 83 * 84 * * Platform Factors: Each OS makes use of wildly different memory 85 * management techniques. These differences result in different performance 86 * tradeoffs, different fragmentation patterns, and different hard limits 87 * on the amount of physical and/or virtual memory that we can use before 88 * OOMing. 89 * 90 * 91 * Reasons for scheduling GC 92 * ------------------------- 93 * 94 * While code generally takes the above factors into account in only an ad-hoc 95 * fashion, the API forces the user to pick a "reason" for the GC. We have a 96 * bunch of JS::GCReason reasons in GCAPI.h. These fall into a few categories 97 * that generally coincide with one or more of the above factors. 98 * 99 * Embedding reasons: 100 * 101 * 1) Do a GC now because the embedding knows something useful about the 102 * zone's memory retention state. These are GCReasons like LOAD_END, 103 * PAGE_HIDE, SET_NEW_DOCUMENT, DOM_UTILS. Mostly, Gecko uses these to 104 * indicate that a significant fraction of the scheduled zone's memory is 105 * probably reclaimable. 106 * 107 * 2) Do some known amount of GC work now because the embedding knows now is 108 * a good time to do a long, unblockable operation of a known duration. 109 * These are INTER_SLICE_GC and REFRESH_FRAME. 110 * 111 * Correctness reasons: 112 * 113 * 3) Do a GC now because correctness depends on some GC property. For 114 * example, CC_FORCED is where the embedding requires the mark bits to be 115 * set correctly. Also, EVICT_NURSERY where we need to work on the tenured 116 * heap. 117 * 118 * 4) Do a GC because we are shutting down: e.g. SHUTDOWN_CC or DESTROY_*. 119 * 120 * 5) Do a GC because a compartment was accessed between GC slices when we 121 * would have otherwise discarded it. We have to do a second GC to clean 122 * it up: e.g. COMPARTMENT_REVIVED. 123 * 124 * Emergency Reasons: 125 * 126 * 6) Do an all-zones, non-incremental GC now because the embedding knows it 127 * cannot wait: e.g. MEM_PRESSURE. 128 * 129 * 7) OOM when fetching a new Chunk results in a LAST_DITCH GC. 130 * 131 * Heap Size Limitation Reasons: 132 * 133 * 8) Do an incremental, zonal GC with reason MAYBEGC when we discover that 134 * the gc's allocated size is approaching the current trigger. This is 135 * called MAYBEGC because we make this check in the MaybeGC function. 136 * MaybeGC gets called at the top of the main event loop. Normally, it is 137 * expected that this callback will keep the heap size limited. It is 138 * relatively inexpensive, because it is invoked with no JS running and 139 * thus few stack roots to scan. For this reason, the GC's "trigger" bytes 140 * is less than the GC's "max" bytes as used by the trigger below. 141 * 142 * 9) Do an incremental, zonal GC with reason MAYBEGC when we go to allocate 143 * a new GC thing and find that the GC heap size has grown beyond the 144 * configured maximum (JSGC_MAX_BYTES). We trigger this GC by returning 145 * nullptr and then calling maybeGC at the top level of the allocator. 146 * This is then guaranteed to fail the "size greater than trigger" check 147 * above, since trigger is always less than max. After performing the GC, 148 * the allocator unconditionally returns nullptr to force an OOM exception 149 * is raised by the script. 150 * 151 * Note that this differs from a LAST_DITCH GC where we actually run out 152 * of memory (i.e., a call to a system allocator fails) when trying to 153 * allocate. Unlike above, LAST_DITCH GC only happens when we are really 154 * out of memory, not just when we cross an arbitrary trigger; despite 155 * this, it may still return an allocation at the end and allow the script 156 * to continue, if the LAST_DITCH GC was able to free up enough memory. 157 * 158 * 10) Do a GC under reason ALLOC_TRIGGER when we are over the GC heap trigger 159 * limit, but in the allocator rather than in a random call to maybeGC. 160 * This occurs if we allocate too much before returning to the event loop 161 * and calling maybeGC; this is extremely common in benchmarks and 162 * long-running Worker computations. Note that this uses a wildly 163 * different mechanism from the above in that it sets the interrupt flag 164 * and does the GC at the next loop head, before the next alloc, or 165 * maybeGC. The reason for this is that this check is made after the 166 * allocation and we cannot GC with an uninitialized thing in the heap. 167 * 168 * 11) Do an incremental, zonal GC with reason TOO_MUCH_MALLOC when the total 169 * amount of malloced memory is greater than the malloc trigger limit for the 170 * zone. 171 * 172 * 173 * Size Limitation Triggers Explanation 174 * ------------------------------------ 175 * 176 * The GC internally is entirely unaware of the context of the execution of 177 * the mutator. It sees only: 178 * 179 * A) Allocated size: this is the amount of memory currently requested by the 180 * mutator. This quantity is monotonically increasing: i.e. the allocation 181 * rate is always >= 0. It is also easy for the system to track. 182 * 183 * B) Retained size: this is the amount of memory that the mutator can 184 * currently reach. Said another way, it is the size of the heap 185 * immediately after a GC (modulo background sweeping). This size is very 186 * costly to know exactly and also extremely hard to estimate with any 187 * fidelity. 188 * 189 * For reference, a common allocated vs. retained graph might look like: 190 * 191 * | ** ** 192 * | ** ** * ** 193 * | ** * ** * ** 194 * | * ** * ** * ** 195 * | ** ** * ** * ** 196 * s| * * ** ** + + ** 197 * i| * * * + + + + + 198 * z| * * * + + + + + 199 * e| * **+ 200 * | * + 201 * | * + 202 * | * + 203 * | * + 204 * | * + 205 * |*+ 206 * +-------------------------------------------------- 207 * time 208 * *** = allocated 209 * +++ = retained 210 * 211 * Note that this is a bit of a simplification 212 * because in reality we track malloc and GC heap 213 * sizes separately and have a different level of 214 * granularity and accuracy on each heap. 215 * 216 * This presents some obvious implications for Mark-and-Sweep collectors. 217 * Namely: 218 * -> t[marking] ~= size[retained] 219 * -> t[sweeping] ~= size[allocated] - size[retained] 220 * 221 * In a non-incremental collector, maintaining low latency and high 222 * responsiveness requires that total GC times be as low as possible. Thus, 223 * in order to stay responsive when we did not have a fully incremental 224 * collector, our GC triggers were focused on minimizing collection time. 225 * Furthermore, since size[retained] is not under control of the GC, all the 226 * GC could do to control collection times was reduce sweep times by 227 * minimizing size[allocated], per the equation above. 228 * 229 * The result of the above is GC triggers that focus on size[allocated] to 230 * the exclusion of other important factors and default heuristics that are 231 * not optimal for a fully incremental collector. On the other hand, this is 232 * not all bad: minimizing size[allocated] also minimizes the chance of OOM 233 * and sweeping remains one of the hardest areas to further incrementalize. 234 * 235 * EAGER_ALLOC_TRIGGER 236 * ------------------- 237 * Occurs when we return to the event loop and find our heap is getting 238 * largish, but before t[marking] OR t[sweeping] is too large for a 239 * responsive non-incremental GC. This is intended to be the common case 240 * in normal web applications: e.g. we just finished an event handler and 241 * the few objects we allocated when computing the new whatzitz have 242 * pushed us slightly over the limit. After this GC we rescale the new 243 * EAGER_ALLOC_TRIGGER trigger to 150% of size[retained] so that our 244 * non-incremental GC times will always be proportional to this size 245 * rather than being dominated by sweeping. 246 * 247 * As a concession to mutators that allocate heavily during their startup 248 * phase, we have a highFrequencyGCMode that ups the growth rate to 300% 249 * of the current size[retained] so that we'll do fewer longer GCs at the 250 * end of the mutator startup rather than more, smaller GCs. 251 * 252 * Assumptions: 253 * -> Responsiveness is proportional to t[marking] + t[sweeping]. 254 * -> size[retained] is proportional only to GC allocations. 255 * 256 * ALLOC_TRIGGER (non-incremental) 257 * ------------------------------- 258 * If we do not return to the event loop before getting all the way to our 259 * gc trigger bytes then MAYBEGC will never fire. To avoid OOMing, we 260 * succeed the current allocation and set the script interrupt so that we 261 * will (hopefully) do a GC before we overflow our max and have to raise 262 * an OOM exception for the script. 263 * 264 * Assumptions: 265 * -> Common web scripts will return to the event loop before using 266 * 10% of the current triggerBytes worth of GC memory. 267 * 268 * ALLOC_TRIGGER (incremental) 269 * --------------------------- 270 * In practice the above trigger is rough: if a website is just on the 271 * cusp, sometimes it will trigger a non-incremental GC moments before 272 * returning to the event loop, where it could have done an incremental 273 * GC. Thus, we recently added an incremental version of the above with a 274 * substantially lower threshold, so that we have a soft limit here. If 275 * IGC can collect faster than the allocator generates garbage, even if 276 * the allocator does not return to the event loop frequently, we should 277 * not have to fall back to a non-incremental GC. 278 * 279 * INCREMENTAL_TOO_SLOW 280 * -------------------- 281 * Do a full, non-incremental GC if we overflow ALLOC_TRIGGER during an 282 * incremental GC. When in the middle of an incremental GC, we suppress 283 * our other triggers, so we need a way to backstop the IGC if the 284 * mutator allocates faster than the IGC can clean things up. 285 * 286 * TOO_MUCH_MALLOC 287 * --------------- 288 * Performs a GC before size[allocated] - size[retained] gets too large 289 * for non-incremental sweeping to be fast in the case that we have 290 * significantly more malloc allocation than GC allocation. This is meant 291 * to complement MAYBEGC triggers. We track this by counting malloced 292 * bytes; the counter gets reset at every GC since we do not always have a 293 * size at the time we call free. Because of this, the malloc heuristic 294 * is, unfortunately, not usefully able to augment our other GC heap 295 * triggers and is limited to this singular heuristic. 296 * 297 * Assumptions: 298 * -> EITHER size[allocated_by_malloc] ~= size[allocated_by_GC] 299 * OR time[sweeping] ~= size[allocated_by_malloc] 300 * -> size[retained] @ t0 ~= size[retained] @ t1 301 * i.e. That the mutator is in steady-state operation. 302 * 303 * LAST_DITCH_GC 304 * ------------- 305 * Does a GC because we are out of memory. 306 * 307 * Assumptions: 308 * -> size[retained] < size[available_memory] 309 */ 310 311 #ifndef gc_Scheduling_h 312 #define gc_Scheduling_h 313 314 #include "mozilla/Atomics.h" 315 #include "mozilla/DebugOnly.h" 316 #include "mozilla/Maybe.h" 317 #include "mozilla/TimeStamp.h" 318 319 #include "gc/GCEnum.h" 320 #include "js/AllocPolicy.h" 321 #include "js/GCAPI.h" 322 #include "js/HashTable.h" 323 #include "js/HeapAPI.h" 324 #include "threading/LockGuard.h" 325 #include "threading/Mutex.h" 326 #include "threading/ProtectedData.h" 327 328 // Macro to define scheduling tunables for GC parameters. Expands its argument 329 // repeatedly with the following arguments: 330 // - key: the JSGCParamKey value for this parameter 331 // - type: the storage type 332 // - name: the name of GCSchedulingTunables getter method 333 // - convert: a helper class defined in Scheduling.cpp that provides 334 // conversion methods 335 // - check: a helper function defined in Scheduling.cppto check the value is 336 // valid 337 // - default: the initial value and that assigned by resetParameter 338 #define FOR_EACH_GC_TUNABLE(_) \ 339 /* \ 340 * JSGC_MAX_BYTES \ 341 * \ 342 * Maximum nominal heap before last ditch GC. \ 343 */ \ 344 _(JSGC_MAX_BYTES, size_t, gcMaxBytes, ConvertSize, NoCheck, 0xffffffff) \ 345 \ 346 /* \ 347 * JSGC_MIN_NURSERY_BYTES \ 348 * JSGC_MAX_NURSERY_BYTES \ 349 * \ 350 * Minimum and maximum nursery size for each runtime. \ 351 */ \ 352 _(JSGC_MIN_NURSERY_BYTES, size_t, gcMinNurseryBytes, ConvertNurseryBytes, \ 353 CheckNurserySize, 256 * 1024) \ 354 _(JSGC_MAX_NURSERY_BYTES, size_t, gcMaxNurseryBytes, ConvertNurseryBytes, \ 355 CheckNurserySize, JS::DefaultNurseryMaxBytes) \ 356 \ 357 /* \ 358 * JSGC_ALLOCATION_THRESHOLD \ 359 * \ 360 * \ 361 * The base value used to compute zone->threshold.bytes(). When \ 362 * gcHeapSize.bytes() exceeds threshold.bytes() for a zone, the zone may be \ 363 * scheduled for a GC, depending on the exact circumstances. \ 364 */ \ 365 _(JSGC_ALLOCATION_THRESHOLD, size_t, gcZoneAllocThresholdBase, ConvertMB, \ 366 NoCheck, 27 * 1024 * 1024) \ 367 \ 368 /* \ 369 * JSGC_SMALL_HEAP_SIZE_MAX \ 370 * JSGC_LARGE_HEAP_SIZE_MIN \ 371 * \ 372 * Used to classify heap sizes into one of small, medium or large. This \ 373 * affects the calcuation of the incremental GC trigger and the heap growth \ 374 * factor in high frequency GC mode. \ 375 */ \ 376 _(JSGC_SMALL_HEAP_SIZE_MAX, size_t, smallHeapSizeMaxBytes, ConvertMB, \ 377 NoCheck, 100 * 1024 * 1024) \ 378 _(JSGC_LARGE_HEAP_SIZE_MIN, size_t, largeHeapSizeMinBytes, ConvertMB, \ 379 CheckNonZero, 500 * 1024 * 1024) \ 380 \ 381 /* \ 382 * JSGC_SMALL_HEAP_INCREMENTAL_LIMIT \ 383 * JSGC_LARGE_HEAP_INCREMENTAL_LIMIT \ 384 * \ 385 * Multiple of threshold.bytes() which triggers a non-incremental GC. \ 386 * \ 387 * The small heap limit must be at least 1.7 to maintain performance on \ 388 * splay-latency. \ 389 */ \ 390 _(JSGC_SMALL_HEAP_INCREMENTAL_LIMIT, double, smallHeapIncrementalLimit, \ 391 ConvertTimes100, CheckIncrementalLimit, 1.70) \ 392 _(JSGC_LARGE_HEAP_INCREMENTAL_LIMIT, double, largeHeapIncrementalLimit, \ 393 ConvertTimes100, CheckIncrementalLimit, 1.10) \ 394 \ 395 /* \ 396 * JSGC_HIGH_FREQUENCY_TIME_LIMIT \ 397 * \ 398 * We enter high-frequency mode if we GC a twice within this many \ 399 * millisconds. \ 400 */ \ 401 _(JSGC_HIGH_FREQUENCY_TIME_LIMIT, mozilla::TimeDuration, \ 402 highFrequencyThreshold, ConvertMillis, NoCheck, \ 403 mozilla::TimeDuration::FromSeconds(1)) \ 404 \ 405 /* \ 406 * JSGC_LOW_FREQUENCY_HEAP_GROWTH \ 407 * \ 408 * When not in |highFrequencyGC| mode, this is the global (stored per-zone) \ 409 * "HeapGrowthFactor". \ 410 */ \ 411 _(JSGC_LOW_FREQUENCY_HEAP_GROWTH, double, lowFrequencyHeapGrowth, \ 412 ConvertTimes100, CheckHeapGrowth, 1.5) \ 413 \ 414 /* \ 415 * JSGC_HIGH_FREQUENCY_SMALL_HEAP_GROWTH \ 416 * JSGC_HIGH_FREQUENCY_LARGE_HEAP_GROWTH \ 417 * \ 418 * When in the |highFrequencyGC| mode, these parameterize the per-zone \ 419 * "HeapGrowthFactor" computation. \ 420 */ \ 421 _(JSGC_HIGH_FREQUENCY_SMALL_HEAP_GROWTH, double, \ 422 highFrequencySmallHeapGrowth, ConvertTimes100, CheckHeapGrowth, 3.0) \ 423 _(JSGC_HIGH_FREQUENCY_LARGE_HEAP_GROWTH, double, \ 424 highFrequencyLargeHeapGrowth, ConvertTimes100, CheckHeapGrowth, 1.5) \ 425 \ 426 /* \ 427 * JSGC_MALLOC_THRESHOLD_BASE \ 428 * \ 429 * The base value used to compute the GC trigger for malloc allocated \ 430 * memory. \ 431 */ \ 432 _(JSGC_MALLOC_THRESHOLD_BASE, size_t, mallocThresholdBase, ConvertMB, \ 433 NoCheck, 38 * 1024 * 1024) \ 434 \ 435 /* \ 436 * Number of bytes to allocate between incremental slices in GCs triggered \ 437 * by the zone allocation threshold. \ 438 */ \ 439 _(JSGC_ZONE_ALLOC_DELAY_KB, size_t, zoneAllocDelayBytes, ConvertKB, \ 440 CheckNonZero, 1024 * 1024) \ 441 \ 442 /* \ 443 * JSGC_URGENT_THRESHOLD_MB \ 444 * \ 445 * The point before reaching the non-incremental limit at which to start \ 446 * increasing the slice budget and frequency of allocation triggered slices. \ 447 */ \ 448 _(JSGC_URGENT_THRESHOLD_MB, size_t, urgentThresholdBytes, ConvertMB, \ 449 NoCheck, 16 * 1024 * 1024) \ 450 \ 451 /* \ 452 * JSGC_NURSERY_EAGER_COLLECTION_THRESHOLD_KB \ 453 * JSGC_NURSERY_EAGER_COLLECTION_THRESHOLD_PERCENT \ 454 * JSGC_NURSERY_EAGER_COLLECTION_TIMEOUT_MS \ 455 * \ 456 * JS::MaybeRunNurseryCollection will run a minor GC if the free space falls \ 457 * below a threshold or if it hasn't been collected for too long. \ 458 * \ 459 * To avoid making this too eager, two thresholds must be met. The free \ 460 * space must fall below a size threshold and the fraction of free space \ 461 * remaining must also fall below a threshold. \ 462 * \ 463 * See Nursery::wantEagerCollection() for more details. \ 464 */ \ 465 _(JSGC_NURSERY_EAGER_COLLECTION_THRESHOLD_KB, size_t, \ 466 nurseryEagerCollectionThresholdBytes, ConvertKB, NoCheck, ChunkSize / 4) \ 467 _(JSGC_NURSERY_EAGER_COLLECTION_THRESHOLD_PERCENT, double, \ 468 nurseryEagerCollectionThresholdPercent, ConvertTimes100, \ 469 CheckNonZeroUnitRange, 0.25) \ 470 _(JSGC_NURSERY_EAGER_COLLECTION_TIMEOUT_MS, mozilla::TimeDuration, \ 471 nurseryEagerCollectionTimeout, ConvertMillis, NoCheck, \ 472 mozilla::TimeDuration::FromSeconds(5)) \ 473 \ 474 /* \ 475 * JSGC_BALANCED_HEAP_LIMITS_ENABLED \ 476 * JSGC_HEAP_GROWTH_FACTOR \ 477 */ \ 478 _(JSGC_BALANCED_HEAP_LIMITS_ENABLED, bool, balancedHeapLimitsEnabled, \ 479 ConvertBool, NoCheck, false) \ 480 _(JSGC_HEAP_GROWTH_FACTOR, double, heapGrowthFactor, ConvertDouble, NoCheck, \ 481 50.0) \ 482 \ 483 /* \ 484 * JSGC_MIN_LAST_DITCH_GC_PERIOD \ 485 * \ 486 * Last ditch GC is skipped if allocation failure occurs less than this many \ 487 * seconds from the previous one. \ 488 */ \ 489 _(JSGC_MIN_LAST_DITCH_GC_PERIOD, mozilla::TimeDuration, \ 490 minLastDitchGCPeriod, ConvertSeconds, NoCheck, \ 491 TimeDuration::FromSeconds(60)) \ 492 \ 493 /* \ 494 * JSGC_PARALLEL_MARKING_THRESHOLD_MB \ 495 */ \ 496 _(JSGC_PARALLEL_MARKING_THRESHOLD_MB, size_t, parallelMarkingThresholdBytes, \ 497 ConvertMB, NoCheck, 4 * 1024 * 1024) \ 498 \ 499 /* \ 500 * JSGC_GENERATE_MISSING_ALLOC_SITES \ 501 */ \ 502 _(JSGC_GENERATE_MISSING_ALLOC_SITES, bool, generateMissingAllocSites, \ 503 ConvertBool, NoCheck, false) \ 504 \ 505 /* \ 506 * JSGC_NURSERY_MAX_TIME_GOAL_MS \ 507 */ \ 508 _(JSGC_NURSERY_MAX_TIME_GOAL_MS, mozilla::TimeDuration, \ 509 nurseryMaxTimeGoalMS, ConvertMillis, NoCheck, \ 510 mozilla::TimeDuration::FromMilliseconds(4)) \ 511 \ 512 /* \ 513 * JSGC_STORE_BUFFER_ENTRIES \ 514 * JSGC_STORE_BUFFER_SCALING \ 515 */ \ 516 _(JSGC_STORE_BUFFER_ENTRIES, size_t, storeBufferEntries, ConvertSize, \ 517 CheckNonZero, 16384) \ 518 _(JSGC_STORE_BUFFER_SCALING, double, storeBufferScaling, ConvertTimes100, \ 519 NoCheck, 0.25) 520 521 namespace js { 522 523 class ZoneAllocator; 524 525 namespace gc { 526 527 struct Cell; 528 529 /* 530 * Default settings for tuning the GC. Some of these can be set at runtime, 531 * This list is not complete, some tuning parameters are not listed here. 532 * 533 * If you change the values here, please also consider changing them in 534 * modules/libpref/init/all.js where they are duplicated for the Firefox 535 * preferences. 536 */ 537 namespace TuningDefaults { 538 539 /* JSGC_MIN_EMPTY_CHUNK_COUNT */ 540 static const uint32_t MinEmptyChunkCount = 1; 541 542 /* JSGC_SLICE_TIME_BUDGET_MS */ 543 static const int64_t DefaultTimeBudgetMS = 0; // Unlimited by default. 544 545 /* JSGC_INCREMENTAL_GC_ENABLED */ 546 static const bool IncrementalGCEnabled = false; 547 548 /* JSGC_PER_ZONE_GC_ENABLED */ 549 static const bool PerZoneGCEnabled = false; 550 551 /* JSGC_COMPACTING_ENABLED */ 552 static const bool CompactingEnabled = true; 553 554 /* JSGC_NURSERY_ENABLED */ 555 static const bool NurseryEnabled = true; 556 557 /* JSGC_PARALLEL_MARKING_ENABLED */ 558 static const bool ParallelMarkingEnabled = false; 559 560 /* JSGC_INCREMENTAL_WEAKMAP_ENABLED */ 561 static const bool IncrementalWeakMapMarkingEnabled = true; 562 563 /* JSGC_SEMISPACE_NURSERY_ENABLED */ 564 static const bool SemispaceNurseryEnabled = false; 565 566 /* JSGC_HELPER_THREAD_RATIO */ 567 static const double HelperThreadRatio = 0.5; 568 569 /* JSGC_MAX_HELPER_THREADS */ 570 static const size_t MaxHelperThreads = 8; 571 572 /* JSGC_MAX_MARKING_THREADS */ 573 static const size_t MaxMarkingThreads = 2; 574 575 } // namespace TuningDefaults 576 577 /* 578 * Encapsulates all of the GC tunables. These are effectively constant and 579 * should only be modified by setParameter. 580 */ 581 class GCSchedulingTunables { 582 #define DEFINE_TUNABLE_FIELD(key, type, name, convert, check, default) \ 583 MainThreadOrGCTaskData<type> name##_; 584 FOR_EACH_GC_TUNABLE(DEFINE_TUNABLE_FIELD) 585 #undef DEFINE_TUNABLE_FIELD 586 587 public: 588 GCSchedulingTunables(); 589 590 #define DEFINE_TUNABLE_ACCESSOR(key, type, name, convert, check, default) \ 591 type name() const { return name##_; } 592 FOR_EACH_GC_TUNABLE(DEFINE_TUNABLE_ACCESSOR) 593 #undef DEFINE_TUNABLE_ACCESSOR 594 595 uint32_t getParameter(JSGCParamKey key); 596 [[nodiscard]] bool setParameter(JSGCParamKey key, uint32_t value); 597 void resetParameter(JSGCParamKey key); 598 599 private: 600 void maintainInvariantsAfterUpdate(JSGCParamKey updated); 601 void checkInvariants(); 602 }; 603 604 class GCSchedulingState { 605 /* 606 * Influences how we schedule and run GC's in several subtle ways. The most 607 * important factor is in how it controls the "HeapGrowthFactor". The 608 * growth factor is a measure of how large (as a percentage of the last GC) 609 * the heap is allowed to grow before we try to schedule another GC. 610 */ 611 mozilla::Atomic<bool, mozilla::Relaxed> inHighFrequencyGCMode_; 612 613 public: 614 GCSchedulingState() : inHighFrequencyGCMode_(false) {} 615 616 bool inHighFrequencyGCMode() const { return inHighFrequencyGCMode_; } 617 618 void updateHighFrequencyModeOnGCStart(JS::GCOptions options, 619 const mozilla::TimeStamp& lastGCTime, 620 const mozilla::TimeStamp& currentTime, 621 const GCSchedulingTunables& tunables); 622 void updateHighFrequencyModeOnSliceStart(JS::GCOptions options, 623 JS::GCReason reason); 624 }; 625 626 struct TriggerResult { 627 bool shouldTrigger; 628 size_t usedBytes; 629 size_t thresholdBytes; 630 }; 631 632 using AtomicByteCount = mozilla::Atomic<size_t, mozilla::Relaxed>; 633 634 /* 635 * Tracks the size of allocated data. This is used for both GC and malloc data. 636 * It automatically maintains the memory usage relationship between parent and 637 * child instances, i.e. between those in a GCRuntime and its Zones. 638 */ 639 class HeapSize { 640 /* 641 * The number of bytes in use. For GC heaps this is approximate to the nearest 642 * ArenaSize. It is atomic because it is updated by both the active and GC 643 * helper threads. 644 */ 645 AtomicByteCount bytes_; 646 647 /* 648 * The number of bytes in use at the start of the last collection. 649 */ 650 MainThreadData<size_t> initialBytes_; 651 652 /* 653 * The number of bytes retained after the last collection. This is updated 654 * dynamically during incremental GC. It does not include allocations that 655 * happen during a GC. 656 */ 657 AtomicByteCount retainedBytes_; 658 659 public: 660 explicit HeapSize() { 661 MOZ_ASSERT(bytes_ == 0); 662 MOZ_ASSERT(retainedBytes_ == 0); 663 } 664 665 size_t bytes() const { return bytes_; } 666 size_t initialBytes() const { return initialBytes_; } 667 size_t retainedBytes() const { return retainedBytes_; } 668 669 void updateOnGCStart() { retainedBytes_ = initialBytes_ = bytes(); } 670 671 void addGCArena() { addBytes(ArenaSize); } 672 void removeGCArena() { 673 MOZ_ASSERT(retainedBytes_ >= ArenaSize); 674 removeBytes(ArenaSize, true /* only sweeping removes arenas */); 675 MOZ_ASSERT(retainedBytes_ <= bytes_); 676 } 677 678 void addBytes(size_t nbytes, bool updateRetainedSize = false) { 679 mozilla::DebugOnly<size_t> initialBytes(bytes_); 680 MOZ_ASSERT(initialBytes + nbytes > initialBytes); 681 bytes_ += nbytes; 682 if (updateRetainedSize) { 683 retainedBytes_ += nbytes; 684 } 685 } 686 void removeBytes(size_t nbytes, bool updateRetainedSize) { 687 if (updateRetainedSize) { 688 MOZ_ASSERT(retainedBytes_ >= nbytes); 689 retainedBytes_ -= nbytes; 690 } 691 MOZ_ASSERT(bytes_ >= nbytes); 692 bytes_ -= nbytes; 693 } 694 }; 695 696 /* 697 * Like HeapSize, but also updates a 'parent' HeapSize. Used for per-zone heap 698 * size data that also updates a runtime-wide parent. 699 */ 700 class HeapSizeChild : public HeapSize { 701 public: 702 void addGCArena(HeapSize& parent) { 703 HeapSize::addGCArena(); 704 parent.addGCArena(); 705 } 706 707 void removeGCArena(HeapSize& parent) { 708 HeapSize::removeGCArena(); 709 parent.removeGCArena(); 710 } 711 712 void addBytes(size_t nbytes, HeapSize& parent) { 713 HeapSize::addBytes(nbytes); 714 parent.addBytes(nbytes); 715 } 716 717 void removeBytes(size_t nbytes, bool updateRetainedSize, HeapSize& parent) { 718 HeapSize::removeBytes(nbytes, updateRetainedSize); 719 parent.removeBytes(nbytes, updateRetainedSize); 720 } 721 }; 722 723 class PerZoneGCHeapSize : public HeapSizeChild { 724 public: 725 size_t freedBytes() const { return freedBytes_; } 726 void clearFreedBytes() { freedBytes_ = 0; } 727 728 void removeGCArena(HeapSize& parent) { 729 HeapSizeChild::removeGCArena(parent); 730 freedBytes_ += ArenaSize; 731 } 732 733 void removeBytes(size_t nbytes, bool updateRetainedSize, HeapSize& parent) { 734 HeapSizeChild::removeBytes(nbytes, updateRetainedSize, parent); 735 freedBytes_ += nbytes; 736 } 737 738 private: 739 AtomicByteCount freedBytes_; 740 }; 741 742 // Heap size thresholds used to trigger GC. This is an abstract base class for 743 // GC heap and malloc thresholds defined below. 744 class HeapThreshold { 745 protected: 746 HeapThreshold() 747 : startBytes_(SIZE_MAX), 748 incrementalLimitBytes_(SIZE_MAX), 749 sliceBytes_(SIZE_MAX) {} 750 751 // The threshold at which to start a new incremental collection. 752 // 753 // This can be read off main thread during collection, for example by sweep 754 // tasks that resize tables. 755 MainThreadOrGCTaskData<size_t> startBytes_; 756 757 // The threshold at which start a new non-incremental collection or finish an 758 // ongoing collection non-incrementally. 759 MainThreadData<size_t> incrementalLimitBytes_; 760 761 // The threshold at which to trigger a slice during an ongoing incremental 762 // collection. 763 MainThreadData<size_t> sliceBytes_; 764 765 public: 766 size_t startBytes() const { return startBytes_; } 767 size_t sliceBytes() const { return sliceBytes_; } 768 size_t incrementalLimitBytes() const { return incrementalLimitBytes_; } 769 size_t eagerAllocTrigger(bool highFrequencyGC) const; 770 size_t incrementalBytesRemaining(const HeapSize& heapSize) const; 771 772 void setSliceThreshold(ZoneAllocator* zone, const HeapSize& heapSize, 773 const GCSchedulingTunables& tunables, 774 bool waitingOnBGTask); 775 void clearSliceThreshold() { sliceBytes_ = SIZE_MAX; } 776 bool hasSliceThreshold() const { return sliceBytes_ != SIZE_MAX; } 777 778 protected: 779 static double computeZoneHeapGrowthFactorForHeapSize( 780 size_t lastBytes, const GCSchedulingTunables& tunables, 781 const GCSchedulingState& state); 782 783 void setIncrementalLimitFromStartBytes(size_t retainedBytes, 784 const GCSchedulingTunables& tunables); 785 }; 786 787 // A heap threshold that is based on a multiple of the retained size after the 788 // last collection adjusted based on collection frequency and retained 789 // size. This is used to determine when to do a zone GC based on GC heap size. 790 class GCHeapThreshold : public HeapThreshold { 791 public: 792 void updateStartThreshold(size_t lastBytes, 793 mozilla::Maybe<double> allocationRate, 794 mozilla::Maybe<double> collectionRate, 795 const GCSchedulingTunables& tunables, 796 const GCSchedulingState& state, bool isAtomsZone); 797 798 private: 799 // This is our original algorithm for calculating heap limits. 800 static size_t computeZoneTriggerBytes(double growthFactor, size_t lastBytes, 801 const GCSchedulingTunables& tunables); 802 803 // This is the algorithm described in the optimal heap limits paper. 804 static double computeBalancedHeapLimit(size_t lastBytes, 805 double allocationRate, 806 double collectionRate, 807 const GCSchedulingTunables& tunables); 808 }; 809 810 // A heap threshold that is calculated as a constant multiple of the retained 811 // size after the last collection. This is used to determines when to do a zone 812 // GC based on malloc data. 813 class MallocHeapThreshold : public HeapThreshold { 814 public: 815 void updateStartThreshold(size_t lastBytes, 816 const GCSchedulingTunables& tunables, 817 const GCSchedulingState& state); 818 819 private: 820 static size_t computeZoneTriggerBytes(double growthFactor, size_t lastBytes, 821 size_t baseBytes); 822 }; 823 824 // A fixed threshold that's used to determine when we need to do a zone GC based 825 // on allocated JIT code. 826 class JitHeapThreshold : public HeapThreshold { 827 public: 828 explicit JitHeapThreshold(size_t bytes) { startBytes_ = bytes; } 829 }; 830 831 #ifdef DEBUG 832 833 // Counts memory associated with GC things in a zone. 834 // 835 // This records details of the cell (or non-cell pointer) the memory allocation 836 // is associated with to check the correctness of the information provided. This 837 // is not present in opt builds. 838 class MemoryTracker { 839 public: 840 MemoryTracker(); 841 void fixupAfterMovingGC(); 842 void checkEmptyOnDestroy(); 843 844 // Track memory by associated GC thing pointer. 845 void trackGCMemory(Cell* cell, size_t nbytes, MemoryUse use); 846 void untrackGCMemory(Cell* cell, size_t nbytes, MemoryUse use); 847 void swapGCMemory(Cell* a, Cell* b, MemoryUse use); 848 849 // Track memory by associated non-GC thing pointer. 850 void registerNonGCMemory(void* mem, MemoryUse use); 851 void unregisterNonGCMemory(void* mem, MemoryUse use); 852 void moveNonGCMemory(void* dst, void* src, MemoryUse use); 853 void incNonGCMemory(void* mem, size_t nbytes, MemoryUse use); 854 void decNonGCMemory(void* mem, size_t nbytes, MemoryUse use); 855 856 private: 857 template <typename Ptr> 858 struct Key { 859 Key(Ptr* ptr, MemoryUse use); 860 Ptr* ptr() const; 861 MemoryUse use() const; 862 863 private: 864 # ifdef JS_64BIT 865 // Pack this into a single word on 64 bit platforms. 866 uintptr_t ptr_ : 56; 867 uintptr_t use_ : 8; 868 # else 869 uintptr_t ptr_ : 32; 870 uintptr_t use_ : 8; 871 # endif 872 }; 873 874 template <typename Ptr> 875 struct Hasher { 876 using KeyT = Key<Ptr>; 877 using Lookup = KeyT; 878 static HashNumber hash(const Lookup& l); 879 static bool match(const KeyT& key, const Lookup& l); 880 static void rekey(KeyT& k, const KeyT& newKey); 881 }; 882 883 template <typename Ptr> 884 using Map = HashMap<Key<Ptr>, size_t, Hasher<Ptr>, SystemAllocPolicy>; 885 using GCMap = Map<Cell>; 886 using NonGCMap = Map<void>; 887 888 static bool isGCMemoryUse(MemoryUse use); 889 static bool isNonGCMemoryUse(MemoryUse use); 890 static bool allowMultipleAssociations(MemoryUse use); 891 892 size_t getAndRemoveEntry(const Key<Cell>& key, LockGuard<Mutex>& lock); 893 894 Mutex mutex MOZ_UNANNOTATED; 895 896 // Map containing the allocated size associated with (cell, use) pairs. 897 GCMap gcMap; 898 899 // Map containing the allocated size associated (non-cell pointer, use) pairs. 900 NonGCMap nonGCMap; 901 }; 902 903 #endif // DEBUG 904 905 static inline double LinearInterpolate(double x, double x0, double y0, 906 double x1, double y1) { 907 MOZ_ASSERT(x0 < x1); 908 909 if (x < x0) { 910 return y0; 911 } 912 913 if (x < x1) { 914 return y0 + (y1 - y0) * ((x - x0) / (x1 - x0)); 915 } 916 917 return y1; 918 } 919 920 } // namespace gc 921 } // namespace js 922 923 #endif // gc_Scheduling_h