elm_rebase.sh (13724B)
1 #!/bin/bash 2 3 # This script exists to help with the rebase process on elm. It rebases 4 # each patch individually to make it easier to fix rebase conflicts 5 # without jeopardizing earlier, sucessfully rebased commits. In order to 6 # limit rebase conflicts around generated moz.build files, it regenerates 7 # moz.build file commits. It also ensures any commits with 'FLOAT' in the 8 # commit summary are pushed to the top of the fast-forward stack to help 9 # the sheriffs more easily merge our commit stack from elm to moz-central. 10 # 11 # Occasionally, there will be upstream vendored commits that break the 12 # build file generation with follow on commits that fix that error. In 13 # order to allow the rebase process to work more smoothly, it is possible 14 # to annotate a commit with the string '(skip-generation)' and normal 15 # build file generation (detected with changes to BUILD.gn files) is 16 # disabled for that commit. The script outputs instructions for handling 17 # this situation. 18 # 19 # Note: the very first rebase operation may require some manual 20 # intervention. The user will need to provide, at minimum, the first 21 # commit of the fast-forward stack if the script is unable to determine 22 # it automatically. Example: 23 # MOZ_BOTTOM_FF=30f0afb7e4c5 \ 24 # bash dom/media/webrtc/third_party_build/elm_rebase.sh 25 # 26 # Assumes the top of the fast-forward stack to rebase is the current revision, 27 # ".". 28 29 function show_error_msg() 30 { 31 echo "*** ERROR *** $? line $1 $0 did not complete successfully!" 32 echo "$ERROR_HELP" 33 } 34 ERROR_HELP="" 35 36 # Print an Error message if `set -eE` causes the script to exit due to a failed command 37 trap 'show_error_msg $LINENO' ERR 38 39 source dom/media/webrtc/third_party_build/use_config_env.sh 40 41 GENERATION_ERROR=$" 42 Generating build files has failed. The most common reason for this 43 failure is that the current commit has an upcoming '(fix-xxxxxx)' commit 44 that will then allow the build file generation to complete. If the 45 current situation seems to fit that pattern, adding a line with 46 '(skip-generation)' to the commit message will ensure that future rebase 47 operations do not attempt to generate build files for this commit. It may 48 be as simple as running the following commands: 49 HGPLAIN=1 hg log -T '{desc}' -r tip > $TMP_DIR/commit_message.txt 50 ed -s $TMP_DIR/commit_message.txt <<< $'3i\n(skip-generation)\n\n.\nw\nq' 51 hg commit --amend -l $TMP_DIR/commit_message.txt 52 bash $0 53 " 54 COMMIT_LIST_FILE=$TMP_DIR/rebase-commit-list.txt 55 export HGPLAIN=1 56 57 if [ "x$MOZ_TOP_FF" = "x" ]; then 58 MOZ_TOP_FF="" 59 fi 60 if [ "x$MOZ_BOTTOM_FF" = "x" ]; then 61 MOZ_BOTTOM_FF="" 62 fi 63 if [ "x$STOP_FOR_REORDER" = "x" ]; then 64 STOP_FOR_REORDER="" 65 fi 66 67 # After this point: 68 # * eE: All commands should succeed. 69 # * u: All variables should be defined before use. 70 # * o pipefail: All stages of all pipes should succeed. 71 set -eEuo pipefail 72 73 # always make sure the repo is clean before doing the rebase 74 CHANGED_FILE_CNT=`hg status | wc -l | tr -d " "` 75 if [ "x$CHANGED_FILE_CNT" != "x0" ]; then 76 echo "There are modified or untracked files in the repo." 77 echo "Please cleanup the repo before running" 78 echo "$0" 79 exit 1 80 fi 81 82 RESUME_FILE=$STATE_DIR/elm_rebase.resume 83 if [ -f $RESUME_FILE ]; then 84 source $RESUME_FILE 85 else 86 87 # on first run, we want to verify sanity of the patch-stack so 88 # ending guidance is appropriate regarding changes in 89 # third_party/libwebrtc between the old central we're currently 90 # based on and the new central we're rebasing onto. 91 echo "Restoring patch-stack..." 92 ./mach python $SCRIPT_DIR/restore_patch_stack.py --repo-path $MOZ_LIBWEBRTC_SRC 93 echo "Verify vendoring..." 94 ERROR_HELP=$"When verify_vendoring.sh is successful, run the following in bash: 95 (source $SCRIPT_DIR/use_config_env.sh ; 96 ./mach python $SCRIPT_DIR/save_patch_stack.py \\ 97 --repo-path $MOZ_LIBWEBRTC_SRC \\ 98 --target-branch-head $MOZ_TARGET_UPSTREAM_BRANCH_HEAD \\ 99 --separate-commit-bug-number $MOZ_FASTFORWARD_BUG ) 100 101 Then resume running this script: 102 bash $0 103 " 104 bash $SCRIPT_DIR/verify_vendoring.sh || (echo "$ERROR_HELP" ; exit 1) 105 ERROR_HELP="" 106 107 if [ "x" == "x$MOZ_TOP_FF" ]; then 108 MOZ_TOP_FF=`hg log -r . -T"{node|short}"` 109 110 ERROR_HELP=$" 111 The topmost commit to be rebased is not in the public phase. Should it be 112 pushed to elm first? If this is intentional, please rerun the command and pass 113 it in explicitly: 114 MOZ_TOP_FF=$MOZ_TOP_FF bash $0 115 " 116 if [[ $(hg phase -r .) != *public ]]; then 117 echo "$ERROR_HELP" 118 exit 1 119 fi 120 ERROR_HELP="" 121 122 ERROR_HELP=$" 123 The topmost commit to be rebased is public but has descendants. If those 124 descendants should not be rebased, please rerun the command and pass the commit 125 in explicitly: 126 MOZ_TOP_FF=$MOZ_TOP_FF bash $0 127 " 128 if [ "x" != "x$(hg log -r 'descendants(.) and !.' -T'{node|short}')" ]; then 129 echo "$ERROR_HELP" 130 exit 1 131 fi 132 ERROR_HELP="" 133 fi 134 135 hg pull central 136 137 ERROR_HELP=$" 138 Automatically determining the bottom (earliest) commit of the fast-forward 139 stack has failed. Please provide the bottom commit of the fast-forward 140 stack. The bottom commit means the commit following the most recent 141 mozilla-central commit. This could be the sha of the .arcconfig commit 142 if it is the bottom commit. 143 That command looks like: 144 MOZ_BOTTOM_FF={base-sha} bash $0 145 " 146 if [ "x" == "x$MOZ_BOTTOM_FF" ]; then 147 # Finds the common ancestor between our top fast-forward commit and 148 # mozilla-central using: 149 # ancestor($MOZ_TOP_FF, central) 150 MOZ_OLD_CENTRAL=`hg id --id --rev "ancestor($MOZ_TOP_FF, central)"` 151 # Using that ancestor and $MOZ_TOP_FF as a range, find the commit _after_ 152 # the the common commit using limit(range, 1, 1) which gives the first 153 # commit of the range, offset by one commit. 154 MOZ_BOTTOM_FF=`hg id --id --rev "limit($MOZ_OLD_CENTRAL::$MOZ_TOP_FF, 1, 1)"` 155 fi 156 if [ "x" == "x$MOZ_BOTTOM_FF" ]; then 157 echo "No value found for the bottom commit of the fast-forward commit stack." 158 echo "$ERROR_HELP" 159 exit 1 160 fi 161 ERROR_HELP="" 162 163 MOZ_NEW_CENTRAL=`hg log -r central -T"{node|short}"` 164 165 echo "bottom of fast-foward tree is $MOZ_BOTTOM_FF" 166 echo "top of fast-forward tree (webrtc-fast-forward) is $MOZ_TOP_FF" 167 echo "new target for elm rebase $MOZ_NEW_CENTRAL (tip of moz-central)" 168 169 hg log -T '{rev}:{node|short} {desc|firstline}\n' \ 170 -r $MOZ_BOTTOM_FF::$MOZ_TOP_FF > $COMMIT_LIST_FILE 171 172 # move all FLOAT lines to end of file, and delete the "empty" tilde line 173 # line at the beginning 174 ed -s $COMMIT_LIST_FILE <<< $'g/- FLOAT -/m$\ng/^~$/d\nw\nq' 175 176 MOZ_BOOKMARK=`date "+webrtc-fast-forward-%Y-%m-%d--%H-%M"` 177 hg bookmark -r elm $MOZ_BOOKMARK 178 179 hg update $MOZ_NEW_CENTRAL 180 181 ERROR_HELP=$" 182 Running ./mach bootstrap has failed. For details, see: 183 $LOG_DIR/log-bootstrap.txt 184 " 185 echo "Running ./mach bootstrap..." 186 ./mach bootstrap --application=browser --no-system-changes &> $LOG_DIR/log-bootstrap.txt 187 echo "Done running ./mach bootstrap" 188 ERROR_HELP="" 189 190 echo "Running ./mach clobber..." 191 ./mach clobber &> $LOG_DIR/log-sanity-clobber.txt 192 echo "Done running ./mach clobber" 193 194 # pre-work is complete, let's write out a temporary config file that allows 195 # us to resume 196 echo $"export MOZ_BOTTOM_FF=$MOZ_BOTTOM_FF 197 export MOZ_TOP_FF=$MOZ_TOP_FF 198 export MOZ_OLD_CENTRAL=$MOZ_OLD_CENTRAL 199 export MOZ_NEW_CENTRAL=$MOZ_NEW_CENTRAL 200 export MOZ_BOOKMARK=$MOZ_BOOKMARK 201 " > $RESUME_FILE 202 fi # if [ -f $RESUME_FILE ]; then ; else 203 204 if [ "x$STOP_FOR_REORDER" = "x1" ]; then 205 echo "" 206 echo "Stopping after generating commit list ($COMMIT_LIST_FILE) to" 207 echo "allow tweaking commit ordering. Re-running $0 will resume the" 208 echo "rebase processing. To stop processing during the rebase," 209 echo "insert a line with only 'STOP'." 210 exit 211 fi 212 213 # grab all commits 214 COMMITS=`cat $COMMIT_LIST_FILE | awk '{print $1;}'` 215 216 echo -n "Commits: " 217 for commit in $COMMITS; do 218 echo -n "$commit " 219 done 220 echo "" 221 222 for commit in $COMMITS; do 223 echo "Processing $commit" 224 FULL_COMMIT_LINE=`head -1 $COMMIT_LIST_FILE` 225 226 function remove_commit () { 227 echo "Removing from list '$FULL_COMMIT_LINE'" 228 ed -s $COMMIT_LIST_FILE <<< $'1d\nw\nq' 229 } 230 231 if [ "$FULL_COMMIT_LINE" == "STOP" ]; then 232 echo "Stopping for history editing. Re-run $0 to resume." 233 remove_commit 234 exit 235 fi 236 237 IS_BUILD_COMMIT=`hg log -T '{desc|firstline}' -r $commit \ 238 | grep "file updates" | wc -l | tr -d " " || true` 239 echo "IS_BUILD_COMMIT: $IS_BUILD_COMMIT" 240 if [ "x$IS_BUILD_COMMIT" != "x0" ]; then 241 echo "Skipping $commit:" 242 hg log -T '{desc|firstline}' -r $commit 243 remove_commit 244 continue 245 fi 246 247 IS_SKIP_GEN_COMMIT=`hg log --verbose \ 248 -r $commit \ 249 | grep "skip-generation" | wc -l | tr -d " " || true` 250 echo "IS_SKIP_GEN_COMMIT: $IS_SKIP_GEN_COMMIT" 251 252 echo "Generate patch for: $commit" 253 hg export -r $commit > $TMP_DIR/rebase.patch 254 255 echo "Import patch for $commit" 256 hg import $TMP_DIR/rebase.patch || \ 257 ( hg log -T '{desc}' -r $commit > $TMP_DIR/rebase_commit_message.txt ; \ 258 remove_commit ; \ 259 echo "Error importing: '$FULL_COMMIT_LINE'" ; \ 260 echo "Please fix import errors, then:" ; \ 261 echo " hg commit -l $TMP_DIR/rebase_commit_message.txt" ; \ 262 echo " bash $0" ; \ 263 exit 1 ) 264 265 remove_commit 266 267 if [ "x$IS_SKIP_GEN_COMMIT" != "x0" ]; then 268 echo "Skipping build generation for $commit" 269 continue 270 fi 271 272 MODIFIED_BUILD_RELATED_FILE_CNT=`bash $SCRIPT_DIR/get_build_file_changes.sh \ 273 | wc -l | tr -d " "` 274 echo "MODIFIED_BUILD_RELATED_FILE_CNT: $MODIFIED_BUILD_RELATED_FILE_CNT" 275 if [ "x$MODIFIED_BUILD_RELATED_FILE_CNT" != "x0" ]; then 276 echo "Regenerate build files" 277 ./mach python build/gn_processor.py \ 278 dom/media/webrtc/third_party_build/gn-configs/webrtc.json || \ 279 ( echo "$GENERATION_ERROR" ; exit 1 ) 280 281 MOZ_BUILD_CHANGE_CNT=`hg status third_party/libwebrtc \ 282 --include 'third_party/libwebrtc/**moz.build' | wc -l | tr -d " "` 283 if [ "x$MOZ_BUILD_CHANGE_CNT" != "x0" ]; then 284 bash dom/media/webrtc/third_party_build/commit-build-file-changes.sh 285 NEWEST_COMMIT=`hg log -T '{desc|firstline}' -r tip` 286 echo "NEWEST_COMMIT: $NEWEST_COMMIT" 287 echo "NEWEST_COMMIT: $NEWEST_COMMIT" >> $LOG_DIR/rebase-build-changes-commits.log 288 fi 289 echo "Done generating build files" 290 fi 291 292 echo "Done processing $commit" 293 done 294 295 ERROR_HELP=$" 296 Running the sanity build has failed. For details, see: 297 $LOG_DIR/log-sanity-build.txt 298 299 Please fix the build, commit the changes (if necessary, the 300 patch-stack will be fixed up during steps following the 301 build), and then run: 302 bash $0 303 " 304 echo "Running sanity build..." 305 ./mach build &> $LOG_DIR/log-sanity-build.txt 306 echo "Done running sanity build" 307 ERROR_HELP="" 308 309 # In case any changes were made to fix the build after the rebase, 310 # we'll check for changes since the last patch-stack update in 311 # third_party/libwebrtc/moz-patch-stack. 312 # TODO: this is copied from verify_vendoring.sh. We should make 313 # make this reusable. 314 # we grab the entire firstline description for convenient logging 315 LAST_PATCHSTACK_UPDATE_COMMIT=`hg log -r ::. --template "{node|short} {desc|firstline}\n" \ 316 --include "third_party/libwebrtc/moz-patch-stack/*.patch" | tail -1` 317 echo "LAST_PATCHSTACK_UPDATE_COMMIT: $LAST_PATCHSTACK_UPDATE_COMMIT" 318 319 LAST_PATCHSTACK_UPDATE_COMMIT_SHA=`echo $LAST_PATCHSTACK_UPDATE_COMMIT \ 320 | awk '{ print $1; }'` 321 echo "LAST_PATCHSTACK_UPDATE_COMMIT_SHA: $LAST_PATCHSTACK_UPDATE_COMMIT_SHA" 322 323 # grab the oldest, non "Vendor from libwebrtc" line 324 CANDIDATE_COMMITS=`hg log --template "{node|short} {desc|firstline}\n" \ 325 -r "children($LAST_PATCHSTACK_UPDATE_COMMIT_SHA)::. - desc('re:(Vendor libwebrtc)')" \ 326 --include "third_party/libwebrtc/" | awk 'BEGIN { ORS=" " }; { print $1; }'` 327 echo "CANDIDATE_COMMITS:" 328 echo "$CANDIDATE_COMMITS" 329 330 EXTRACT_COMMIT_RANGE="" 331 if [ "x$CANDIDATE_COMMITS" != "x" ]; then 332 EXTRACT_COMMIT_RANGE="$CANDIDATE_COMMITS" 333 echo "EXTRACT_COMMIT_RANGE: $EXTRACT_COMMIT_RANGE" 334 fi 335 336 # This is blank in case no changes have been made in third_party/libwebrtc 337 # since the previous rebase (or original elm reset). 338 PATCH_STACK_FIXUP="" 339 340 echo "Checking for new mercurial changes in third_party/libwebrtc" 341 FIXUP_INSTRUCTIONS=$" 342 Mercurial changes in third_party/libwebrtc since the last rebase have been 343 detected (using the verify_vendoring.sh script). Running the following 344 commands should help remedy the situation: 345 346 ./mach python $SCRIPT_DIR/extract-for-git.py \\ 347 $MOZ_OLD_CENTRAL::$MOZ_NEW_CENTRAL $EXTRACT_COMMIT_RANGE 348 mv mailbox.patch $MOZ_LIBWEBRTC_SRC 349 (cd $MOZ_LIBWEBRTC_SRC && \\ 350 git am mailbox.patch) 351 bash $SCRIPT_DIR/verify_vendoring.sh 352 353 When verify_vendoring.sh is successful, run the following in bash: 354 (source $SCRIPT_DIR/use_config_env.sh ; 355 ./mach python $SCRIPT_DIR/save_patch_stack.py \\ 356 --repo-path $MOZ_LIBWEBRTC_SRC \\ 357 --target-branch-head $MOZ_TARGET_UPSTREAM_BRANCH_HEAD \\ 358 --separate-commit-bug-number $MOZ_FASTFORWARD_BUG ) 359 " 360 echo "Restoring patch-stack..." 361 # restore to make sure new no-op commit files are in the state directory 362 # in case the user is instructed to save the patch-stack. 363 ./mach python $SCRIPT_DIR/restore_patch_stack.py --repo-path $MOZ_LIBWEBRTC_SRC 364 echo "Verify vendoring..." 365 bash $SCRIPT_DIR/verify_vendoring.sh &> $LOG_DIR/log-verify.txt || PATCH_STACK_FIXUP="$FIXUP_INSTRUCTIONS" 366 echo "Done checking for new mercurial changes in third_party/libwebrtc" 367 368 # now that we've run all the things that should be fallible, remove the 369 # resume state file 370 rm $RESUME_FILE 371 372 REMAINING_STEPS=$" 373 The rebase process is complete. The following steps must be completed manually: 374 $PATCH_STACK_FIXUP 375 hg push -r tip --force && \\ 376 hg push -B $MOZ_BOOKMARK 377 " 378 echo "$REMAINING_STEPS"