Use cases¶
Table of Contents
- Case scenario 1: Bisecting permanent issue
- Case scenario 2: Bisecting intermittent issue
- Case scenario 3: Retrigger an intermittent job on a changeset until hit
- Case scenario 4: Bisecting Talos
- Case scenario 5: After uplift we need a new baseline for release branches
- Case scenario 6: New test validation
- Case scenario 7: Fill in a changeset
- Case scenario 8: Developer needs to add missing platforms/jobs for a Try push
- Case scenario 9: We generate data to build a dynamic TryChooser UI
Case scenario 1: Bisecting permanent issue¶
- We have a job failing
- There are jobs that were coalesced between the last good and the first bad job
- We need to backfill between good revision and the bad revision
This has been completed by the trigger.py with –backfill.
Case scenario 2: Bisecting intermittent issue¶
- We have an intermittent job
- We want to determine when it started happening
- It is not only a matter of coalescing but also a matter of frequency
- We want to give a range of changesets and bisect until spotting culprit
NOTE: We trigger more than one job compared to case scenario 1
The script trigger.py helps with triggering multiple times the same jobs. The script generate_cli.py helps with tracking filed intermittent oranges in bugzilla.
This use case requires implementing the following milestones:
Case scenario 3: Retrigger an intermittent job on a changeset until hit¶
https://bugzilla.mozilla.org/show_bug.cgi?id=844746
- This is more of an optimization.
- The intent is to hit the orange with extra debugging information.
- We’re not bisecting in here.
- We can trigger batches (e.g. 5 at a time)
Not in scope at the moment.
This could be done with a modification of trigger.py where we monitor the jobs running until one of them fails.
This use case requires implementing the following milestones:
and this issue:
Case scenario 4: Bisecting Talos¶
- We have a performance regression
- We want to determine when it started showing up
- Given a revision that _failed_
- Re-trigger that revision N times and all revisions prior to it until the last data point + 1 more
We can do this manually by using –back-revisions::
python scripts/trigger.py \
--buildername "Rev5 MacOSX Yosemite 10.10 fx-team talos dromaeojs" \
--rev e16054134e12 --back-revisions 10
We currently don’t have the ability to determine the previous baseline and trigger everything up to the last known data point close to the baseline.
Case scenario 5: After uplift we need a new baseline for release branches¶
- We need several data points to establish a baseline
- After an uplift we need to generate a new baseline
- Once there is a baseline we can determine regression
This can already be accomplished like this::
python alltalos.py --repo-name try --times 2 --rev 921277f744d9
Case scenario 6: New test validation¶
- New test validation
- Re-triggering to determine indeterminacy
- Single revision
- All platforms running test
NOTE: Review this thread on how to determine on which job and which platforms do we run a specific test.
Not in scope at the moment.
Case scenario 7: Fill in a changeset¶
- We know that a changeset is missing jobs
- We want to add all missing jobs
This would need grabbing the list of builders for such repo, determine if there has been a successful job for each builder and trigger it if not. We might need to filter out some periodic builders (i.e. a repo can have pgo periodic jobs).
If we want to watch that revision until it is completely filled we will need to accomplish the following milestones:
Case scenario 8: Developer needs to add missing platforms/jobs for a Try push¶
- The developer pushes to try specifying only a subset of all jobs
- The developer realizes that it needs more jobs to run on that push
- The developer uses mozci to not have to push again to try with the right syntax
This has been filed as issue 109
Case scenario 9: We generate data to build a dynamic TryChooser UI¶
- TryChooser UI is always out of date
- mozci can generate the data we need to create an up-to-date TryChooser UI
See write_tests_per_platform_graph.py for an example on how to generate the data needed.