Add MAUI iOS Inner Loop measurements for CI

[davidnguyen-tech]

Summary

Adds MAUI iOS Inner Loop performance measurements to CI, supporting both iOS simulators and physical devices.

What's included:

• iOSInnerLoopParser.cs — Binlog parser extracting iOS build task/target timings
• Startup.cs/Reporter.cs — Wiring and null-safety fix
• ioshelper.py — iOS simulator and physical device management
• runner.py — IOSINNERLOOP execution branch (build → deploy → measure → incremental)
• Scenario scripts — pre.py, test.py, post.py, setup_helix.py
• maui_scenarios_ios_innerloop.proj — Helix workitem definition (simulator + physical device)
• Pipeline YAML — Job entries in sdk-perf-jobs.yml and routing in run_performance_job.py

Measurements:

• First deploy: full build + install + launch timing via binlog parsing
• Incremental deploy: source edit → rebuild + reinstall + relaunch timing

Targets:

• iOS Simulator (iossimulator-arm64) on macOS Helix machines
• Physical iPhone (ios-arm64) on Mac.iPhone.17.Perf queue

Based on:

• Existing Android Inner Loop CI scenario (working reference)
• Local iOS implementation from feature/measure-maui-ios-deploy branch

[davidnguyen-tech]

NOTE: The scenario basically works, there are just some infra issues - MI & Xcode.
Recently a there's been a discussion in our team whether this scenario is worth the extra maintenance cost at the moment.

Postponing this PR until we agree on a yes/no.

[davidnguyen-tech]

Update - We've come to conclusion that we would like to move this PR forward.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 10 comments.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (8)

src/scenarios/shared/runner.py:295

• The RuntimeIdentifier rewrite regex can accidentally eat subsequent MSBuild properties when --msbuild-args uses ; separators (e.g. ...RuntimeIdentifier=foo;/p:MtouchLink=None) because \S+ will consume the ;/p:... suffix. Limit the match to stop at ; or whitespace so only the RID value is replaced.

            if ios_rid_env and 'RuntimeIdentifier=' in self.msbuildargs:
                self.msbuildargs = re.sub(
                    r'RuntimeIdentifier=\S+',
                    f'RuntimeIdentifier={ios_rid_env}',
                    self.msbuildargs)

src/scenarios/shared/runner.py:243

• --inner-loop-iterations is documented as "(1+)" but the argparse definition currently allows 0/negative values, which would silently skip the incremental loop and produce incomplete/empty counters. Enforce a valid range (consistent with the Android inner loop parser).

        iosinnerloopparser.add_argument('--device-type', choices=['simulator', 'device'], help='Target device type: simulator (default) or physical device. Auto-detected from RuntimeIdentifier if not set.', dest='devicetype', default=None)
        iosinnerloopparser.add_argument('--inner-loop-iterations', help='Number of incremental build+deploy+startup iterations (1+)', type=int, default=10, dest='innerloopiterations')
        self.add_common_arguments(iosinnerloopparser)

src/scenarios/shared/runner.py:1466

• The regex that re-syncs IOS_RID from /p:RuntimeIdentifier=... has the same \S+-style issue as earlier: if --msbuild-args ever uses ; separators, it can capture too much. Make the capture stop at ; or whitespace so IOS_RID is set correctly.

                # Keep IOS_RID env var aligned so versionmanager uses the same
                # RID for the linked-DLL lookup below.
                m = re.search(r'/p:RuntimeIdentifier=(\S+)', self.msbuildargs)
                if m:
                    os.environ['IOS_RID'] = m.group(1)

src/scenarios/shared/ioshelper.py:249

• When falling back to simctl list devices booted -j, this returns the first any booted device across all runtimes (iOS, watchOS, tvOS). If a non‑iOS simulator is booted during a local run, this can target the wrong UDID and make install/launch fail in confusing ways. Filter to iOS runtimes when selecting a booted simulator UDID.

            data = json.loads(result.stdout)
            for runtime_devices in data.get('devices', {}).values():
                for dev in runtime_devices:
                    if dev.get('state', '').lower() == 'booted':
                        return dev['udid']

eng/pipelines/sdk-perf-jobs.yml:20

• Hard-coding this block to if false disables all public correctness jobs for the pipeline, regardless of the runPublicJobs parameter. This will mask regressions and break expected CI behavior if merged. Restore the parameter-gated condition (or add a new dedicated WIP flag if you need temporary gating).

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:57

• This second if false also disables the public job group unconditionally. If both public blocks are meant to remain, they should still be guarded by parameters.runPublicJobs rather than a constant false so CI can be turned on/off via the pipeline parameter.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:826

• Hard-coding scheduled private jobs to if false disables all scheduled performance coverage regardless of the runScheduledPrivateJobs parameter. This is a major CI/ops change and should be controlled by the existing parameter (or a separate explicit flag), not a constant.

- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:370

• This private-jobs block is also hard-disabled with if false, which removes most private CI coverage when runPrivateJobs is true. If the intent is to run only iOS inner loop temporarily, consider introducing a dedicated WIP flag that defaults to false, rather than checking in a constant false that will be easy to forget and will silently reduce coverage.

  # Scenario benchmarks
  - ${{ if false }}: # disabled: iOS-only WIP test run
    - template: /eng/pipelines/templates/build-machine-matrix.yml
      parameters:
        jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (5)

src/scenarios/shared/runner.py:1541

• StartupWrapper.parsetraces() defaults to copying TRACEDIR into the Helix upload dir (and optionally uploading) on every call. In this inner-loop flow you parse multiple binlogs and later do a single final copytree(const.TRACEDIR, ...) + upload, so this first parse should pass copy_traces=False to avoid redundant I/O.

                                       scenarioname=scenarioprefix + " - First Build and Deploy",
                                       upload_to_perflab_container=False)
                startup.parsetraces(self.traits)

src/scenarios/shared/runner.py:1616

• This per-iteration parse clears the Helix upload traces/ dir and then calls StartupWrapper.parsetraces(), which (by default) recopies the entire accumulated TRACEDIR each iteration. That creates O(N²) copy cost as binlogs accumulate. Use copy_traces=False for each iteration parse (and rely on the single final copytree(const.TRACEDIR, ...) at the end), which also removes the need to delete traces/ here.

                    # Clear stale traces upload dir so copytree in parsetraces doesn't collide
                    helix_upload_dir = helixuploaddir()
                    if helix_upload_dir is not None:
                        traces_upload = os.path.join(helix_upload_dir, 'traces')
                        if os.path.exists(traces_upload):

eng/pipelines/sdk-perf-jobs.yml:20

• This hard-disables the public correctness job block regardless of the runPublicJobs parameter, which will stop CI from running the normal public validation jobs. If this PR is meant to add iOS inner-loop jobs, it should not disable unrelated job groups; keep the original parameter gating so existing CI coverage remains intact.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:826

• Scheduled private jobs are currently unreachable because this condition is hard-coded to false, regardless of the runScheduledPrivateJobs parameter. This will disable scheduled perf coverage for the repo. Restore the parameter condition so schedules can still run when enabled.

- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:368

• Large parts of the private job matrix are now wrapped in - ${{ if false }} blocks (e.g. the main scenario benchmark matrix). That effectively removes the normal private perf coverage and makes this pipeline definition iOS-only. If this is intended as a temporary WIP, it should be isolated (separate branch/pipeline) rather than landed into the main job template.

  # Scenario benchmarks
  - ${{ if false }}: # disabled: iOS-only WIP test run
    - template: /eng/pipelines/templates/build-machine-matrix.yml