flakiness - fix multicluster restore like nomesh (#4302)

# Summary

- this hopefully fixes our third most flaky test (not 100% confident) -
[HC](REDACTED)
   - passed 3 times in a row
- fixing it similar to
https://github.com/10gen/ops-manager-kubernetes/pull/4252/files#diff-10ddb3bb8e5ff0e06e75d6dbf0b1f878f060eacc97fd15b0f3232677012927e3
- this is done by making sure the cluster is fully healthy and ready
before doing any backup related things, which can cause inconsistent
cluster state
- this related PR makes it faster and also should help:
10gen/ops-manager-kubernetes#4144
## Proof of Work

-
https://spruce.mongodb.com/task/ops_manager_kubernetes_e2e_multi_cluster_kind_e2e_multi_cluster_backup_restore_patch_e7de2802b95ed1b6dcec5cf638f5de64c27e9833_67ffed6f0eaa6e0007e90483_25_04_16_17_48_35/tests?execution=2&sortBy=STATUS&sortDir=ASC
-
[patch2](https://spruce.mongodb.com/task/ops_manager_kubernetes_e2e_multi_cluster_kind_e2e_multi_cluster_backup_restore_patch_16656228bc9c19d62146964b0965f04a9c6b1987_680014302b1b740007eab4f4_25_04_16_20_33_53/logs?execution=0)

## Checklist
- [ ] Have you linked a jira ticket and/or is the ticket in the title?
- [ ] Have you checked whether your jira ticket required DOCSP changes?
- [ ] Have you checked for release_note changes?

## Reminder (Please remove this when merging)
- Please try to Approve or Reject Changes the PR, keep PRs in review as
short as possible
- Our Short Guide for PRs:
[Link](REDACTED)
- Remember the following Communication Standards - use comment prefixes
for clarity:
  * **blocking**: Must be addressed before approval.
  * **follow-up**: Can be addressed in a later PR or ticket.
  * **q**: Clarifying question.
  * **nit**: Non-blocking suggestions.
  * **note**: Side-note, non-actionable. Example: Praise
  * --> no prefix is considered a question

Author: nammn

docker/mongodb-enterprise-tests/tests/multicluster/multi_cluster_backup_restore.py

@@ -477,13 +477,21 @@ def test_add_test_data(self, mongodb_multi_one_collection):
     def test_mdb_backed_up(self, project_one: OMTester):
         project_one.wait_until_backup_snapshots_are_ready(expected_count=1)
 
+    @mark.e2e_multi_cluster_backup_restore
+    def test_mongodb_multi_one_running_state(self, mongodb_multi_one: MongoDBMulti):
+        mongodb_multi_one.assert_reaches_phase(Phase.Running, ignore_errors=True, timeout=1200)
+
     @mark.e2e_multi_cluster_backup_restore
     def test_change_mdb_data(self, mongodb_multi_one_collection):
         now_millis = time_to_millis(datetime.datetime.now())
         print("\nCurrent time (millis): {}".format(now_millis))
         time.sleep(30)
         mongodb_multi_one_collection.insert_one({"foo": "bar"})
 
+    @mark.e2e_multi_cluster_backup_restore
+    def test_mongodb_multi_one_running_state(self, mongodb_multi_one: MongoDBMulti):
+        mongodb_multi_one.assert_reaches_phase(Phase.Running, ignore_errors=True, timeout=1200)
+
     @mark.e2e_multi_cluster_backup_restore
     def test_pit_restore(self, project_one: OMTester):
         now_millis = time_to_millis(datetime.datetime.now())
@@ -495,6 +503,14 @@ def test_pit_restore(self, project_one: OMTester):
 
         project_one.create_restore_job_pit(pit_millis)
 
+    @mark.e2e_multi_cluster_backup_restore
+    def test_mdb_ready(self, mongodb_multi_one: MongoDBMulti):
+        # Note: that we are not waiting for the restore jobs to get finished as PIT restore jobs get FINISHED status
+        # right away.
+        # But the agent might still do work on the cluster, so we need to wait for that to happen.
+        mongodb_multi_one.assert_reaches_phase(Phase.Pending, timeout=1200)
+        mongodb_multi_one.assert_reaches_phase(Phase.Running, timeout=1200)
+
     @mark.e2e_multi_cluster_backup_restore
     def test_data_got_restored(self, mongodb_multi_one_collection):
         assert_data_got_restored(TEST_DATA, mongodb_multi_one_collection, timeout=1200)