Generalize tracking whenever we fail to fetch or process a federated event

[matrixbot]

This issue has been migrated from #13700.

---

Follow-up to matrix-org/synapse#13621. As discussed at matrix-org/synapse#13589 (comment) and matrix-org/synapse#13589 (comment)

---

• Track when we fail to process a pulled event over federation
  • Added in Keep track when we try and fail to process a pulled event matrix-org/synapse#13589
  • Additionally Record any exception when processing a pulled event matrix-org/synapse#13814
• Maybe: Track when we ask /backfill from a certain event but the /backfill response doesn't include it
  • See Keep track when we try and fail to process a pulled event matrix-org/synapse#13589 (comment)
• Track when an event signature check fails (_check_sigs_and_hash_and_fetch_one, Signature on retrieved event was invalid)
  • Tracked by We keep trying to fetch events whose keys have expired or have been invalidated matrix-org/synapse#13676
  • Added in Track when the pulled event signature fails matrix-org/synapse#13815
• Track when we fail to fetch or process /event/{eventId}
• Track when we fail to fetch or process /event_auth/{roomId}/{eventId}
• Track when we fail to fetch or process /state/{roomId}?event_id=$xxx
• Track when we fail to fetch or process /state_ids/{roomId}?event_id=$xxx
• Track when we fail process an event from /get_missing_events
• Can we also consider soft-failed events because of the spam checker?
  • See Instrument _check_sigs_and_hash_and_fetch to trace time spent in child concurrent calls matrix-org/synapse#13588 (comment)

We will need to add more record_event_failed_backfill_attempt calls for the failed to fetch scenarios though.