Add job trace architecture section to ci docs

What does this MR do and why?

We recently had an incident with live trace race conditions. I realized 1 part of my mental model of this domain area was off (we havn't had to work on this area much since since it was created. I created an architectural diagram and monitoring links to assist with incidents and development changes.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.