Phil/controller backoffs

estuary/flow #2038

Description:

Resolves #2032
Adds backoffs to controller-initiated publications and discovers, so that they don't retry them too frequently when the controller is invoked frequently for various different reasons. This ought to dramatically reduce the rate of failed publications and discovers, which are being run due to frequent shard failures, which trigger controller runs.

In all of these, the goal is to still show the controller as being in error status when it's waiting on a backoff, so that failures are still obvious when looking at the overall status.

Also fixes a bug in the runtime, which caused us to fail to set lastSourcePublishedAt in materialization stats documents if no documents were read from the source.

LGTM

14 files14 files	⊥	r1	r2	r3	r4	r5
Commits	⊥	r1	r2	r3	r4	r5
crates/crates/
agent/src/crates/agent/src/
controllers/crates/agent/src/controllers/
dependencies.rs	⊥			r3	r4	r5
materialization.rs	⊥	r1	r2	r3	r4	r5
mod.rs	⊥	r1	r2	r3	r4	r5
periodic.rs	⊥	r1	r2	r3	r4	r5
capture/crates/agent/src/controllers/capture/
auto_discover.rs	⊥		r2	r3	r4	r5
integration_tests/crates/agent/src/integration_tests/
auto_discovers.rs	⊥		r2	r3	r4	r5
dependencies_and_activations.rs	⊥			r3	r4	r5
harness.rs	⊥			r3	r4	r5
periodic_publications.rs	⊥	r1	r2	r3	r4	r5
schema_evolution.rs	⊥			r3	r4	r5
source_captures.rs	⊥				r4	r5
models/src/status/crates/models/src/status/
mod.rs	⊥	r1	r2	r3	r4	r5
runtime/src/materialize/crates/runtime/src/materialize/
protocol.rs	⊥					r5

+39
+39