Implement halt on target logic

[agarakan]

Description of the issue

Implements halt on a target when Sender calls fail() and resumes reading from target when Sender calls done()

Description of changes

Workflow:
When Sender detects request failure, the batch.fail() is invoked, which in turn invokes the fail callback to set bool halted to true.
When bool halted is true, the queue waits for halted to become false before sending the next batch to the SenderPool tasks queue.
When Sender detects request success, the batch.done() is invoked, which in turn invokes the done callaback setting bool halted to false.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

All unit tests passing

make lint
make fmt
make fmt-sh
make test

Requirements

Before commiting your code, please do the following steps.

1. Run make fmt and make fmt-sh
2. Run make lint

---

Integration Tests

To run integration tests against this PR, add the ready for testing label.

[sky333999]

Say a bad batch from a target caused this to halt. Now that bad batch is re-tried for 14 days and eventually dropped - but this never gets resumed in that case right? So this target is blocked forever in that scenario?

[github-actions]

This PR was marked stale due to lack of activity.