Handling failed tasks

[DutChen18]

Currently, failed tasks (with non-zero exit code) are not reassigned. Should they be?

[BoySanic]

I think they should be reassigned.
We just increment the tasks_to_assign by 1, I think, if the exit code is non-zero or if validation (#7 ) fails. But that depends on #22 merging. Or at least the changes it aims to make.

I think we shouldn't increment tasks_to_assign if it fails... Say 4 times. Doesn't have to be 4, we can make that configurable. But after some number of times, it should stop attempting the same task with the same input data on the assumption something is wrong with the code and something may be either non-deterministic if we can't validate, or there's a serious problem with the binary affecting every machine attempting to run it.

[DutChen18]

We had some talk about this in vc but I don't remember if you were there. We decided that tasks should not be reassigned when they have a non-zero exit code. It will be up to the validator to determine that a non-zero exit code means a failed task.

[DutChen18]

Fixed by #53