[WIP] Put the output of the model into existing NDArrays when provided

[zemei]

Introducing an optional output NDList parameter on the Predictor::predict interface. When the output parameter is provided the underlying engine will copy the inference result into the corresponding NDArray instead of creating new objects.

This functionality is useful for high throughput systems as it reduces the number of memory allocations and reduces the load on the garbage collector.

TODO:

• The engine specific implementation is only provided for PyTorch:
  • On the Java side, the copy is only implemented for the NDArray type.
  • The PyTorch lib side was refactored since and not included in the PR.

[frankfliu]

In general, this is a good idea to reuse the output NDArrays, but define NDList at Predictor is debatable.

I think, this can be implemented at Translator level instead, and we can document how to reuse the NDList in Translator implementation.

The outputs support at Block level is necessary to support reuse the NDList, but may not work for all block, we can add TODO to some of the Block implementation to make it reusable.

[lanking520]

@frankfliu @zemei Maybe we can try to tweak around on the ctx through implementing a cached NDArray holder. User are responsible to control the life-cycle of it.

public O processInput(Context ctx, I input) {
      NDArray array = ctx.getCachedNdArray("name");
}

This CachedNDArray will not be vanished until the end of Model life cycle. But there is a risk for memory leak if use misuse this component.

[frankfliu]

@zemei

Are you still interested in this PR? I took a close look today. The idea to reuse input/output tensor is great, however I don't think this implementation is optimal:

1. This optimization is only needed for inference, we don't really need touch so many classes, we only need add Predictor.setOutputTensor(NDList)
2. In this PR, the output tensor is still created in JNI, and copied to existing NDArray, I don't think this saved anything. I actually don't know if it's possible to re-use output tensor in libtorch.
3. I feel re-use input tensor might be a more feasible approach