Data Product Translation Properties

[greenc-FNAL]

Introduction

This discussion started on ZOOM with the following participants:

• @beojan
• @greenc-FNAL
• @marcpaterno
• @sabasehrish
• @wlav

Brief

We will be continuing discussion and planning of the Data Product Translation subsystem. Specifically, Kyle has asked that we develop a list of properties, and provided some proposals as a seed:

• Some proposed properties

1. Translators do not remove or replace any data-product provenance
2. Translators may add additional data-product provenance to record translation having taken place
3. Translators may need to propagate (a reference to) the original data product for object-ownership purposes
4. Translators should not be duplicated if the same translation is required by multiple computational nodes
5. Translators are part of the computational graph

As a way of firing up our thinking in this direction, please—if you have time:

1. Consider the above suggestions and make a note or two for or against.
2. Think about a couple of new properties to suggest and discuss .

Notes

1. Translators do not remove or replace any data-product provenance
   • Provenance -> includes labels, here
   • WL: currently labels are modified in order to distinguish the translated product from the original, and to allow the node requiring the translated input to find it in preference to the original. We need an alternative to this if we are to avoid replacing labels in provenance.
2. Translators may add additional data-product provenance to record translation having taken place
   • Perhaps by adding a tag to the original?
   • @greenc-FNAL: User nodes will see something that they will treat as a concrete data product, regardless of whether it is a "primary" entity created by another user node or provider, or whether it is the result of a translation. 99% of user node metadata needs will be ignorant of translation. Rarely, translation-specific metadata (assuming it exists) may need to be accessed.
3. Translators may need to propagate (a reference to) the original data product for object-ownership purposes
   • @marcpaterno: "May"
   • @greenc-FNAL: Object management is required by the system, and must account for translations and their association with output data products. It is unclear to me whether and how translator nodes need to be aware of and interact with this functionality.
   • @wlav: An algorithm that is preceded by a translator node for its input, can be expected to have a translator node to its output. Whereas the minimum lifetime for the input data products is clear: at least until after the algorithm has completed, there's no equivalent for the output. In particular, the algorithm only produces an intermediary and it is the output translator node that produces the expected output data product. The intermediary has no further place in the graph (because no one can possibly have asked for it). However, it is beneficial in cases to keep the intermediary alive to prevent copying. A mechanism should be in place to express that.
   • [ZOOM discussion truncated due to lack of time]
4. Translators should not be duplicated if the same translation is required by multiple computational nodes
   • [No ZOOM discussion due to lack of time]
5. Translators are part of the computational graph
   • Yes, but: (@marcpaterno) They are part of the computational graph, but they are not computational nodes. A translator will never be recorded as the "creator" of a translated data product.
   • @wlav: possible complication. Currently limited in Python to up to three data products explicitly. In order to support an arbitrary number, we need to collect all the data products into an "internal" collection. Wim has been treating these collector nodes as "translators." They may need to be classified as something else (e.g. aggregators), but integrating translation of individual data products into the process has implications for graph construction and node interaction.

[marcpaterno]

@wlav: An algorithm that is preceded by a translator node for its input, can be expected to have a translator node to its output.

Why is this to be expected? This seems to be an important assumption for what you wrote following this, but I do not understand why this should be expected.

[marcpaterno]

I think, but am not sure, that by "data products have some type of 'default' representation" you mean "data-product concepts have some default concrete data-product type" (all names referring to Figure 3.2 and the related text in the design doc).

If this is what you mean, then I do not think this needs to be the case. But before we go into further discussion, I want to make sure I understood you correctly.

[wlav]

Yes, that is currently the implementation. I'd love to move away from that, but if we do, then both scenarios mentioned above will have more complicated life time management as now the first is no longer transactional but open-ended, just like the second.

My current thinking was along the lines of adding translator nodes first, then scrapping them in an optimization pass after. Such an optimization pass can also adjust life time. The opposite, of constructing a more abstract graph and then adding translation nodes where needed would have my vote if technically feasible.

[wlav]

Aside, for the Numba implementation, I use a union type together with FFI. As long as results are returned on the stack, the implementation doesn't care what the interpretation is of the returned bits, it all works fine. It's only the templates and the graph construction that do and will refuse to execute if C++ types don't match.

[knoepfel]

1. A concrete data product created by a C++ algorithm will have a "C++" representation; and a concrete data product created by a Python algorithm will have a "Python" representation. Any downstream node that wants to consume a concrete data product may require a different representation (i.e., thus requiring a translator). I suppose we could think of the concrete data-product representation at creation time as the "default" representation. But I don't believe we need to.

2. The lifetimes of data products are currently managed in the implementation via std::shared_ptr. The framework never explicitly calls reset on these pointers—the C++ STL simply destroys the managed objects once the use count becomes zero. I think the same thing can happen for translated data products, with the extra complication that some objects may need to be bound in their lifetimes (e.g., an underlying std::vector object needs to exist as long as the numpy array, which serves as the translated data product). But I think that just means we may need to have more kinds of "messages" that we pass around the data-flow graph. That's doable. To the extent possible, we should let other systems do the necessary caching (C++ STL, oneTBB flow-graph join nodes, etc.)

[wlav]

To 1.: yes, ideally, we wouldn't need to think of a "default" representation, but the current template-based graph construction enforces it. By implication, it currently simplifies one part of the life time management by making the translator nodes transactional, which is what I described above. (I make explicit use of this to handle the Python ref-counting.)

To 2.: same thing here: because C++ is the "default" in the current implementation, if forces the use of shared_ptr inside the "Python" (or any other) representation. Hence the "lifeline" objects as a way of doing this without affecting the actual (user-visible) Python representation.

Yes, whether a "message" or something else, it's all about the translator node communicating to the framework what its needs are and lifetime being adjusted accordingly, e.g. by caching a shared_ptr for the duration.