Address @artem’s C++ changes here:
#1846 (comment)
2 tasks:
- dataset and dataset_view struct contain no information.
template <typename ContainerType, typename DataT, typename IdxT, typename Accessor>
struct dataset;
template <typename ContainerType, typename DataT, typename IdxT, typename Accessor>
struct dataset_view;
Only concrete dataset types like padded_dataset, standard_dataset, and vpq_dataset have information. However, that information is replicated for each of the concrete datasets even though the code logic is the same. This can be abstracted out and moved one level up to reduce code redundancy. In particular: view() function and n_rows() function are same regardless of concrete dataset type and can be abstracted one level up into the dataset struct. This makes it so that later we can just have operations on some basic common dataset struct features without specifying concrete dataset type and only dispatch on actual concrete dataset type later in the nested calls.
- We have:
template <typename DataT, typename IdxT, typename Accessor>
using dense_owning_matrix = std::conditional_t<Accessor::is_device_accessible,
raft::device_matrix<DataT, IdxT, raft::row_major>,
raft::host_matrix<DataT, IdxT, raft::row_major>>;
template <typename DataT, typename IdxT, typename Accessor>
using dense_view_matrix =
std::conditional_t<Accessor::is_device_accessible,
raft::device_matrix_view<const DataT, IdxT, raft::row_major>,
raft::host_matrix_view<const DataT, IdxT, raft::row_major>>;
Have dense dataset use implementation of mdarray because it's just the same thing. For compressed dataset this would be something different, perhaps several mdarrays for codebooks and any additional info. We have a rhombus of indirection where we are first converting from concrete dataset to dense dataset and then to underlying mdarray when we can just directly go from concrete dataset to mdarray and get rid of the middle layer. The dense_dataset layer of indirection is unecessary. Get rid of the conditional.
This may involve removing host or device accessor to get rid of indirection in dense dataset. Just pass that host or device accessor directly to mdarray as shown here through a comment on the BBQ dataset PR: #2506 (comment)
Address @artem’s C++ changes here:
#1846 (comment)
2 tasks:
Only concrete dataset types like padded_dataset, standard_dataset, and vpq_dataset have information. However, that information is replicated for each of the concrete datasets even though the code logic is the same. This can be abstracted out and moved one level up to reduce code redundancy. In particular: view() function and n_rows() function are same regardless of concrete dataset type and can be abstracted one level up into the dataset struct. This makes it so that later we can just have operations on some basic common dataset struct features without specifying concrete dataset type and only dispatch on actual concrete dataset type later in the nested calls.
Have dense dataset use implementation of mdarray because it's just the same thing. For compressed dataset this would be something different, perhaps several mdarrays for codebooks and any additional info. We have a rhombus of indirection where we are first converting from concrete dataset to dense dataset and then to underlying mdarray when we can just directly go from concrete dataset to mdarray and get rid of the middle layer. The dense_dataset layer of indirection is unecessary. Get rid of the conditional.
This may involve removing host or device accessor to get rid of indirection in dense dataset. Just pass that host or device accessor directly to mdarray as shown here through a comment on the BBQ dataset PR: #2506 (comment)