Remove redundant `toType` overloads with different nullability in convert API

[koperagen]

So this causes an exception, as expected:

dataFrameOf("a" to columnOf(123, 322), "b" to columnOf(1.0, null)).convert { a and b }.to<Double>()

Method threw 'org.jetbrains.kotlinx.dataframe.exceptions.CellConversionException' exception.
I think this exception is the reason why two overloads for each operation were introduced:

public fun <T> Convert<T, Any>.toDouble(): DataFrame<T> = to<Double>()
public fun <T> Convert<T, Any?>.toDouble(): DataFrame<T> = to<Double?>()

But i want to argue it's actually redundant, public fun <T> Convert<T, Any?>.toDouble(): DataFrame<T> = to<Double?>() is enough:

If mixed nullability columns are selected, to<Double?>() is resolved

dataFrameOf("a" to columnOf(123, 322), "b" to columnOf(1.0, null)).convert { a and b }.toDouble()

And its result is accurate, a still has not-null Double type, just as originally it was Int.

a: Double
b: Double?

So in terms of type accuracy public fun <T> Convert<T, Any>.toDouble(): DataFrame<T> = to<Double>() is not needed.

Now let's return to the exception. If we deceive the compiler and resolve wrong overload, that normally wouldn't be resolved, then we get exception (that we might expect, since all our columns are Int! (not really))

dataFrameOf("a" to columnOf(123, 322), "b" to columnOf(1, null)).convert { (a and b).cast<Int>() }.toDouble()

Let's check other column selectors. all() -> ColumnSelector<Any?>, so again "nullable" overload is resolved.

dataFrameOf("a" to columnOf(123, 322), "b" to columnOf(1, null)).convert { all() }.toDouble()

Looks like we can remove ~27 overloads from convert.kt and everything will work same as before

[Jolanrensen]

How does the compiler plugin take the taget type of convert {}.to<T>() into account, though? Does it always introduce nullability if T is marked as nullable?
In other words, does the target column get a different type at compile time when converted .to<Double?>() or .to<Double>()? If so, it might make sense to keep both overloads.

And actually, trying to convert to a non-null type (so .to<Double>()) when the source column does contain nulls throws a TypeConversionException currently (see AnyCol.convertToTypeImpl()), so the target type is used actually

[koperagen]

Issue is with shortcuts, not with to<type>.

And actually, trying to convert to a non-null type (so .to()) when the source column does contain nulls throws a TypeConversionException currently (see AnyCol.convertToTypeImpl()), so the target type is used actually

Yes:

So this causes an exception, as expected:

dataFrameOf("a" to columnOf(123, 322), "b" to columnOf(1.0, null)).convert { a and b }.to<Double>()
Method threw 'org.jetbrains.kotlinx.dataframe.exceptions.CellConversionException' exception.

But this is again about to<Double>

But at the same time public fun <T> Convert<T, Any>.toDouble(): DataFrame<T> = to<Double>() is never resolved if column with null is selected

[Jolanrensen]

I see, but what I worry about is

public fun <T> Convert<T, Any?>.toDouble(): DataFrame<T> = to<Double?>()

being resolved for non-null columns and we suddenly gain columns with nullability, either in the compiler plugin or in the KType of the new column.

[koperagen]

being resolved for non-null columns and we suddenly gain columns with nullability, either in the compiler plugin or in the KType of the new column.

Me too. At first i assumed this is what happens, but library actually correctly created not-null columns where expected. Initially i started investigation looking at the issue in compiler plugin! (and it's now fixed):

a: Int
b: Int?

df.convert { a and b }.toDouble() =>

 a: Double?
b: Double?

Because, well, public fun <T> Convert<T, Any?>.toDouble(): DataFrame<T> = to<Double?>() was actually resolved, for which i took nullability = true. Turns out even if nullable variant is resolved, it should keep original nullability

Library does exactly this:

 a: Double
b: Double?