Fix depth in SDSM shadows

[jeannekamikaze]

This fixes #2620, though I am sure not in the cleanest way. Please provide feedback or feel free to throw this whole PR away.

#2620 shows a sample app for the SDSM filter that also uses multisampling. SDSM was not written with multisampling in mind. This change makes SDSM work with and without it.

As I explained in the issue, the problem is that the depth texture that the SDSM uses becomes multisampled, so texture2D becomes texture2DMS, and the texture cannot be sampled but only fetched. Moreover, the logic in the FilterPostProcessor to handle multisampling mostly deals with materials, which the two compute shaders in SDSM are agnostic of.

The change is to have SdsmFitter check whether the input depth texture is multisampled and compile the two shaders accordingly. This introduces some CPU overhead but is the simplest solution without having to track and maintain additional state.

The crux of the change is in MultiSample.glsllib. I have added additional overloads there to work with/out multisampled textures.

I think there are still three problems not addressed here, though:

• The existing getDepth(in sampler2DMS tex, in vec2 texC) is kind of cooked. It calls textureFetch(), which is doing an average of all MS samples. This is rarely what you want for depth. Specifically, at an edge between geometry and background, it'll average values that range between "foreground" and "background" and give something in between. Rather, for depth, you typically want either a max or a min of the samples, depending on the algorithm. Here's a random post about this: https://wickedengine.net/2016/11/how-to-resolve-an-msaa-depthbuffer/ For SDSM in particular, I went with max. I also did not want to change the existing getDepth() because that might break a whole bunch of other shaders that I have not tested.
• Going with max with SDSM may defeat the purpose of multisampling a bit. Like I mentioned in Setting 'setNumSamples' on FilterPostProcessor causes SdsmDirectionalLightShadowFilter to crash. #2620, for some shadow algos it's often better to run the algo on each sample and then average the results. This typically gives a smoother shadow. However, that would also increase the cost of the shader by m_NumDepthSamples. The max should be just as bad as in the no-multisampling case, but not worse. I'm also not entirely familiar with SDSM anyway.
• I'm not really sure it should be up to SdsmFitter and individual compute shader-based post-filters to deal with multisampling. I think this would ideally be abstracted away, either by having the FilterPostProcessor manage compute shaders better, or by resolving the MS textures before the filters are called. But this would require a larger change and more discussion.

Testing

I have tested the sample app in the issue with and without the line fpp.setNumSamples(2).

[codex128]

Unrelated to your changes, but I found a bug while testing this PR. #2631

[jeannekamikaze]

Hello, thank you. Will address your comments soon. I am a Java noob and admittedly should have done a better self-review.

[codex128]

Looks good. Thanks for the contribution!

[riccardobl]

Thank you, there are some changes that are required to get this well integrated into the engine.
Can you look into them?

[riccardobl]

this should not be handled here. GLSLCompat already redefined texture2D to texture

[jeannekamikaze]

Unfortunately that does not work on my end:

com.jme3.renderer.RendererException: Compute shader compilation failed: 0:66(9): error: no function with name 'texture2D'
0:66(2): error: could not implicitly convert return value to vec4, in function `fetchTextureSample'

It sounds like I might be using a core profile, but I don't know why anything would be different on my end. I have not changed any of the project settings. Ubuntu 25.10, OpenJDK 21, Netbeans 29.

[riccardobl]

It is likely because of this

jmonkeyengine/jme3-core/src/main/resources/Common/ShaderLib/GLSLCompat.glsllib

Line 2 in a8b89f8

#ifdef FRAGMENT_SHADER

This, and other similar ifdef in this file should be changed to
#if defined(FRAGMENT_SHADER} || defined(COMPUTE_SHADER)

and COMPUTE_SHADER needs to be always appended to the compute shader defines

[riccardobl]

same here: GLSLCompat already handles this

[riccardobl]

same here: GLSLCompat already handles this