V4 filters are differ from V3. You can't just put there shader and assume that texture coords are in [0,1] range.
I am sorry that you have to learn all of that, and I will make sure that the process will be easier for pixi-v5.
Filter Area
Thanks to @bQvle and @radixzz
First, lets work with the AREA. When you apply filter to container, PIXI calculates the bounding box for it. We are working with bounding box.
Invariant: maximal vTextureCoord multiplied by "filterArea.xy" is the real size of bounding box.
Don't try to think about it: its like that because of performance reasons, its not logical in user-experience sense. Neither vTextureCoord dimensions, neither filterArea.xy are predictable, but they multiplication is what we need.
Area can have padding, so please don't use it to get "displacement texture" coordinates or any second-hand textures you are adding to the shader, use "mappedMatrix" for it (see below)
If you want to get the pixel coordinates, use "uniform filterArea", it will be passed to the filter automatically.
uniform vec4 filterArea;
...
vec2 pixelCoord = vTextureCoord * filterArea.xy;
They are in pixels. That wont work if we want something like "fill the ellipse into a bounding box". So, lets pass dimensions too! PIXI doesnt do it automatically, we need manual fix:
filter.apply = function(filterManager, input, output)
{
this.uniforms.dimensions[0] = input.sourceFrame.width
this.uniforms.dimensions[1] = input.sourceFrame.height
// draw the filter...
filterManager.applyFilter(this, input, output);
}
Lets combine it in shader!
uniform vec4 filterArea;
uniform vec2 dimensions;
...
vec2 pixelCoord = vTextureCoord * filterArea.xy;
vec2 normalizedCoord = pixelCoord / dimensions;
Here's the fiddle: https://jsfiddle.net/parsab1h/ . You can see that shader uses "map" and "unmap" to get to that pixel
Now let's assume that you somehow need real coordinates on screen for that thing. You can use another component of filterArea, zw:
uniform vec4 filterArea;
...
vec2 screenCoord = (vTextureCoord * filterArea.xy + filterArea.zw);
I dont have an example for that, but may be you need that value for something?
Fitting problem
Thanks to @adam13531 at github.
One small problem: those values become wrong when PIXI tries to fit bounding box: here's the fiddle: http://jsfiddle.net/xbmhh207/1/
Please use this line to fix it:
filter.dontFit = true;
Bleeding problem
Thanks to @bQvle
The temporary textures that are used by FilterManager can have some bad pixels. It can bleed. For example, displacementSprite can look through the edge, try to move mouse at the bottom edge of http://pixijs.github.io/examples/#/filters/displacement-map.js. You see that transparent (black) zone, but it could be ANYTHING if it wasnt clamped. To make sure it doesnt happen in your case, please use clamping after you map coordinates:
uniform vec4 filterClamp;
vec2 pixelCoord = WE_CALCULATED_IT_SOMEHOW
vec2 unmappedCoord = pixelCoord / filterArea.xy;
vec2 clampedCoord = clamp(unmappedCoord, filterClamp.xy, filterClamp.zw);
vec4 rgba = texture2D(uSampler, clampedCoord);
Both FilterClamp and FilterArea are provided by FilterManager, you dont have to calculate pass it in "filter.apply", here's pixi code that takes care of that: https://github.com/pixijs/pixi.js/blob/dev/src/core/renderers/webgl/managers/FilterManager.js#L297
OK, now we have "transparent" zone instead of random pixels. But what if we want it to be fit completely?
displacementFilter.filterArea = app.screen; // not necessary, but I prefere to do it.
displacementFilter.padding = 0;
That'll do it. Why did I modify filterArea there, PIXI will "fit" it anyway, right? I dont want PIXI to have time calculating the bounds of container, because maggots are actually changing it, crawling in places that we dont see!
No extra transparent space, and if you put it into http://pixijs.github.io/examples/#/filters/displacement-map.js , and you move mouse to bottom edge, you'll see the grass.
Mapped matrix
When you want to use extra texture to put in the filter, you need to position it as a sprite somewhere. We are working with sprite that is not renderable but exists in the stage. Its transformation matrix will be used to position your texture in the filter. Please use https://github.com/pixijs/pixi.js/blob/dev/src/filters/displacement/DisplacementFilter.js and http://pixijs.github.io/examples/#/filters/displacement-map.js as an example.
Look for a mapped matrix:
this.uniforms.filterMatrix = filterManager.calculateSpriteMatrix(this.maskMatrix, this.maskSprite);
maskMatrix is temporary transformation that you have to create for the filter, you dont need to fill it. Sprite has to be added into stage tree and positioned properly.
You can use only texture that are not trimmed or cropped. If you want the texture to be repeated, like a fog, make sure it has pow2-dimensions, and specify it in baseTexture before its uploaded on GPU/rendered first time!
texture.baseTexture.wrapMode = PIXI.WRAP_MODES.REPEAT;
If you want to use an atlas texture as a secondary input for a filter, please wait for pixi-v5 or do it yourself. Add clamping uniforms, use them in shader and make better mapping in "filterMatrix"
but then... the lines between tiles are back when moving the container...
