Hi everyone!

I've been testing an alternative way of rendering point clouds, using DOF to help give a more photographic feel and actually use the 3D information in 2D renders. They look like this:


http://www.flickr.com/photos/kylemcdonald/4136191836/in/photostream/

http://www.flickr.com/photos/kylemcdonald/4135432059/in/photostream/

Unfortunately, the only way I've found to get them looking nice is by rendering into a floating point FBO without depth testing, so that I can draw each point as part of a GL_POINTS call with their point size and opacity based on distance from the camera's focal plane (opacity is 1/(distance^2), point size is just distance).

This means that there are a lot of points that are very large being drawn to the screen. So every pixel has to be passed over multiple times. Right now with about 315k points I only get 3 frames per second if I turn off point smoothing (i.e., use a square aperture).

Does anyone have any ideas for how you might get a similar effect with a single (or just fewer) passes? Something that looks about the same, but could be rendered in realtime? I'm familiar with some approaches like this http://developer.amd.com/media/gpu_assets/Scheuermann_DepthOfField.pdf but skeptical that they would approximate the effect I have here with point clouds.

Any help is hugely appreciated!

hey this looks really amazing - nice work!

One thing I had pretty good luck with is having 3 to 5 FBOs. Each one is blurred (with a shader) by a different amount. Then you draw each point into each FBO with the opacity being a percentage of the blur you want.

This could end up being more calculations however because you are drawing each point more than once per frame.

Something that could help your performance is VBOs - I think memo has used them quite a bit when working with large numbers of particles.

Good luck!
T

Thanks!

Multiple FBOs could help, but then you have the issue of drawing multiple transparent FBOs on top of each other, too. And there seem to be some unresolved issues with drawing FBOs with transparency on backgrounds. The whole "layer" based idea is really good though -- I can imagine doing it in N passes, where you divide all the points into one of N bins based on their distance from the focal plane (drawing only 1x1 px points) and then blur the entire layer using a frag shader to account for the "point size" and opacity. I feel like the artifacts might be really obvious using that technique, though -- you'd need to have a high N value.

I was considering VBOs (that's what Memo recommended too), but I'm positive they're not the bottleneck. If I just replace glPointSize(distance) with glPointSize(1), everything speeds up to realtime. So it's definitely the fill rate of the points that's killing the fps.

that pretty much says it all! maybe you can roughly calculate how pixels you are drawing to the screen (sum of pi * r^2) and compare it to you graphics card's fillrate (usually givin in Million pixels per second) and see if you are indeed near it.

The only way to avoid fillrate bandwidth issues, is to draw less pixels! This can be done through depth testing, alpha testing (i.e. do not draw to screen if alpha is say, less than 0.5) or other tests, but it seems you need to draw each pixel and blend them.... not looking good

hmm maybe you could use opaque particles instead? Then you could make use of openGLs occlusion culling and most likely discard alot of them before they get sent to the GPU. some sort of spatial hashing / octree based on their position could help alot aswell because you could discard a whole bunch of particles with just one occlusion query. (also consider frustum culling)

Another benefit of opaque particles is that you could of course use depth test and maybe even an early z-pass!

This leaves the question open on how to get the DOF effect. Actually when well done, the technique described in the paper could work pretty well, i myself implemeted it a while back:
http://vimeo.com/1746286

Another easy trick could be using point sprites and one big texture with different blur stages of a circle (or whatever you prefer). Then you could adjust the texCoords depending on the distance to the cam focal plane in the shader.

The look might not be exactly as smooth as the blended ones but you might get a big performance improvement. Also mipmapping the particle texture will help alot.

To get something more similar to your current aesthetics, you could simply add a bloom postprocessing effect or something similar i guess!

Hope that helps.

Thanks for all the replies!

moka, I can't use opaque particles with depth testing enabled, unfortunately, as the effect relies on the ability to see through points. Here's a video that might help explain how essential that is:



But I'm still thinking about this layering approach you and Theo have described... it seems the most promising. Some bloom probably wouldn't be bad either I am definitely interested in a photographic feel.

Memo, I did the calculation and I need anywhere from 1-100 billion pixels filled per frame. I calculated this by summing the area of all the points in different circumstances. My graphics card is a mediocre laptop model http://www.m3fe.com/fillratetestweb/ViewResult.php?id=608 apparently on the order of 1 billion texels/sec...

hmm well I dont really get why the trasparency is essential. can you maybe show a screenshot without blending?

Wooah Kyle that video looks amazing!

This is using your structured light stuff, non?
Do you have any code available for this stuff? I remember some related posts but my memory fails me...

Personally I think you're in a tight spot - any different techniques are going to look, well, _different_ as the combination of point size and opacity gives it its particular look. If you want to speed it up you will probably have to compromise and use a different technique, but it may not look the same....

It's definitely worth trying Moka's suggestion, rendering opaque particles and then blurring in a shader, but you probably will lose the ghostly, transparent effect.

moka -- certainly, I suppose that might have been more helpful than the video


http://www.flickr.com/photos/kylemcdonald/4141401350/sizes/o/

You can see that the large opaque points end up occluding each other and creating sort of modernist forms rather than a photographic feel.

Pierre -- thanks! Yes, this is the structured light stuff. All these scans were done in about 2 seconds using a digital camera and projector, capturing 3 frames. There is an addon I'm working on called "ofxStructuredLight" hosted here http://code.google.com/p/structured-light/. Chris Sugrue has helped a ton guiding me towards making a sort of binary app that people can run. I'll be finishing an initial version and releasing that in a couple of weeks (before December 10th).

I had an idea last night that is an extension of the layering idea... moka mentioned octtrees, but he was thinking for throwing away occluded points. I could try instead using octtrees the way they're used for Barnes-Hut -- if a bunch of points are sufficiently blurred and close together, I can approximate them as one large blurred point.

ah i see, good idea with the layering, definately worth a try!

hi kyle,

i built a DOF system recently for textured quads using imposters. with imageMagick i created 8 textures for the same image, each with an exp/^2 increasing gaussian blur effect applied. then i worked out the focal length/image plane difference and rendered the appropriate image. this worked quite well, even with only 8 bands. from memory i may have also blended between two adjacent images to make it look nicer - ie rather than an integer index i used a real index and then rendered twice using the two images on the integer indices on either side of the real, blended appropriately, eg 3.2 -> image 3 with 0.8 alpha and image 4 with 0.2 alpha.

so, what springs to my mind for your case is that you could make use of the depth-banded nature of DOF to help you reduce the horizontal aspect of the rewriting issue (not necessarily the vertical though). essentially this is about re-stating the problem in terms of the distance from the image plane to the point, rather than in terms of the image plane (x/y).

1. the effect you end up with at the moment clearly has a kind of a banding effect. this could be exploited in an LoD kind of way.

2. at the centre of a given band, the opacity is the same across the entire band. therefore it should be possible to precalculate the final colour of the pixels within that band easily without multiple writes in the image-plane-distance x axis.

3. once you have precalculated the final colour of a given band, then you should be able to render that band in one go. rendering a line of pixels with precalculated alpha (even if the line deviates from straight) should be more efficient than writing multiple times to each pixels.

4. also, notice that as the blur amount increases, the bands get wider and wider. essentially, when the individual points are very out-of-focus the resultant pixel data reduces to smooth colour fields, which could be rebuilt/precomputed somehow since they visually do not benefit from multiple writes. just thinking now - perhaps you could even do this by clumping the image points together when the blur value is high: collapse 9 or 25 adjacent image points into one larger one by averaging colour values..?

5. as with any LoD based method the transition between detail levels will be crucial. a smooth transition is better than a binary switch.

i hope this is helpful. essentially you're dealing with a compression problem i suppose, and that means that you just have to figure out where you can throw source data away by reducing it to a more abstract form that still looks as good. throw away individual blurred points and reduce to smooth colour fields. JPEG compression in 3d innit.

Damian, that's super helpful! It's essentially what I was imagining when I wrote above about Barnes-Hutstyle octtrees for approximating regions where there are solid color fields, but you've described it to a point where I'm more confident it will work.

I'm surprised you can get a good effect with only 8 bands. I wonder if you even need an octtree-type structure to store things, then? Maybe you can just go through and do a single O(n) pass, classifying each point and writing it to a buffer for that layer/band.

Are you calling glPointSize for each point? Would be a lot quicker to pass in a vertex attribute array containing the point sizes. Here's a bit of code I was using for something similar. All my particles were the same colour but you could easily pass in a colour attribute as well.



vertex shader:


fragment shader:

Yes - I did this for the VBO particle system here:
viewtopic.php?f=9&t=2639&hilit=vbo&start=15

re good effect with 8 bands - yeah, i was surprised too, but in retrospect it makes sense. the key is to make sure that the transitions are smooth. i mean, with photographs taken with a real camera, you'd be hard-pressed to group them into 8 categories of out-of-focus-ness, which means our brains aren't so clever at identifying. it's just when it suddenly switches that we notice..

i think i could've even gotten away with a situation where there's a transition region (10% of the band width, for eg) when two images are drawn and blended, just to make it look nicer when animating, leaving the other 90% of the region rendered with just one (static) texture.