Softbodies and soft bugs...

In the interests of interest, I have been working on the softbody section of code in the fluid plugin. Thats right, its called Physics.jar for a reason. The entire structure and methods used are there so that fluids, softbodies and rigid bodies can be integrated into one single engine. This will allow fluids to interact with these objects. But it does mean less optimized "sub solutions".

Currently the softbodies are working, with some known shortcomings while I sort out the finer details. Also I keep finding bugs. Right now there needs to be a little bit of a clean up of the raw simulation code. I have too many methods that don't properly document the contract for the engine. So boundaries are a bit of a interesting mix of problems really. Its all experimental code at the core and thus its far from clear what methods work well in practice. A good example is the no slip boundary conditions that are frequently used. When you are using fixed mass particles this turns out to be a really bad approximation, as the boundary layer is smaller that the particle resolution. Simple friction with velocity normals work very well and are much more stable.

I have now read more papers than I care to count on the subject of SPH. There is a general pattern however, and that's that its generally about a 1st to 2nd order method at best. Boundaries are where the side is let down, and even thou there are corrections (that I use) the smoothing kernels limit the local approximation across the parameters. To make a long story short, I'm not aiming for a 3rd/4th order method. In fact I estimate that I have 1st order at the boundaries and 2nd order throughout the fluid. I use only first order explicit time integrators. The combination turns out to not be so bad after all. So I keep the code that works, and its a little adhoc right now.

So the current softbody list of features and antifeatures:

• Interact with each other
• Interact with fluids. ie can float
• Must be a triMesh
• the mesh can be set to approximating to smooth deformations.
• Interactions with each other is from vertex to face. So few vertexs means they tend to interpenetrate a bit.
  
• Unfortunately you need quite small time steps to keep stable because of the explicit integration scheme.
  

And a very small example...
Soft body and fluids

The first Alpha Beta release....

The final dusting to the poly boundary code have been completed. So now you can play with the new code. There are a few extra options now for boundary's that I will try to explain here. At this stage its about making options avalible so that others can play with setting to what works and what does not.

The first new option is the PolyBoundry. This makes it a polygon type boundary rather than a vertex based boundary object. The remaining options don't work unless PolyBoundry is true.

The next option is the calculate boundary outside poly. What this means is does a polygon effect extend out from the edges rather than the projected normal volume. In more technical terms, if we project a particle with the plane normal onto the plane that contains the polygon, do I do anything if its outside the polygon. Generally setting this to false works best. But for convex shapes there is the possibility of "leak vectors" . Setting it on mean that concave sections give too much boundary force.

The last new option is really for the case where Calculate boundary outside is set to false. Generally we will get very slow leaks over time. If we provide a constant velocity bias to the boundary conditions we help move particles that are close to leaking, back inside. But this is adding energy to the system and therefore can lead to pretty weired results. Below are some examples, but this should be small 0.01 or smaller. Always try 0 first. Even at low values that can make still fluids "boil".

There are some limitations to the current implementation. The biggest is the complete lack of friction. All current boundary's are frictionless and that can lead to bizzar results from what you might expect. Another is getting fluid boundaries to match the boundary surfaces for rendering. Currently there is no way to ensure that ray tracing a glass with water will be done correctly as far as I know. Its also not currently optimized either meaning that its slow that it needs to be. But its still faster than particle boundaries. The last limitation is the need to put the boundary triangle into all voxels that it overlaps. This means that using smaller particle sizes will increase memory requirements for the boundary objects.

So here is an example of the unstable rotation that is now, hopefully fixed.
rotator mpeg 4

This a example scene with the outside boundary's set to true.
Hour Cup
Note the large distance between the walls and the particles.

The same thing but with outside boundary's set to false.
Leaking
It now leaks, but only slowly.

Then we add just a little velocity bias (0.01).
bias fixes leak
We can really see the lack of friction here.

Getting stupid with bias (0.05).
Too much bias
Now the lack of friction is compounded with the extra energy added by the boundary.

Finally the file I have used.
AoI example file

This plugin is now only for AoI 2.6. But only because I don't use 2.5. Also its a beta-alpha for a reason.

Download the Physics Plugin

Polys for boundaries and stupid bugs

Oh yes, Its finally done. We have polygon boundaries so you never need to subdivide a mesh again. This should make it about 200 times easier to use now and I will get much less over/under subdivided meshes, asking where it all when wrong.

Ok, so its not finished. The source objects only use vertices's, and we still need to convert to triangle meshes and you still can't have a initial condition that includes a full glass of water. But otherwise this is working with the only regression I have noticed is the save state issue discussed below.

However there was one bug that took almost the whole weekend to find. When I had a object rotating under a fluid source, it would go unstable. Even with time steps down to 1e-6 it would still go unstable. To make a long story short I did not really find the bug. But i did prove that it *only* occurs on perfectly axis aligned vertices's rotating in the x plane! Even rotated .1 deg in the y axis would prevent the problem. I think I fixed it but I'm really not sure. Its possible that you just need everything to be perfectly aligned in some other axis....

But its good enough for me. So I will package this up on Monday as a beta-Alpha release for those who want to give it a try. Remember that performance on the old highly subdivided meshed will be very poor at this point in time, so a "clean" slate may be best. Also remember that there is no guarantee that there will be any level of AoI file backward file compatibility.

Also I will update the blog with a link to some new movies Monday as well.

Real Triangles, well infinate planes anyway...

The first big steps to removing the need to subdivide meshes has been taken. The biggest chunk of the required infrastructure is in place and working. The triangles however are approximated as infinite planes instead. But due to the fact that I use the particle boundary code there should not be to many issues to iron out as most of the code has been used a quite a bit.

I make one quite big assumption. I approximate all properties across the face of a triangle as a linear combination of the vertex values. This means that velocity of the face is not accurate for large polygons that are rotating, but this can be solved simply with subdivision for these perhaps quite rare cases. Another consideration is that the force of the fluid is transfered to rigid bodies in a similar fashion for rigid body dynamics, and this could also lead to inaccuracies and perhaps instability.

The code is not in the best state, and there is still quite a bit of try and comment out code as well as out right hacks in there. The is already one regression. The plugin does not preserve state of object that are created after the physics plugin for example.

In order to catch regressions early, I was thinking of releasing regular snapshots that are simply totaly untested. If folks would like this it would mean that regressions would be picked up sooner rather than later. Otherwise the whole debug cycle gets a little slow. Let me know what you think.

So the next *proper* beta release (0.7) I will hopefully have the following fetures.

• Full polygon boundaries for moving and stationary objects.
• Basic Rigid body dynamics.
• Animating parameters.

I have also noticed some serious performance issues. Its just much slower than it should be. This will perhaps not get looked at in the next release.

Laptop

In the interests of getting work done when I'm on a train, I just got myself a new laptop. This is in fact my very first laptop and unfortunately it has been cursed with Vista. But I already have cygwin working and will put Linux on it once I have at least some good uptime with a live DVD.

In the mean time this means that it will be much easier for me to work on things like the fluids plugin when I'm away from my fixed machines.

Other Things

This is just to give a update on what i have been doing. Although it was not on the AoI fluids plugin. But it is not forgotten, and if i ever do abandon it, I will release a full source version. Remember I want this plugin too. Your feed back was read and will be reread, so its not wasted.

So what have i been doing. Well I been to New Zealand and back. I have got a contract in the wings that means I can stay in Austria for another 2 years (maybe I will learn some German?). I have noted my lack of ability to Get Things Done, found a book with that title and I am finding it really helps to, well get things done.

Last but not least I have migrated all my code from subversion to Git. Oh what a joy git is compared to cvs and svn.

Its interesting how I found out about Git. I started using CVS for everything in about 1995-7 (can't remember exactly as I cleaned and reset my repository in 2000). I was not that impressed. I could not do a lot of the things I wanted to like move files. Binarys was not well handled and some things were far slower than I thought they should be. I did try a merge from a branch once.... and only once...

Then I ended up working for some companies that used source safe. Now I hate working on windows at the best of time. Nothing personal, I just find that MS support and documentation is terrible compared to other companies I have worked with. But what the Hell was MS thinking with source safe? Surely they didn't use it in house? To make a bad tool worse, the company I was working with had very stupid polices regarding file locking.

To make a long story short, I fell in love with CVS because she was the prettiest girl at the party. Not because she was pretty.

I kept looking for a better way because there must be a better way. Hence I found subversion. I have now used it for about 3 years. The problem is that its not really a better way, but the same way. Combined with the tools in IDE's (eclipse mainly but netbeans as well) my repository kept getting into limbo states preventing commits. It was difficult to fix without a complicated rolling backup system. Further to that problem, which directly affected the fluid plugin, was that I would end up out of sync with home all the time (I had no internet).

It was the fluid plugin devlopment that got me thinking that my personal devlopment model was in fact distributed. It didn't take long to find git. Unfortunately the admin at work never would give me a time when he would install it at work. So i waited until our machines got upgraded which was last week. Now all my repositories have been transfered to git. I don't use IDE tools for version control anymore since they just don't work that well. But with git is does not matter because the checked out directory is clean.

So why Git? Here is my list.

• Distributed development.
• Fast, very very fast.
• Merges work so branching becomes common. This means there is better history too.
• Unix tool set.
  
• Very good repository integrity checking. This is a big deal IMO.
  
• Easy to use (edit) configuration file.
  
• Frequent branching makes "frozen" release much easier to manage. Debugging on the frozen branch can be easily merged with the main devlopment trunk.

The is even some tools to import a subversion repository with history. So I still have a history back to 2000. In short, even if you are a solo developer I would still recommend Git first and distributed version control over everything else.

Feedback.....

Just to let anyone know that may care. I'm still waiting for feedback on the current state of the plugin before I update to the new version.

Also a few comments on what people get wrong.

• Yes you do need to divide the meshes up finely. But not that fine. As long as each vertex is about a particle size away your fine. I have seen some meshes with 10k+ vertex's. Thats too fine unless particle size is smaller that .001
• Normals pointing in the wrong direction. It seems that there are few issues with normals. Use approximated polygon objects, convert to triangle mesh and then set smooth shading.
  
• You need a lot of particles to get a "good" fluid. 10,000 is not a lot. Rendering time still dominate in my experience.
  
• Time steps are too large. If the fluid just disappears, its because the simulation blew up.

So unless i get comments to the contrary I will work on the following.

1. Remove the need for subdivided meshes. This should fix the normal problems too. This is however quite a bit of code.
   
2. Some small adjustments to the GUI, but they won't be big. Yet.
3. Try to get some adaptive time steps in there. This will be a experimental feature.
   

Please keep the feedback coming.

Beta 0.6

The servers are still down and could be for a while. But i found a good file sharing site mediafire. It works on all the browsers i tried (Ok it didn't work with lynx) and didn't swamp me with popups. Nik is going to also host the main jar as well (Thanks). Watch out for his link latter.

So there are big changes with the beta 0.6. Most are behind the scenes to get everything ready for other physics and particle work. Rigid body and softbodys should not be that far away, and including them with this makes fluid interaction much easier. Also gases and fluid driven Materials and textures (for the procedural texture/material editor) will also be not too far away.

But for the current uses heres the bugs that will impact you the most:

1. Found and fixed a bug in the find neighbors code. Now you can use larger time steps and not blow up.
   
2. Found and fixed a density calculation bug when the spacing parameter is smaller than about 1.2. Now any spacing parameter less than 2 can be specified.
3. Some bugs in the thread code that lead to things being done twice on dual core machines was fixed.
4. XSPH bugs were fixed and is now exposed to users.
5. Many more i can't remember.

Some things that I have not yet fixed:

1. The flash/flicker that some have seen in rendering is almost defiantly a thread issue. Where it renders a next frame before the data for that frame is loaded properly.
2. It is still difficult to get a "smooth" surface to the fluid. This may require a lot of work to fix properly. Some surface tracking methods are more complicated that the fluid simulation. We will see how it pans out.

The big things added:

1. Mesh previewer. Its is however slow and not that accurate for the resolution. Its not adaptive and will scale with the bounding box. Also you might want to turn it off when you render a ray traced movie.
   
2. Sampled Grid to speed up ray tracing. On very simple scenes on the dual core machine at home there is almost no improvement. But when AA or extra lights are added this starts to change quickly. At work on a slower single core machine its about 2x faster with java 1.6 and about 5x faster with java 1.5. Note that you get very weird results if Sample Grid factor is too small. 4 or larger are fine. Too large and you spend a lot of time building the sampled grid. Note this does not use a tree so the grid build time scales with the volume of fluid. This should make it fast enough to render transparent fluids.
   
3. Improved GUI for object management. Again its not finished, but its easier to use both for a user and a coder.
   

Unfortunately this has added a issue or two:

• Sometimes a when you load a file and raytrace it there is no fluid. I can't locate the bug yet but a workaround is to turn on the sampled grid (under preview/view tab) render and then turn it off again. Preview renders seem to give you a good idea of whats going on.
• What little documentation there was is now largely wrong. There is no other documentation at this stage.
• The mesh preview will rarely have a large missing strip on it. The fix is to either move the physics object very very slightly (The fluid won't move because its in world coordinates) or add a small about to the mesh size under the preview/view panel to not be a whole fraction or multiple of the particle size. ie mesh size = 0.050001.
  
• Previous save files will not load properly with this version. However AoI is robust and will load everything else and you will just need to delete the not loaded physics and add a new one and configure and save.

Quite a long list here. I will try to now just comment of a few catches when using the plugin.

1. When you change parameters like particles size. You must reset the simulation and rebake. If you don't the simulation can blow up no matter how small the step size. Generally if in doubt rebake.
2. If you bake a simulation but did not save the parameters set, then when loaded from a save file everything can be inconstant eg. particle size, and hence raytraces and meshes will be wrong.
3. Use lots of particles for realistic effects and use a high density threshold for emitters. The advanced option of spacing can be reduced, but will increase simulation time.
   
4. ensure that all boundary and emitter meshes are subdivied enough and that the smoothing method is set to shading.
5. If you try XSPH reduce the step size.
   

So where are the files:
Here
The plugin is the Physics.jar file.
Everything will move to the server I was using before when its back online. And don't forget. Its still a beta.

Downed Servers.

Unfortunately the raid (So much for redundant) of our server at work has died. I can get no word on when it will be fixed. 1 week tops. Very sorry about this. I might put the files in some temp place for a week or so.

Update

The Plugin has been updated. So this is version 0.51. Don't forget you need java 1.5.
So what was done?

1. Fixed a bug in the bake file saving code. Now you shouldn't get "stuck" as was happening before.
2. Found and fixed 2 bugs in the simulation code. It now should be more accurate. Note that in the advanced options you can select a single pass simulation mode. It might be faster, it will certainly be less accurate and may need smaller time steps.
3. Implemented a direct gradient finder for the ray tracer.
4. Provided a good default for max gradient. This makes ray tracing much faster. You might get some rendering artifacts however. Just increase the max density adjustment under the views and previews panel. Thanks goes to Peter for implementing this suggestion so quickly.
5. Fixed a another bug in the bake file saving where the y velocity component was getting placed on both the y and z axis. This would cause rebakes to be totally incorrect.
6. The coordinate system is now world coordinates, not the physics plugin local coordinates. This will change the direction gravity acts. Default is the -z direction.
7. Moving between frames with a lot of particles was very slow. This has been somewhat fixed and should be a lot faster now.
   

So because of these bugs you should update ASAP. Here are some results from the debug process when its all pear shaped.

Simulate

Render. Problem!

Now it works and only 12 mins to render.


Don't forget with these render times that I have a AMD X2 5000 and use java 1.6 (about 1.5x faster than 1.5). The simulation time was under 10 mins, but i don't remember exactly how long it was. Also the simulation code is single threaded at this time.

And almost forgot. The Plugin is in the same place. Here

Good news everyone....

Think the Professor from Futurama when you read the title.

Yep. The beta 0.5 is out. However the documentation is even more sparse than I would like. Remember this really is a beta.

Basic HTML getting started

PDF of the same thing

Failed attempt at a Flash tut I used wink, unfortunately screen grabs when a menu is selected/pulled down does not work. So its not so good. But maybe it will help someone.

Image dump of the above flash. Warning, quite large.

And finally the link you are all waiting for:
The Plugin and it comes with the whole No warranty deal.

And finally I forgot one very important thing. Trying to ray trace transparent materials will be Really Really slow (Glacial speed). This will be fixed at a later time. Also in the mean time report bugs at the AoI source forge forums.

No posts does not mean zero progress

Well something like that anyway. I have been working away at the plugin. First of all we have added everything for the 0.5 beta release. I just need to fix the last few bugs that bug me. Secondly I'm moving the plugin over to the AoI 2.5 format. So when will it be ready? Well soon.

Last weekend I just got sick of computers and took my family to Venice for the weekend. It was nice and much cheaper than i thought it would be. This weekend could be the weekend. Lets hope.

Anitbugs

An antibug is a bug that combines with another bug and "fixes" it. Last night I found quite a few bugs and then had to spend most of the night finding the corresponding antibugs. These were infact quite big bugs in the sense that they where mathematical in nature. So if the size of the particles was changed, the results were quite simply wrong. Fixing this has meant some performance improvements as well. Unfortunately it means I didn't finish the plugin to a beta 0.5 yet. Also I have two more things to add to the list of todo's. Firstly I want to put in the advanced Options and a "reset" button.

So here is the suface version of the jump with the old code:
Old code Jump

Here is a new version of the jump with the fixed code:
New code Jump
New code Jump, Side View

Reading the tea

So I hit the literature last night too see how much the field has progressed. How surfaces are generally done and incompressible SPH. There is finally a few papers on divergence free projection methods out there now. Because this has been used with other methods espisally for visually effects where accuracy is not so important (aka RealFlow I think) I was surprised that it took so long to be implemented with SPH.

In the future I will probably implement a divergence free fluid solver (via projection) as this will allow for very large time steps and allows for "vortex forcing", that is make the fluid more turbulent which is often the desired visual effect. As for surface tracking I will probably try a "colour" method to determine the normal at the surface. The unnormalized normal is a proxy measure of distance from the surface which provides a nice way to integrate with the isosurface solver in AoI.

Oh and here is a full simulation of the bent channel flow. It took 8 hours and has a maximum of 20000 particles with another 10000 boundary particles. This should be perhaps a bit faster really.
Zigging and Zaging
IsoSurface render

I also did a hydraulic jump. The iso render is cooking.
Jumping fluid

The reference to tea is because I cannot drink the large amounts of coffee I would usually drink. So now its lots of tea......

Final Beta 0.5 Stages

To my surprise I got to spend about 6 hours on the code yesterday. This was enough to get real close to a releasable beta.

What was added:

• Moving fluid inflow
• Moving fluid outflow
• Improved GUI
• Bounds parameter (removes particles outside the bound)
• Max particle parameter
• re-coded the interface between GUI objects and fluid Solver Objects

What needs to be done:

• Add support for save and load.
• Fix some minor bugs
• Documentation. You will need some to use the beta (perhaps alpha?)
• Add some parameters to the GUI, like max particle count.
  
• Add tool tips. Makes docs easier.

There are some other issues that have come to light. First is that there will need to be some serious improvement on the surface representation to get a nice look. This could be some work and will be done later. Secondly without adaptive time steps there are large performance issues. In particular I want to add adaptive time steps both in time and space.

So the 2nd Generation Fluid solver will have adaptive time steps and support for multi core CPU's and will be what I work on after the beta release. The new surface stuff will need to be later. We will see. Anyway heres some movies of the latest incarnation of the plug in.

Ugly Surfaces
Same as above in preview mode(15MB)
30000 particles is a bit slow

Appendicitus

I have had a sore stomach for some time. It finally got bad enough that I couldn't go to work and had to see a doctor. To make a long story short I got my appendix removed, and spent most of last week in hospital. The good news is that the Austrian Health system is fantastic. All up its going to cost me 30 Euros!!. Seems I got lucky and went to one of the better Hospitals.

Of course the bad news is that work is now further behind along with the fluids plugin. I was hoping to do some while I was away from work. But unfortunately It looks like that won't happen. Sorry. There isn't much to do for a beta. But its still not ready. The goal is to be at version 1.0 by the end of the year.

I'm very sorry it won't be sooner. But quite a few things have gone a little pear shaped this year so far. Nothing serious of course, well not too serious, but enough that things just don't get finished when we wanted.

So thanks for all the comments, they do help. All i can say about the beta is that it will be out before duke nukem forever.

Edging closer

I managed to write some code. Create some regresion bugs and then fix them. Here are the short movies. There are quite a few "bugs" that you can see in these movies, however thay are perfectly normal and with high resolution meshes and fluids you won't have problems with them.

Preview renders
A few seconds with the ray tracer

Note that the liquid surface needs work. This will come later.

Slowly

Very sorry for the lack of updates. I knew that march and April would be a write off. May is looking that way too unfortunately. This is why I tried to get it out in feb, since long delays were going to be the order of the day otherwise. Frankly I'm surprised that I have not had a few more "bumps" from the AoI users.

So I want to make one thing very clear. Its fine to request info on updates. Its makes a big difference on what in my spare time gets the time. I would like to say that I'm a super disciplined machine with great work/fun allocation skills. However I'm not. So don't feel you can't keep asking whats going on.

The other thing I want to make clear, is that it will get finished to version 1. I will push it that far and then open up the source. I may take to the end of the year, but it will get there.

Oh and just in case your interested. I have been teaching and writing for work. Also add some unprodctivness in there as well.

Options

So for the first beta there is going to be a lot of options. For the most part folk won't want to adjust them, however they should provide work arounds for most problems you will encounter. In particular since I still don't have adptive steps there is the problem of choosing small enough step sizes, otherwise the fluid blows up, more or less.

There is still a bit to do. I need to put in motion avraging code, theres the partical density and the scale factor to add. Also there is some code cleaning to do. At this point I will stick with the main AoI realease and work with the new beta stuff later. As a result renders are a little slower than they need to be at this point.

Same slosh without the box and using the raytracer

GUI's

Been working on the GUI again. Its getting there. It won't be finished by a 0.5 beta but then again I want some feed back on the general GUI design. Hoping to make some serious progress over easter. Although I have 6 hours of leasons to get ready. Heres the latest Vid.

Slosh in a Box