Keyshot for zBrush. Out of memory crashes.

[Hellborn]

Ok.

As Keyshot don't support HD geometry from zBrush I did some tests to instead go higher on ordinary in subdivision in zBrush as it now supports 64bit, it should be possible.

So I transferred a close to 20Mil polygon model and while it took a long time to transfer it worked really well once it was finished.
Both Keyshot and zBrush running along just fine.
Memory usage run at about 10Gb with a peek up to 15Gb (I have 16Gb) and after completed transfer it leveled out at about 13Gb.

So I tried another model.

Just as the previous model it had fibers, about twice as much but the total model was of modest 2,8Mil polygons.

This transfer took even longer and in the end Keyshot crashed with an out of memory message.

How can a 2,8Mil polygon model run out of memory when a 20Mil polygon model worked fine?

In order to figure out how much resources are used, I loaded the same parts into my Octane render engine, expecting it to crash as I only have 4Gb VRAM. But, it loaded the models in no time. I can see that the polygon data  only uses 0,5Gb VRAM.
I understand that in the transfer to Keyshot textures are created as well but that can hardly explain why so much more memory is used. If I consider the almost 4Gb RAM that  zBrush used then the bridge and / or Keyshot  is using more than 12Gb.
Any ideas?

[Hellborn]

Removed averything except the fiber hair from the model and got it down to 3.3Mil polygons.
When I start the transfer System and zBrush uses about 4Gb of my memory.
When Keyshot loads and starts to load the data memory usage rises to about 8Gb.
Memory usage varies a little but climbs slowly to about 10Gb.
Then the setup part starts and when it reaches about 80% of the progress bar memory starts to rise and tops at 15,7Gb and after 10 minutes or so from starting the transfer I get a message that warns for instability due to low memory resources.
After that point nothing more seems to happen so I closed down Keyshot.

I can't see why such a small file should use such huge amount of resources.

I also loaded the 20Mil polygon model again (saved as a Keyshot file) to get some info about the resources such a model uses.
Keyshoot reports it having more than 56Mil triangles.
Ram usage by Keyshoot is 6.2Gb.
When i transfered that model to Keyshot, RAM went up to over 15Mb as well but only for a second or so.


I can send you the 3.3Mil polygon fiber hair model if you want to use it for testing.

[guest84672]

Yes - please send it to us. You can send to me directly using keyshot.wetransfer.com, using e-mail thomas@luxion.com.

Fiber hair is converted to polygons. So even though you may only see 3.3M polygons in Z-Brush, you will see a much higher polycount inside KeyShot. And higher polycount means higher memory usage.

Thanks, Thomas

[Hellborn]

I will send the file to you.

But until you get it.

As a comparison.
I just transferred a model with 29Mil polygons whereof 2.5Milj polygons being fibers.

In the start of transfer Keyshot uses 6Gb RAM and a while into setup it jumps up to 13GB.
When transfer completes the memory usage of Keyshot is around 3,5Gb.
The imported model is reported having 89Mil triangles.

Without the model Keyshot uses 0.5Gb so the model makes up for about 3Gb.
It seems to me that during transfer Keyshot uses 4 times the memory the model actually take up once it's imported.
That don't seem to be a very resource effective transfer.

Anyway, a will make an extra attempt to transfer the failing fiber hair before sending it to you.
Will be back with results in short.

[Hellborn]

Ok.

Made new test with same result as before.
As message say, "Out of memory (requesting 48241 MB)!!! This may cause instabilities..."
I thought that maybe it's not that much more memory it needs so I closed down zBrush after merging was done, thus freeing up 250Mb extra, but I got the same message anyway.

File should be on it's way.

[Hellborn]

Made even more tests and I'm quite sure that it's just a matter of running out of memory.
Fibers generates not many, but rather a massive amount of polygons.
1Mil fiber polygons seems to end up generating about 30Mil triangles with the default settings for fibers.

Max fibers with 16Gb RAM would probably be around 3Mil.


Question is.

Can I use better settings that looks just as god but uses less polygons?

Is there any chance that you will be able to optimize the import so that it won't use such huge amount of memory. Being such a fresh release one can always hope that there is room for improvements.
My test indicates that there is a short peek at the end of the setup phase in witch Keyshot uses 4 times the memory that the model use when transfer is completed. If you could get that down to 2 times it would only use half as much memory, making a huge impact on memory demands?

[KeyShot]

Fibermeshes will be optimized soon. We do have native curve and hair rendering in KeyShot.

It would be good if you could share the model, so we can investigate the memory spike. The bip file should be fine. I assume that if you save the file in KeyShot and load it again you also see the spike in memory usage? Or is it only during the data transfer?

[Hellborn]

I did send a zBrush file to thomas@luxion.com. If he has not recived it then i might have done something wrong.

I tried to load a 89Mil triangle bip file right now and I can see the same peek.
It uses all memory on last part of setup.


Using close to 100% RAM is of course not a bad thing.
That's just using the assets to maximum for maximum performance.
At least as long as its not taken to far...

I guess that is what you do during the end of the setup phase.

Could it be that the transfer is more sensitive for some reason?


With some luck, you will get the same error if you transfer the zbrush file i sent on a  computer with 16Gb RAM.

[guest84672]

I got it. Thanks for sending.

[Hellborn]

Have you tried the file yet?

Tried to increased the virtual memory from 16Gb to 32Gb in case it not only runs out of memory but virtually memory as well.

Made no difference.
Still get the same Out Of Memory message.

[Hellborn]

Tested a bit more..

Decreased BPR SubD setting for the fibers from 2 to 1 and Sides from 4 to 3 and got the model to load.

Increased to 4 Sides with went well.

SubD 2 with 3 Sides also loaded but this time the peek probably used a 2-3Gb of virtual mem.

At the settings used for the the transfer that fails almost all RAM is used before the peek so I estimate some 5-10GB of virtual mem being needed in this case.

So, the model will need virtual mem to load but with 32Gb virtual mem there is plenty of it.

[TheKid]

Hello

I just stubled across this thread while searching for answers regarding my out-of-memory problems when transferring fibremesh. I haven't gone into as detailed analysis of spikes and such, but all the symptoms are similar, so my guess is that the problem is the same too. I am glad to see, that there is some hope of a curve-based solution and I hope that it will be possible to get rid of the spikes along the way.

[KeyShot]

We have identified the increased memory usage. It happens with large single meshes. We will work on optimizing it for KeyShot 6.

[Hellborn]

Great!

How far away is Keyshot 6?

Also, when will we get the optimized fiber support?

[TheKid]

Now I have done some testing based on the statement that the problem occurs with large single meshes: I intentionally made a very heavy fibremesh - about 4-5 times the fibres/segments than the mesh that I originally had problems with. The new mesh had about 20 polygroups and before transferring it to Keyshot I did a split/groups on it. The result was that it transferred without problems, so it seems we have a functional workaround. That being said a curve-based option would be better.