shadow color/transparency

andy.engelkemier · September 27, 2016, 10:09:53 AM

I'm sure most of you have had to deal with this issue. You render out your product on a white background, but were smart enough to save it with alpha so you can drop it onto something else.
The issue is, your shadow isn't black, it's grey.

I was thinking perhaps it was premultiplied on white. When that is the case, you can open up your image in AfterEffects and change the interpretation to premultiplied, then choose the color. Or in Photoshop you can choose Layer>Matting>Remove White Matte. Both options should solve the issue.
That's not the case, which leads me in the direction that something else isn't what we are expecting. I'm hoping to head from Keyshot on that one.

What we Were doing was changing the background color to black prior to your final render. That way your shadow is actually black. And yes, that works. Only, the sampling is done on the image, without taking alpha into consideration (that's my guess). So when you render on black, it's only producing the minimum amount of samples. So either you have to increase those Considerably, which will take forever to render, or live with a pretty bad looking shadow.

So the solution that I have been using is to just render it on white, then select the alpha of that layer, create a new layer behind that, and fill with black. That gives you a black shadow. Then mask your render so that there is no shadow.

That solution is fairly quick, although shouldn't be necessary, but doesn't produce perfect results. The antialiased edges end up with a black thin halo around your object.

If you Really have some more time, you can create one render with no ground shadow.
Then duplicate your entire object and add an emissive material to the whole thing with a value of black. You can then turn off visibility to camera. Render another pass that was and you'll get Only the shadow, which you'll also have select the alpha of that and create a new layer to fill as black because it will have the same "grey shadow" issue.

Even with these tricks, my ground shadows lately have had some ugly artifacting on them. I even tried using a ground plane instead of accepting the shadow. I've just been adding a little surface blur in photoshop to fix it for most of the renders since ground shadows aren't all that important to be 100% accurate.

I personally find all of these tricks to be a bit of a pain and wish they were unnecessary.
Does anyone have any other solutions?

monson67 · September 28, 2016, 07:40:32 AM

I can relate. I've dealt with most of this myself. Very time-consuming to get the desired end-result.

andy.engelkemier · September 28, 2016, 07:46:53 AM

I'm hoping some folks from Luxion end up commenting on this. The quality issues, how they intended to use shadows, etc.

I feel like either the shadows should be saved as premultiplied, or Only include the shadow color and no background color. Shadows should only include the light color, not the background color. In most all cases, that should be black.
I would LOVE to be able to set a sampling value for the shadow though, all by itself. They currently look like garbage while the rest of the scene looks fine. I don't want to sacrifice render time just for a ground shadow. It's odd that some of the shadow also looks much worse than others. Where it's close to the product, it's much less quality. As it gets further away the sampling suddenly changes. It almost looks like jpeg artifacts (that's obviously not what it is though) just on the ground shadow. The sampling is an ugly repeating pattern, so even when blurred, you still see the pattern.

Chad Holton · September 28, 2016, 08:19:21 AM

Pseudo ground shadows may be a good alternative at the moment:

https://youtu.be/Tvh5t2UyuFE

andy.engelkemier · September 28, 2016, 08:24:23 AM

If you're making your shadows that way, unless you're doing an animation, there's absolutely no reason that you should be rendering them every time.
But I see where you're coming from. It's a definitely alternative, especially if you're rendering an animation.

But that's not why I would post something like this. That trick has been around for probably 20 years. It's a shame to still have to use it.
Wouldn't it be nice if the ground shadows just worked ok?

monson67 · September 28, 2016, 08:36:03 AM

It's not just shadows that get the background color blended in. It's the 3D object as well.

Anywhere there is an alpha transparency taking place, it gets blended with the background color. The reason why it's more noticeable in the shadows is because shadows have alpha throughout, beginning at the edge of the 3D object, and increasing in transparency to the outer fringes of the shadow.

But look at the edges of the 3D object itself in your renders. It has an anti-aliased alpha transparency as well, albeit very thin, around the edge of the object. Change your background to bright red, render your image with alpha, and you'll see a thin red line all around the edge of your object.

With the line of work that I'm in, this is a considerable inconvenience. Images that I'm creating get overlapped with other images, each using different alpha channels. The resulting composite images have these harsh lines around the edges of each individual image, making it very obvious that the end result is a composite of multiple images, thereby making it feel less authentic.

Ideally, KeyShot should allow users to render images without blending the background color into the alpha.

andy.engelkemier · September 28, 2016, 08:50:53 AM

You make a great point, however I am fairly sure that's impossible. BUT I have a solution for you. I'll get to that slightly later after I describe for others why it isn't easy, but I suppose it is possible.
The way antialiasing works is the camera "ray" is shot into the scene to see what it hits. If the "ray" is between two surfaces, say a black and white, then that pixel ends up 50% grey. The next "ray" up may end up more of a 25/75 split, so could end up being 25% grey, and so on.
So when you're talking about the edge of an object, what is the color?

Ideally, the 'ray' shot out would detect that the object edge has been hit. You could just provide that color for 100% of that pixel and discount the background. The alpha would reflect the split instead of the color. Currently Both the color, and the alpha share that same split, which is why we have the shadow issue, and the halo around your images of whatever background color (this is why you should never render on a backplate directly if you plan on putting your image elsewhere later).
My guess is that if you only take into consideration the value of the edge of the object, you'd actually end up getting an odd halo around your object , but I'm not sure. I think all of the other software I use has this issue as well, but I'm not quite sure.

SO, how to fix it? Well, in AfterEffects it's really easy. You have several options. Right click on your import and choose "interpret footage." You can then change from ignoring alpha, premultiplied on black, white, or a custom color. If you rendered on a backplate, you're pretty much hosed in terms of removing that in any sort of easy way.

In photoshop you can remove the matte as well. A the bottom of the "Layer" menu are some options to remove black/white matte, as well as "defringe" which will help any other color. The degringe option will screw up your shadows or semi transparent objects.

Any object that is somewhat transparent is going to have to render what it's seeing through to. It's transparent right? You'll be refracting Something. So that poses it's own issue.

sethcshort · September 28, 2016, 10:19:13 AM

I didn't know about the "Remove White Matting" option in the Layer menu -- thanks for the tip.

I was just playing around with that, and discovered something. If you hit Remove White Matting TWICE, it seems to create the effect you're looking for, IE, black shadows that aren't gray, and no white halo.

Examples attached.

The models in this image still appear to have a white edge -- but I'm attributing that to their specularity in a very white environment. And in the white-background version, there seems to be a thin black edge to the metal object, which might be due to the de-matting or might be a reflection.

In addition, the de-matting effect seems to be strengthening the shadows CONSIDERABLY. I'm attaching the original on white background for reference, too.

In other words ... it's not perfect, but maybe more palatable than multiple renders composited together.

Seth

andy.engelkemier · September 28, 2016, 10:24:28 AM

hahahaha. Hmmmm.
That makes me feel like they Almost got it right and are somehow premultiplying wrong?
I'll have to check that in after effects as well and precomp that, then reinterpret the footage a second time to see if it's removed.

I'm not sure why I never tried hitting the button Twice. I will definitely play around with that to see if there are any issues aside from semi-transparent objects. It will also remove extra white from those, which may not be desired.

Also, have to tried different color backgrounds? I think hitting it twice Might create a darker halo around your object. But I'm not sure. I don't have time to check that today myself. This is a model and render day, not a photoshop day.

sethcshort · September 28, 2016, 10:34:03 AM

You're right -- there's definitely a tiny dark halo around the black plastic model. Dang.

andy.engelkemier · September 28, 2016, 10:37:34 AM

Shoot.
I'm hoping that Keyshot can just fix this some time soon. It sounds like a multiplier just isn't set correctly somewhere. Tracking something like that down is probably Quite difficult though. Hopefully they can shed some light on the situation soon.
I'd like to minimize my workflow as much as possible.

Rendering out the shadow all by itself, then the product with no ground shadow is still definitely the best quality, And the most flexible. I keep meaning to put a quick-tip video up on how best to accomplish that, and create the photoshop file for best flexibility. Perhaps I will do that soon.

sethcshort · September 28, 2016, 11:03:53 AM

In that vein, here's the previous image overlaid by another render without ground shadow. It's got a bit of white halo, unfortunately.

So I applied a single Remove White Matte. (A second application resulted in a heavy black line.)

One more try -- reducing the background (which we're only using for shadows at this point) to 25% opacity to more accurately represent the original intent of the render.

And then that takes us back to my personal go-to technique, similar to what you mentioned -- an untreated render with ground shadows enabled in the BG (set to Multiply mode to remove the white), overlaid by the same render but with no ground shadows. And, thanks to this conversation, the addition of a single application of Remove White Matte if needed.

There may be something useful here in a pinch -- but yeah, fixing the problem in the renderer is the only true solution. Fingers crossed.

Seth

sethcshort · September 28, 2016, 11:10:20 AM

And just to test that we haven't broken anything with that last comp, here it is on white compared to the original, and on black.

Rex · September 28, 2016, 05:51:51 PM

Quote from: andy.engelkemier on September 28, 2016, 07:46:53 AM
I would LOVE to be able to set a sampling value for the shadow though, all by itself. They currently look like garbage while the rest of the scene looks fine. I don't want to sacrifice render time just for a ground shadow.

You can do this by increasing the Shadow Quality in the Lighting tab.

andy.engelkemier · September 28, 2016, 06:02:03 PM

Haha. That number reminds me of a cooking show where a chef asks a kid how much salt to add to a dish, to which the kid replies "five". Aturally you would ask "five what?" But get no response.

I always change the shadow quality. But it hasn't been changing the artifacting described which Looks like a sampling issue. Thinking back to hypershot I was thinking that quality number actually represented shadow map resolution, which would have not much to do with more than the edge quality of shadows. But GI shadows don't work like that unless keyshot is just making a dome of lights based on the value of the image at that location ( old school faked gi kind of look that renders pretty quickly).

I'll have to look into the issue and see what combination of things causes/fixes it.