Shadows problem in 3DS MAX

So, I'm having a slight problem in 3D Studio MAX.
My fan film is toon-shaded and that can cause some weirdness with shadows. Ordinarily, that's not a problem, but in this case...well, here, let the picture tell the story:

[image=http://img.photobucket.com/albums/v293/rocketgrrl/Stairs.jpg]

Okay, these characters are walking up stairs. STAIRS, for @#$% sake! (You should, uh, probably ignore the fact that Aeon Flux is kinna, um...standing behind them. Heh.)
I've tried putting some texture on the stairs so you can see different levels, but it didn't really work. I've tried adjusting the lights so that shadows are actually cast by them, with similarly craptacular results (as you can see). I could move the camera behind them so the actually-shaded vertical parts of the stairs are visible, but that's what the next shot actually is.

So I'm kinna out of options, here. Unless someone on TFN has something I haven't thought of. Anyone got any ideas? Any help that actually accomplishes my goal of realizing this shot would be appreciated. Thanks!

 

I have almost no experience with toon shaders in max, otherwise it'd probably be an easy fix. Are you using the standard scanline renderer?

 

Pretty much, yeah, though I've also got a material plugin for the toonshading on the characters; the stairs (and the rest of the environment; pretty much any objects that aren't characters) are colored using default MAX textures that have been set to 100 self-illuminating. But I think that ultimately it's going to be a purely artwork fix; that's so much easier than trying to force MAX to do shadows the way I want it to.

For example, here's something I'm trying:
[image=http://img.photobucket.com/albums/v293/rocketgrrl/Stairs02.jpg]

It's ugly and I don't like it...but it does at least sell the idea that there's different levels there. I MIGHT even be able to get away with a thinner gradient; there's only a three-step color increment between each level of stairs...I haven't seen yet how two or even one looks, but I'm afraid they might be too slight to really notice.

This may have been a case of having to spell out the problem for others before finding the solution myself...but, as I said, I don't really like it. I'm open to other suggestions.

 

If it's set to 100% self-illumination, it shouldn't receive shadows at all, I think.
Why can't you apply a toon shader to it instead?

 

Ah, yes...perhaps I should explain. I set it to 100% illumination, yes, but then in the Diffuse map slot, I put a Gradient Ramp map which has its Gradient Type set to Illumination.
You can see the results of that in a picture like this:
[image=http://img.photobucket.com/albums/v293/rocketgrrl/Firefly.jpg]

Look at the buildings; they're all shaded in exactly this way...but if you notice, none of them are actually casting or receiving shadows, which is exactly the problem I'm having with these stairs.

Because it wouldn't work; I'd end up with exactly the same problem I'm having now. If you notice, the characters in those pictures above are shaded, yes...so are the stairs, but none of them are casting or receiving shadows themselves. And because the shaded part of the stairs are pointed away from the camera, you can't see them to be able to tell that they are separate objects.
For example, here is what the next shot in the sequence looks like:
[image=http://img.photobucket.com/albums/v293/rocketgrrl/Stairs03.jpg]

As you can see, the stairs are MUCH more distinct that way 'round because you can see the parts that are shaded.

Also, I decided a long time ago (in a galaxy far, far away) that I would try to make the characters distinct from the background, sort of in the old-school cel style, which is part of why I'm using my own shading method versus the toon shader I used for the characters. Changing it now, when I have more than 10 minutes already finalized, would be deeply counter-productive, sez I.

 

I don't work in 3DS Max, but the solutions I see to the problem...

-Could you outline the shape of the stairs? Basically like traditional animation's black outlines.
-Change the color of each step. You'd basically get a gradient/gray rainbow, but they'd be distinguishable.
-Redo the texture of the steps so they have some sort of pattern, or so that the texture contains the shadow.
-If the steps had a shadow of an object... a pillar or something... cast onto them, that would show the shadow being distorted as it lands on the separate steps.

Not sure if any of this is possible, just thoughts I had.

-Vaportrail

 

Nope. As I said earlier, that's reserved solely for characters; these steps are background.

Did that...see above.

Tried that, didn't work.

There's no object in the scene, though I suppose I could make one...hmm...maybe a special light that doesn't illuminate the characters but forces them to cast shadows in a way that the light representing the sun doesn't...that MIGHT work.

Well, that last one has possibilities and the gradient one I already tried (I'm wondering how you missed the example, but no worries)...thanks!

 

-Redo the texture of the steps so they have some sort of pattern, or so that the texture contains the shadow.

This. If it were me, I'd just "paint" the stairs different colors (each step containing half the proper color and its shadow color). It might be easier to do that than fight with lights.


EDIT: When you say you tried this but didn't work, was it "didn't look right" or "didn't render correctly"?

You might wanna just go find some steps that are angled how you want them and try taking some reference pics, if you've not done so already.



Rick McCallum loves you!

 

I think you're looking at something else that I said I'd tried and it didn't work. I DID try it and it DID work...I even posted a picture of it, which VaporTrail managed to miss too for some reason, so here it is again:

[image=http://img.photobucket.com/albums/v293/rocketgrrl/Stairs02.jpg]

Now, I think it's kinna ugly, but it gets the job done.

 

I think the reason I'm not reading that properly is because I'm seeing it as a somewhat continuous gradient flowing down the steps rather than shadows 'cut' by the forms of the steps.

I couldn't find a _great_ image, but look at this:
http://www.uoregon.edu/~j15/Pics/BoundPics/AltStrBound.JPG
specifically the steps at the left... there's a dark and clearly defined line of where the shadows begin and end. Hell, just try it out in Photoshop on your still before going into 3DSMax.



Rick McCallum loves you!

 

Yeah, but the problem is that there's no way for me to actually cast that shadow without some serious cheating.
I mean, if the characters don't cast a shadow, but something else does, that's gonna look pretty weird, don't you think? And the characters are moving, so a simple texture map won't do the job on them. And there genuinely is no object there to cast that shadow anyway. So...

 

I'm still inclined to just say "fake it" on the shadow-out-of-nowhere on that shot and see how it looks.

Here's some more production-design-oriented suggestions:

(1) Instead of trying to make an object cast a shadow over the stairs (for the distorted shadow effect), what if you put a red carpet or something that went up the stairs, we'd see it in sections on the steps. Maybe flank it with little yellow lights like at the movie theater. (Or maybe JUST the yellow lights, no red carpet!)

(2) Maybe you could mount little red safety lights on the "lips" of stairs to delineate the lines... or install lights in the risers. Anything from small red glints to big panels of light like in Cloud City.



Rick McCallum loves you!

 

I think the simplest solution would be a simple repeating pattern on the stairs, like it was tiled. The gradient isn't apparent enough to work; unless you plan on making it much harsher I don't think that'd the way to do it. It would certainly be buggered on something like YouTube.

 

Okay, I think I found a way to cheat it with textures:
[image=http://img.photobucket.com/albums/v293/rocketgrrl/Stairs04.jpg]

It's not perfect--you can see that there is no corresponding shadow on Van's boot, there--but you can see the divisions in the stairs now without a gradient pattern. It's actually more distinct on my home computer here; for some reason Photobucket keeps resizing my pictures...

It was actually a fairly easy fix in the end, though I wouldn't mind feedback on this, just in case.

 

the problem now is that the far end of it (on the character's left) looks like a slope instead of a staircase, ruining the whole thing.

 

Turn the self Illumination down to about 75% just to see what happens.

 

No, no...it's supposed to look like that. These stairs form a kind of angled wedge shape, so shadows would look like that even if they were real...

 

I'd say this works for me.


Rick McCallum loves you!

 

Yeah, me too. Thanks, guys...I mostly went my own way with this, but some of you influenced me in the right direction. I think I just needed to talk it out to find the solution...you know how that is?

 

Pretty much works for me too...they look believable.

Is that Aeon Flux there on the left?

EDIT: ya...so i jumped to the latest image first....bah

 

Is that Aeon Flux there on the left?

If only there was some introductory, or first, post in the thread that touched on this point.

(C'mon people!)



Rick McCallum loves you!

 

i'd like to know how Serenity fits into this fanfilm?

 

Cameo.

 

gah! stalker!!!

 

I jumped to the edited a few posts down version...b/c I know RG had already made changes

Lookin' good....have a release date yet?