Booleans have always been a no-go for me. Every time I tried them, my modeling package crashed or gave back unusable meshes. After some time really trying, I couldn't get the results I was after, so I decided booleans weren't for me.
Many years passed, and I trained myself hard to model any shape I was after purely using traditional modeling, you know, the one where you plan your shapes really well before tackling them, and every change means redoing a lot of work.
Then I became a teacher. Teaching is an amazing experience because you can, and this might sound weird, plant a seed that will grow your knowledge on other people's heads. Not everyone grows knowledge the same way, but since your seed includes your workflows, those plants are all the same species, meaning your workflows have variants now that can teach you a lot.
A student in the school that I taught, Jesús, was one of those people that grow workflows like wild weeds, so fast it was hard to keep up with him. One day, while chatting about a tutorial by Eugene Petrov on modeling guns on Zbrush, he taught me his variant of Eugene's workflow.
It was then when I connected back with booleans and introduced them into my high poly workflow, which I'll cover in a separate post soon. This article will teach you what Jesús taught me.
.01 Preparing your Objects for Boolean Operations.
The first step in this method is to prepare your operands. My errors back in the day were to want booleans to be cheap, which meant low poly. Low poly operands can, most of the time, cause tons of shading issues, especially when used with curved objects.
Take a look at the following image:
On the left, a lower resolution mesh. The right shows a denser one. In red, the subtraction object. Let's see how the shading behaves in each of them when the boolean operation takes place.
Notice a difference? The lower poly operation doesn't hold its shading when cut by the red object. On the other hand, our higher-res does hold both a way better silhouette and shading. It's not perfect tho.
To make this a usable object, we need to fix the shading. I usually do this by adding containment loops that will hold the shading within the faces they create. The smaller the faces, the better shading we'll get. It all comes at a cost, of course, but that's always part of the process.
This neat little trick will only work when "No Edge Removal" is active.
This example is simple, but the rule is clear. Use denser geometries that will hold the shading better due to it not depending on larger areas to work. This will mean a bit more cleanup, but even that is still a significant time gain when compared with modeling the shapes by hand.
02. Using Boolean Instances.
When I saw this, I couldn't believe my eyes. When working with booleans, you can set the operands to instances, and they will update with the changes done to these instances in real-time. This is hard to explain with words, so an example will serve this purpose better.
Before starting, make sure your booleans are set to instances and start picking your objects. Once that's done, you'll be able to make changes to them in the sub-object level, like this:
This allows you to add modifiers to the boolean object and continue working with the ability to come back to make edits without wasting any work at all.
I take this a step further and keep my boolean source objects in a separate layer to ensure I have them there if my object breaks. All I have to do in that case is to apply the boolean operations to them.
Here's another example of the flexibility this technique allows with some minor cleanup.
I just wished we had a bit of a better chamfer...
03. Boolean Cleanup
When working with these objects, we have to be ready for some cleanup. The previous examples wouldn't be a great way to impress a client with your tidiness, so we need to do something about that.
I suggest you use key bindings like the ones I use. "Shift + Spacebar" is used to connect vertices, and "CTRL + Spacebar" to remove edges, and "CTRL + Shift + Spacebar" collapses my selection. This allows for a super-fast boolean cleanup.
As you can see, this cleanup is much quicker with shortcuts, plus it even becomes fun to see how fast you can do it!
Each boolean has a different cleanup, so I can't add a general example of how it should be done. Following the rules for game assets are key here, like not having faces with more than 4 sides, or avoiding small and long triangles.
The "Turn to Poly" modifier in max can help you a lot with the cleanup process as well. Just make sure you turn "Keep Polygons Convex" on. When it works, it's a huge time saver. In the example below, a bit of cleanup to remove the long thin triangles would still be required.
04. Low Poly Mesh.
Getting a Low Poly from these objects is as simple as reducing the boolean instances and any modifier that adds geometry to the object, like the chamfer in my case.
On top of that, "Turn to Poly" plus a bit of cleanup will get you a very decent low-resolution mesh.
I hope this short guide served you as a good way to understand how to safely work with booleans. If you have any questions, please let me know!
If you require a more in-depth guide, check out Ben Bolton's guide in Polycount, it's super detailed and gave me a couple of ideas to make mine a tad more complete.
I also covered this workflow on my Artstation Learning Series.
I wish I had the time to go into such detail for this post. In any case, have a great day, and explore these techniques a bit! You won't regret it.