Due to Adobe’s decision to embed smart object source data inside the parent file, things can start to get sluggish with multiple smart objects in the same file.
Transformation options are more limited (i.e. no perspective)
Re-rendering of all smart filters happens on some operations (e.g. Canvas resize) even if the layer visibility is off. Normally fast operations become extremely slow because of this.

What I’d love to have is an alternative to smart objects where layers are just normal layers but have rendering metadata attached to them, so if you want to re-render from scratch for some reason (e.g. tweaking free transform of layer too much over time, wanting to change shadow/highlight setting done earlier) you can. More or less, this would be a special action or script that is associated with a specific layer, just as layer masks and vector masks are attached channels and paths, respectively. Or alternatively it could be thought of as a history list for operations done on a particular layer.
I’ve been working on adding this functionality, with some success, using vector masks, layer names, and text layers to store various types of rendering metadata. With it I can re-render layers from scratch, loading external or internally stored originals and reapplying filter and free transform instructions to them. In my case, having a lazy re-rendering option in addition to smart-objects is extremely helpful. I’d love to see such functionality built into Photoshop at some point.