Baby steps is one of the principles behind the eXtreme Programming and other agile methods. It is, namely, performing changes in small steps on every level, be it implementing a feature, changing the database structure or just coding a couple of methods. The ideas behind this principle correlate well with not programming by coincidence and usually baby steps are implemented with test-driven-development based techniques.
The advantages of applying baby steps come from the fact that the smaller the piece of work is, the easier it is to think it over, plan and verify. Baby steps allow for verifying the direction often and for changing the plan frequently. When applying baby steps every mistake is only a baby mistake that can be noticed early and corrected easily. For example, when transforming the application from a single binary into client-server, the baby steps can help to see very early if a chosen technique (e.g. COM plugins or AJAXy web clients) fits well into your situation and adapt before months of efforts are invested into the transformation.
The biggest disadvantage of the approach is in the need to actually verify the application behavior often, be it the code or user-perceivable feature level. For the team that does not use test-driven-development and does not have an extensive set of unit tests, verifying the functions can take a lot of time. If the customer does not collaborate with the team often, it is difficult to verify early that the feature is done.
The "frequent verification overhead" might be a real problem in some circumstances. However, whenever we manage to make it smaller, whenever we manage to make the unit testing set faster and more reliable, whenever we manage to get the customer evaluation more frequently, the speed and correctness of delivery raises quickly.
What step size do you use in your environment? Do you run the test sets every every 10 minutes, nightly or monthly? What is the verification overhead like in your environment?