"But how can we make sure that we can incorporate all those things that we will only discover well into the future if we don't design the system with an architecture that takes all those things into account?"
You can't. But neither can you (or anybody else for that matter) look into the future with certainty.
We all learn only from things in the past. Even if we think about the future and ‘learn’ from those thought, the process of thinking is in the past, when we learn.
We just can not predict the future.
But we can learn from the past.
We can learn to build systems in such a way, that they are comparatively easy to adapt to new situations and requirements.
That’s what patterns are for
Software Design Patterns are proven approaches to common problems that have worked in the past. That is how the GoF-Book, which made the whole pattern movement available to a broader audience, states the idea of patterns. You've got a problem you want to solve? Perhaps there already is a proven approach. Look it up. Chances are that the approach encompasses sound design principles and makes later changes attainable.
Software is more easily changed and extended if it is well structured. Refactoring, as it has been defined originally, ensures that the system stays well structured and easily modifiable. Use it. Follow ‘red – green – refactor’ if you have the chance or at least ‘refactor mercilessly’
That's what SOLID is for
The SOLID design principles (SRP, OCP, LSP, ISP, DIP) provide guidance on how to build actually change-friendly and robust software.
That's what Clean Code is for
Robert C. Martin also inspired the whole clean code movement which is not only about writing good code, but also provides insights on how to keep your whole development ecosystem in such a way that change stays easy.
You don't need to think about the details of iteration 35 in iteration 1 – if you ‘just’ factor in a small portion of the effort that up-front planning for iteration 35 would have taken for refactoring and restructuring in each iteration and follow sound software design principles you should be fine. After all iteration 35 may turn out to be something completely different from what you think now that it will be.
till next time