There appears to be a hierarchy of evidence for a scientific thesis: analysis, simulation, and experimentation.
- Analysis usually comes in the form of
theoretical, mathematical-style modelling and proofs.
The academics that I'm around seem to consider analysis
to be an extremely-strong form of evidence; i.e., if you
can prove it analytically, then it's gotta be true.
Analysis lets you make those nice, smooth graphs.
- Simulation seems to be running
programs
which implement an analytical models which transform
the system state as the model predicts it would through
time. Simulation appears to be useful, but it's not always
highly respected. For example, this crowd distrusts
the popular ns-2
network simulator for many types of network experiments,
because it predicts behaviors which don't happen.
- Experimentation is everybody's favorite technique for giving evidence, and it's particularly popular here for demonstrating properties of networks. Experimentation lets you test something new (e.g., a proposed active queue-management algorithm for routers) or something existing (e.g., the end-to-end behavior of traffic through Internet paths). Empirical graphs are jagged and bumpy, with spikes and dips caused by complicated and hidden real-world complexities.
Analysis is considered extremely strong because it uses formal proof techniques, but I don't really find it convincing for two reasons: (a) mathematical/theoretical-types always seem to oversimplify their model in favor of elegance and beauty instead of reality; (b) it's too easy to convince a human of something that it wants to believe. For example, if you've ever written a nontrivial program, you probably believed that it worked long before experimentation would support the theory that it actually worked.
Proofs are `strong' only because everyone wants to believe that they're strong. But there are numerous cases of published proofs standing for years, then being shown to be flawed. (E.g., some of the fundamental theorems of Real-Time Systems theory were published by Liu and Layland in 1973, but some of the proofs in it were shown to be flawed in the late 1990's.)
Donald Knuth's famous quote hilights the tension between analysis and reality:
Be careful about using the following code -- I've only proven that it works, I haven't tested it.
So it appears that the chief purpose of analysis is to simplify the life of the would-be scientist.