"People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual"
There is an interesting article in the WSJ today on the sloppy analysis of most science findings (Hat tip: Sissy Willis):
Scientists messing around with data or who have an agenda? Say it isn't so.
We all make mistakes and, if you believe medical scholar John Ioannidis, scientists make more than their fair share. By his calculations, most published research findings are wrong....
These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. "There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," Dr. Ioannidis said. "A new claim about a research finding is more likely to be false than true."
The hotter the field of research the more likely its published findings should be viewed skeptically, he determined.
Statistically speaking, science suffers from an excess of significance. Overeager researchers often tinker too much with the statistical variables of their analysis to coax any meaningful insight from their data sets. "People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual," Dr. Ioannidis said.
In the U. S., research is a $55-billion-a-year enterprise that stakes its credibility on the reliability of evidence and the work of Dr. Ioannidis strikes a raw nerve. In fact, his 2005 essay "Why Most Published Research Findings Are False" remains the most downloaded technical paper that the journal PLoS Medicine has ever published.
Scientists messing around with data or who have an agenda? Say it isn't so.
posted by Helen at 6:07 AM
25 Comments:
These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis.
Self-serving data analysis is not equal to fraud how?
This has been rampant in genetic investigations (on the taxonomic levels) for years and many published papers have promoted author's assumptions (not actual logical conclusions). They do this by picking an algorithm that gives them the results they want: you get to move some biota to some other location on the tree and your name is attached 'forever'.
I find this especially true in the area of the parsimonious relations cult. Yeah, I know I'm dipping into something only a scant few care about, but what hey, it fits the topic.
In many instances, I think that "flawed findings" can be traced back to confusion over what is "statistically significant" and what is "meaningful."
For example, in a paper I've been working on, I found a statistically significant improvement in prediction by adding a variable. But it only improved the prediction by 1.1% which, in reality, doesn't mean a whole lot.
What gets a little notation in SPSS may not mean a whole lot elsewhere.
From Goomp:
"Many people take the word of scienitists as if they were hearing the word of God, and it is good to see a scientist like Dr. Helen debunking that idea."
-- SW
derek...yes. Most people (including probably almost all journalists) misunderstand the notion of statistical significance. If it can be shown that "blondes have more fun" with 99% confidence, that does *not* mean that blondes have 99% more fun or even that blondes have a lot more fun. It only means that it is 99% probably that there is not a *complete* lack of relationship between blondness and fun.
This makes sense, often studies seem to contradict other ones. It's hard to know what is true.
This is all exacerbated by the increased use of computer models and the tremendously idiotic idea that the results of these models are, in and of themselves, scientifically significant.
Doesn't surprise me. Years ago a co-worker taking masters level social work classes happiily exclaimed how the day before a professor had told the class how to massage statistics to get what they wanted. Misusing statistics is part of the curriculum.
Working in marketing research, accuracy of our results is paramount even if the results are not what the customer expect to see. We've had customers run follow-up studies to verfiy results of a first study because they didn't believe the first one.
In academia, accuracy is seconday, getting the grant money is more important. And, you often get that by following someone's agenda.
When I was in medical school in the early 90's I spent a month working with Dr. Ioannidis. He was a very bright, honest, and very nice guy. He was also rather softspoken so I am surprised he is raising this much "muck", but good for him.
Some do research for a living, or to increase their income. Find something new! Get published! Best way to get another grant. If the research results need to be proved right or wrong, it may be ammo for ANOTHER grant to another researcher.
Joe, your sneering is misplaced (or at least overly broad). Computer models/simulations can be science. In fact, some computer models have been shown to be as accurate (ie, as accurately predictive of future results) as analogous experiments. Think tank gunnery.
Of course, there's a caveat. To build a meaningful model, first you've got to understand the physics of your system; you must be able to write down equations that describe the system fully. Then you've got to solve those equations, while using only physically justifiable simplifications. It's a tricky business, and it is very, very hard work. But for many physical systems it can be done.
The results are all around you. Ballistics algorithms are used for weapons targeting and space travel. Drugs are designed and optimized with molecular simulations. Buildings and complex machines are "built" and tested inside computer programs. Computers aren't just for word processing and web browsing; whole fields of science and engineering have been revolutionized by these machines.
All that said, I think I know where you're going. Climate modelers, right? The trouble is that those guys don't meet either of the conditions laid out above. So while they're doing science of a sort (developing theories is science, even in the early stages when the theories are woefully incomplete), their results do not yet seem to be physically meaningful. And it is sadly true that the incompleteness of their theories/models lends itself quite handily to the kind of malfeasance described in Dr. Helen's link.
As someone who deals with bankers and real estate finance almost every day, I am here to tell you that this does not only apply to "science". I don't know how many times I have had loans turned down because my conservative estimates don't meet the forecast profitability levels that all of the over optimistic liars in the business come up with.
The solution is simple, you go back to the bank the next day, say you have found a mistake in your data, that your rental income will be X times 1.2 instead of X, and give them a computer model to prove it. Presto, approval at the next loan committee.
This whole line of thought, and it's derivatives, is what got us into the current real estate recession (which I believe will expand to other sectors and hit the economy a lot harder than anyone now dares say), wave of forclosures, and failure of the credit markets.
It'd be little too complex to explain the math here, so let's just say that everyone on Wall Street, as well as all of the credit rating agencies had agreed, up until about three months ago, that if you put enough shit in a bag, the next morning you would open it and find that alchemy had changed all those stinky turds into gold bars lightly sprayed with the finest perfume.
It wasn't even a theory, it was a "fact", proven by complex computer models. To suggest that the models were flawed was blasphemy...
I can't resist adding just a little math to explain more clearly.
For every 100 turds in the bag, you would actually get only 80 perfumed gold bars. At first, you had to leave the turds with the gold bars to guarantee that they would stay gold. But at the end, as the optimism spiraled out of control, you could take the 20 turds out, put them in a new bag, and in the morning 16 (80%) would have turned to gold. Then you could get a new bag, put your remaining 4 turds in, slice each into 1/5's, creating 20 mini-turds, and in the morning you would have 16 (again 80%) mini-gold bars and 4 mini turds. The next morning you do it again, ad infinitum.
It should now be obvious to most readers that the limit of this function is approaching all gold and no turds; an amazing demonstration of the alchemy of financial engineering.
But then, in June 2007, people began discovering that someone had just painted the turds gold every night; and the shit hit the fan...
brian --
No. There is a science of computer modelling, but computer models are never science. They are tools only.
You've actually said that in your example. The model is only as functional as the physics (math) it uses. Ballistics / architecture / molecular studies is the science, the algorithms (models) are only the tools. The models are useful because we previously understood the science behind them.
There's the rub. A number of models do not utilize previously understood knowledge, such as:
Sociology - where the hell to begin for human presumption here (definitions for lib/con maybe)?
Genetic parsimony - the modelers may adjust variables as to how to group markers in some and oft time write their own algorithms.
Climate - the modelers don't consider some (solar) conditions or don't adjust their measurements (many readers have had their environ change from rural to urban) or ignore conflicts (atmospheric measurements vs ground).
Point is, if a model is build on an assumption, it's making assumptions, not giving hard answers.
Mark Twain:
There are lies, damned lies and then there are statistics.
olig:
I am relieved to hear that theorists do is never science. I will pass the word. The late John Pople will be disappointed, however, as they will likely have to revoke his Nobel Prize. I guess someone will have to take back my degrees, too, but that seems like a trifling matter by comparison ;-)
In seriousness, I'm not sure the distinction you're making between "science" and "tools" holds up. What is popularly called a "computer model" is in actuality a numerical implementation of a scientific theory. Developing and testing theories is what science is all about; without theory, the experimentalist is just gathering useless facts. And he way to test a theory is to use it to make predictions, then test to see if the predictions bear out. Computer models come into play because modern scientific theories are too complicated to make predictions from them by hand.
After a while we start to get enough confidence in a theory (say, coupled-cluster theory for the electronic structure of molecules) that there's no need to confirm any individual result. From then on, we can use theoretical results just as we would use experimental results. In my own field, there's really no functional difference between computing some molecular structure with a very high-level quantum chemical method and measuring it with a spectroscopic method. And if there's no functional differnce, then there's really no basis to say that the spectroscopist is a scientist and the computational chemist is not.
Unless, of course, you define science as purely about getting in a lab and learning some facts about nature, in which case you're free to call all this theoretical development/implementation/testing/use "not science" if you want. But you should understand that this is not a definition that is shared by all (or even most) working scientists.
Back to the task at hand: the revelation that a lot of results may be incorrect doesn't surprise me. In any good research group, it's a common refrain that "there is a lot of crap out there." (And, per my threadjacking above, this is NOT in any way limited to theory; the experimental literature is no better and often worse.) It's frustrating; you go looking for references on a problem and find one that seems directly on point, only to read more deeply and discover that the paper is a useless rag.
That said, this is more of an inconvenience than a big disaster. Scientific conclusions aren't reported as naked, context-fre bullet points. They're published in long papers that fully describe the methodology used. Any scientist qualified to make use of a literature paper is (or should be) qualified to assess its quality. If bad scientists are publishing useless papers, it is annoying, since I have to waste time reading the paper to find out that it's crummy. But it doesn't effect my work, really, because I then just take the bad paper and throw it in the trash. Sure, the publisher's out the money to print the paper and I'm out ten minutes or so. But the main cost is borne by the guy producing the useless science. And his time and energy are most likely no big loss.
brian --
That was humorous, but not what I'm talking about. I also think we're just talking past each other. You said it yourself, it's an "implementation".
I was taking an admittedly pedantic view of Computer models/simulations can be science, and saying they can be used in science but aren't science themselves. Pedantic, unless you're talking about the science of computer modelling.
A theorist using a computer model is using a tool. Remington using a ballistics model is using a tool. An architect using a computer model is using a tool. The spectroscope and chemical model are tools.
This is not a scientific blog frequented by scientists. "What is popularly" done as concerns the basic internet blog readership is to take answers generated by computer models and wonder if they are correct or not. Unfortunately, your average reader has no way to verify, and neither do your fellow scientists if you don't release the algorithms. Without undue hardship, anyway.
I'm just taking mild exception (like I said, pedantic) to the view the model itself is science. It's just a tool and the programming behind it can be wrong and unless you're functioning in a specialized niche such as those you mentioned, with tools that have been previously verified, you should always view the answers from models with suspicion.
Hence.
I have ranted many times over the years about the steadily deteriorating quality of research, but what happened to replication? When I was in grad school (yeah, I know, that sounds like walking five miles in the snow barefoot) studies were replicated. Researchers expected their studies to be replicated. Replications were one thing PhD students were for. Replication is quality control. If the study is valid, another researcher should be able to reproduce the results. But these days, we have researchers who not only don't want their studies replicated, but refuse to release their data.
How bad does research have to get before somebody at the university decides to actually do something about it and stop hiring hack researchers?
olig:
OK. I agree that the question blog readers/commenters have is usually about whether a result is reliable. When answering this question, though, you've got tp actually think about the system in question, not just sneer at entire well-accepted branches of science. Too many commenters like Joe, above, have got the idea that the predictions of computer models can never be trusted and sneer at ALL computational science as fraudulent. I thought that, most of y'all being non-scientists, you could use a little advice on how to evaluate simulation results that ran a little deeper than simulations are tremendously idiotic.
Bad simulations science is bad because the people doing it are incompetent, not because simulations are, as Joe suggests, inherently stupid. And you can do bad science in a lab filled with dangerous chemicals and expensive instruments, too. That's all.
Do we now have outcome based research, also?
If it can't be replicated, as the good professor suggests, who can (who would) trust the results?
Reverse engineering of truth or fact, in an effort to produce a desired effect, is still but a lie.
I have been stuck out west on a project from hell for the last couple weeks, frighteningly close to Rio Linda, for those who know what that means. It appears that out here, anyway, talking heads are replacing the buzz words of "global warming" with a heavily accented "climate change" instead (a verbal version of utilizing the index and middle finger of both hands, as in an Austin Powers movie) so those from Rio Linda will know to lose the old buzz words and embrace the new.
Once this transition is completed, I suppose we will learn where they are going with that.
Well, the problem with "climate change" is that once it's adopted, the theory has been taken entirely out of the realm of science. A theory must predict specific results. "Climate change" allows any results at all to be predicted. That isn't science; it's hokum, particularly with an inherently unstable system, like climate.
brian --
Y'all might just watch that y'all of yern and the 'non-scientists' as well. I gots my degree and I dones work in da field o' biology. Published 'n evrthin'. Writ me computer models too. Found me more money there, 'at's why I done 'puter stuff fer thirty-five plus. Dat's why I knows they's tools and som'times performs no better than a beats up ol' spect'ometer. 'At was my moniker you puts at top o' yer response, not Joe, so's you can see why's I tooks me exception. An' my point about assumptions? Stands. Les't ya takes the assumptive output an' gets some part of reality to jibe, iss jes an assumption. Kinda like that ther twine theory stuff. That there Gaussian? Ya might note it's a set of algorithms which'r highly optimized to calculate properties o' molecules. It helps develop by workin' as a tool to do thin's cain't be done in a life time wit' paper and quill. He made stuff what been around sisty yrs really useful, but face it, them algo's don't work on no hypothesis or make no predictions on they's own. Not applied to other work as a tool, they's jes purty lines onna printout. In fact, they's don't do squat on they's own, but gotta be buried in some gooey stuff to work worth a damn.
Cyber Hillbillies. What next?
視訊做愛視訊美女無碼A片情色影劇kyo成人動漫tt1069同志交友網ut同志交友網微風成人論壇6k聊天室日本 avdvd 介紹免費觀賞UT視訊美女交友..........................
383視訊影音城成人影城成人影城aa-dvdaa-dvdlove 免費視訊美女影音觀賞love 免費視訊美女影音觀賞tvnet0204 我愛你視訊美女拳tvnet0204 我愛你視訊美女拳5 分鐘護半身視訊美女視訊交友愛戀之5 分鐘護半身視訊美女99 視訊美女主播 500 點99 視訊美女主播 500 點msn 視訊美女msn 視訊美女888 視訊美女聊天室888 視訊美女聊天室視訊美女 live0204視訊美女 live0204173liveshow 視訊美女
Post a Comment
<< Home