blueollie

No, Nate Silver was NOT wrong about 2016 and forecasting the 2018 midterms

By the way, for what it is worth, I voted early in the 2016 midterms and participated in a GOTV for swing districts program. This program requires you to print out some form letters, add a note and mail letters; it takes about 15 minutes for every 5 voters and costs you 1 envelope and 1 stamp per voter.

This HAS been shown to make difference in terms of voter turn out; the good news is that you don’t have to plug a particular candidate; just urge them to VOTE.

And yes, I vote for those who I do not back enthusiastically. My state representative and US representative are “ok, whatever” level of support. And our gubernatorial race is a real eye roll fest; two very wealthy, highly unqualified men are facing each other. For me, it was “take an antacid and vote blue” moment.

And yes, it is always a bit discouraging to be a Democrat. You have to put up with lazy people who don’t show up to vote, the issue of “hey, they aren’t letting convicted felons vote” (we are the party of convicted felons?). chronic underachievers, clueless, sanctimonious social justice warriors, etc.

And we have idiots that don’t understand what it means to control the House or Senate: the majority leader or speaker gets to set the calendar, decide WHAT gets to the floor to be voted on, etc. So if it is the choice between a conservative Democrat who will disagree with me 50-60 percent of the time or a Republican, I’ll take the Democrat every time.

In short: this. I love the title of the article too.

But I digress. Now to the main point of this post.

I enjoy reading forecasts of the various races. But when I see people critiquing Nate Silver (fivethirtyeight.com) I often hear “hey Nate, you were wrong about 2016”. Uh, he wasn’t. Here is an old summary of the various models:

Upshot gave Trump a 16 percent chance of winning. Silver gave him a 34 percent chance. Let that sink in. (now if you want to bash Sam Wang, go ahead..now HE was wrong (Princeton)).

To put in in perspective: Upshot (NY times) prediction can be thought of in this way: Clinton’s probability were roughly the same probability that a good NBA free throw shooter makes a free throw (and yes, the DO miss from time to time). Silver’s model put Clinton’s chances at roughly the same as a BAD NBA free throw shooter making the free throw…yes, they make them most of the time, but yes, they miss…a LOT.

In terms of point spreads, Upshot’s prediction was like saying that Clinton was the favorite in the election, to the degree that a college football team was a 14 point favorite. Silver would have made her roughly like a 7 point favorite (and yes, 7 point favorites often lose the game).

(source of football odds)

Yes, I know; point spreads and probability of winning are two different things, but there is a good correlation between the point spread and the probability of victory.

And so we turn to the forecasts. You can read the House forecast here.

Here is the best available model (the “lite” model gives a very similar output)

What does this mean?

1. The Democrats ARE favored to retake the House (about 80 percent) but this is akin to one team being a 10-12 point favorite in football, or a strong NBA free shooter making a selected free shot. Yes, they DO miss from time to time, and teams that are favored by 10-12 points DO lose from time to time.

2. The range of plausible outcomes (80 percent confidence interval) is very broad; they include a 50+ seat landslide for the Democrats to ..yes, the Republicans retaining control. Neither “extreme within likeliness” is the most likely outcome, of course, but the bell is rather flat; there is quite a bit of possible variation in the outcome. The expected outcome is about a 38 seat pick up, but again, there is a ton of variation this year.

I WISH it were hopeless for the Republicans, but it isn’t. But in the House, I’d rather be where we are than where they are.

Here Silver attempts to explain it. I think that he does a good job but..well, some (many? most?) just can’t (won’t?) get statistical reasoning or understand its limitations.

3. Senate: tough map for us. Republicans have about an 80 percent chance of keeping control; it is almost the dual to the House situation.

Ah, it has been a long time since I wrote one of these posts. ๐Ÿ™‚

Off to run in some beautiful weather. I had thought about a 4 mile race, but I am just not in a mood to push myself.

November 3, 2018 Posted by | 2016, social/political, statistics | , | Leave a comment

Bad arguments drive me crazy…

One thing that causes me persistent heartburn are those statistically invalid arguments, often made by those who either do know better or should know better.

It is one thing to, say, pass a statistics class, or even properly use statistics to analyze data. It is a different thing to make statistical thinking as part of one’s every day thought process.

Here are some examples:

1. “X is 60 years old and ran a 3:10 marathon. So what is your excuse?”
The fallacy: X is almost certainly an outlier. Example: a 61 year old lady ran a 3:12 at this year’s Chicago marathon. It was Joan Benoit, who has an Olympic gold medal and has a PR of 2:22.
My response: “Y is your age and ran a 2:05. What is your excuse?”

2. “It is “racist” or “classist” to use the ACT for college admissions (or placement). The HS GPA is a better predictor.
The fallacy: just because one measure *might* be a better predictor doesn’t mean that several measures, taken together, aren’t better predictors than any single measure taken alone. After all, NFL teams measure several things (speed, strength, speed through a cone course, etc.)

3. “X achieved great things and there were naysayers saying that they couldn’t.”
Fallacy: most of the time, the naysayers are right. Most do not achieve elite results, by definition. Related: “hey they laughed at Einstein” (well, not really, he got 4 papers published in a top notch physics journal while still a graduate student that no one ever heard of..I don’t think that is getting “laughed at” but whatever.

4. “The deficit/debt is X dollars..that is more than it has ever been, etc.” Duh. We are a growing country. That is why debt and deficits are measured in terms of GDP. Newsflash: a million dollar deficit might be a lot for a small town, but it is nothing for an entire country, due to the relative size of the economies.

5. “Polls..ha, ha, ha, ha…they predicted Hillary would surely win!” Uh, no. Yes, a couple of models based on polls had her as a heavy favorite. BUT, not all of them, and these were MODELS. Nate Silver game Trump a 34 percent chance (roughly the probability of a below average NBA player missing a free shot). And the predictions were pretty spot on in terms of national vote; they were all just a bit off in a few key states..and off in the same direction.

My own “lower confidence interval” had Trump with 266 EV, 4 short of what he would have needed.

6. “Hey, these athletes get paid a huge amount of money to play a game whereas people who do more important work are paid far less..”
Fallacy: entertainers (athletes, actors, musicians) tend to be either “strike it HUGE” or “amateur” with not much inbetween. Put another way: being 95’th percentile in math probably means that you can earn a nice living as an engineer. Being 95’th percentile at sports means..well, maybe getting playing time at a D-3 program..if you are lucky. Only the most outrageous outliers make it as a professional athlete.

October 20, 2018 Posted by | social/political, statistics | | Leave a comment

Who are the minor league baseball fans anyway? It depends on how you measure…

I’ve discussed this principle before, albeit in a different context (welfare). But I am about to do so again.

I went to the last Peoria Chiefs regular season home game and a couple who has season tickets showed up (ironically: they rarely come). I asked if they were going to the playoff game and they said “no”; in particular they said that they don’t like crowds.

I remarked that Chiefs playoff games are rarely crowded and that surprised them.

This was after the game started but well before the rain delay:

Though the threat of thunderstorms retarded attendance this time around, attendance wasn’t stellar in previous years either.

Regular season games, especially Friday/weekend games are usually much better attended. So what is going on?

I think that it is this way: during a given regular season game, most of those you see there will NOT be a hard core baseball fan. You’ll have some casual fans (“oh, I feel like a game today”), families or groups of friends looking for an outing (“hey, pretty day..let’s go to the ball park!”) along with church groups and workplace groups, youth teams, youth groups, etc. They are there “for the outing” and many who show up care little about the actual game.

Most of these..the bulk of the fans at any given r game, will not show up to a playoff game.

This doesn’t mean that the Chiefs do not care about the baseball fan though. After all, if you count ALL of the tickets sold for ALL of the games put together, I’d venture that most of these tickets are purchased by the baseball fans; the “faithful 1000” that either have season tickets, flexible vouchers (like I do) or those who just go to a lot of games. That is, I’d guess that most of the tickets sold come from a relatively small number of fans that go to a lot of games.

And it is this group that will constitute most of the playoff game crowd, I think.

And a note: the Chiefs swept their first playoff series against the Quad Cities River Bandits! They now move on to the semi-finals and will host game 2 of the “best of 3” series this Sunday; IF neither team sweeps the best of 3, the Chiefs host game 3 on Monday at 6:30 pm. The opponent is yet to be determined (Beloit? Cedar Rapids?). I am looking forward to 1 or 2 more games…and maybe more than that? (finals are best of 5).

Workout notes: I am still coughing but am feeling much better. I sound worse…feel better.

The run: after weights: 3 miles in 31:50 (32:54 for 5k) 10:47, 8:43 (19:30 for 2 miles) then walk/jog/run the final mile to cool down. 2.05 for 20 minutes.

weights: rotator cuff, pull ups (15-15-10-10 felt good), bench: 10 x 70 dumbbells, incline: 6 x 150, decline: 7 x 170. military: 10 x 50, 10 x 45, 10 x 45 standing, 3 sets of 10 x 110 machine rows, plank, twist crunch, side plank, boat, headstand.

Tomorrow: long walk of some sort; marathon 2 weeks away. I have 20’s under my belt..several actually, but do I risk draining myself or do I go more moderately (say, 4 hours..that is it?)

September 7, 2018 Posted by | baseball, illness, running, social/political, statistics, weight training | | 1 Comment

Attitudes and triggering the liberals…

So Trump pardoned Dinesh Dzouza (who violated campaign finance law). This might be a message to those caught in Mueller’s cross hairs, and yet another way to “trigger the libs”. Every time we foam at the mouth, he wins.

Trump is the troll in chief (11 minute MSNBC video; well worth watching).

The upshot: their goal is to draw our scorn, thereby making their base happy with them. And getting scorn from us is easier than, say, passing a bill and signing it into law.

And speaking of scorn: there is scorn for Kaepernick and Barr. But the actions (respectful protest vs. racist tweet) are very different and have different standards of decency. This is a well written article about that, and it is written by a conservative!

And seriously, the NFL protests are really the best kind: peaceful, non destructive and non-intrusive. But yet, many whites are outraged. There really is no protest that they’d approve of.

Speaking of football: I read this quote by Vince Lombardi in a book:

I want to say that it is becoming increasingly difficult to be tolerant of a society who has sympathy only for the misfit, only for the maladjusted, only for the criminal, and only for the loser.

Then paraphrased stuff was added. Well, the longer quote (with more context) is this:

None of us is born equal, in spite of everything they say about it โ€“ we are born only in certain inalienable rights. But we are born rather unequal. However, I want to say that the talented are no more responsible for their birthright than the underprivileged are and the measurement of each man should be what each does and I want to say that it is becoming increasingly difficult to be tolerant of a society who has sympathy only for the misfit, only for the maladjusted, only for the criminal, and only for the loser. I think we should have sympathy for them, certainly. I think we should help them, certainly. But I think it is also the time in this country to cheer for, to stand up for, to slap on the back the doer, the achiever, a man who recognizes a problem and does something about it, the winner.

It sounds different, doesn’t it?

More Politics The 2016 polls were reasonably accurate, by historical standards. It is just that many models didn’t emphasize how small Clinton’s lead was in key states.

June 2, 2018 Posted by | football, politics, politics/social, poll, social/political, statistics | | Leave a comment

The “science” of sports…

First things first: I shuffled through my 8.1 mile course ..and really shuffled; it wasn’t fast, at all. But it had lots of hills and the weather was great.

I was thinking of bailing out and making it 5, but then I passed some older man (ok, my age..perhaps a few years younger) who was resting from riding a bike, and smoking. That ticked me off and the anger fueled me enough to do the second hill segment that made it an 8.1 mile run/shuffle.

Sometimes, contempt can motivate.

Yeah, I know..yadda, yadda…I should seek a more “aware” way..contempt is bad, blah, blah, blah. But it got me going.

And yes, this is coming up on the 20’th anniversary of my first Steamboat 15K:

And what a joke I’ve become. Ok, this shows a “middle aged man slow” morphing into an “older man slower”…which the extra slowness brought on my too many 100 mile finishes, an extra knee operation and a body that won’t tolerate as much training as before. And yes, some of newer half marathons were genuine walks, not runs. But the 5K results tell the story.

So, what is the right formula?

And that leads to my commentary on sports science. Yeah, evidently that is a place for sloppy statistics.

But I get it…sort of. There is just so much variation between people..even between outstanding athletes. I remember discussing Olympic swimming…and seeing medalists in middle distance swimming..one had a super high turnover rate..the other had a much lower, fewer strokes per length..and both blew away some excellent swimmers to dominate their events. Different bodies…different paths to success.

And training: what works for one person will burn out and/or injure another.

It is even worse among the non-elites..and especially for people like me.

So basically, I’ve given up and just try to be as specific as I can for the upcoming event that means the most to me. And I’ll try..but hey, 1023’rd place is a lot like 1047’th place, so why sweat the details?

May 22, 2018 Posted by | running, statistics | , | Leave a comment

Statistical Illiteracy

One of things I am most amused by are approval ratings polls. If you follow the social media of Trump haters (I do not approve of him as president), you’ll hear that his ratings are dropping..always dropping…

But Trump supporters will crow about his ratings…going up and you’ll even hear stuff like this from Trump himself:

Here is what they are linking to:

Yep, he popped above Obama …THIS POLL. Note how the TownHall screen shot managed to put that box right over the recent polls.

But what about those other polls?

Real Clear Politics polling average; (41.8 as of this writing)

FiveThirtyEight.com (40.6 approval as of this writing)

Trump vs. Obama in the Fivethirtyeight average

What is going on:

Even an average of polls shows some “fluctuation”; the polls go up and down with time, even when the trend is steady. This is due to randomness of sampling and perhaps some sample error. So, if in one poll, Trump is 38 in one poll and 40 in the next, his supporters say “Trump is gaining in the polls”. But then if he goes to 37 in the next one, Trump opponents cry “he is dropping like a rock!”

If you hang around Trump opponents, you hear only about the drops, and if you hang around supporters, he hear only about the gains.

Sam Wang got it right:

Reality: Trump is at about 41 percent approval (low for an economy in this shape) and there will be a few minor fluctuations in either direction that don’t mean much, if anything.

Workout notes; sore shoulder special swim: 250 free, 250 fins, 250 pull, 250 free, 250 free/back, 250 breast/free, 100 fins drill/swim, 100 free, 100 drill/swim, 100 swim, 100 pull, 100 swim 50 side, 50 swim. Just got it in; protected the shoulder.

treadmill run: 5 minute froggy to get to 44:50 for 4 miles, then walking (17 mpm, 16 mpm, 14 mpm, 13:30 ) to get to 59:40 for 5 miles. Foot did ok until the faster walking. Still ok.

April 3, 2018 Posted by | political/social, running, statistics, swimming | , , , , | Leave a comment

Confusing the individual with the aggregate

One of the things that fascinated me was radioactive decay. If you were given a certain amount of a radioactive isotope, you can deduce how much will be left (not decayed) after a certain amount of time. In fact, you can do this so accurately that you can base a precision clock on it.

However, it is impossible to determine WHICH atom will decay, no matter how much information you have about it. I don’t mean that it is practically impossible but rather that it is literally impossible. And the individual atoms will decay at different times.

In short, you have information about the aggregate but not about the individual. Of course, in this example, we are in the range of quantum phenomena.

But this principle, (aggregate vs. the individual) applies when one attempts to make inferences about what will happen with a population in which there is a high level of variance within the said population, and people often get confused.

Example: suppose you have two groups of students who are, say, starting a program of study in engineering. One group is the group of students whose math ACT scores are 22, and the other group has math ACT scores of 30. The harsh reality is that the group of students with a score of 22 will have very little success; there may well be a few individuals who make it, but the vast majority won’t. And yes, the group with a score of 30 will have some failures, but they will have many more successes.

So, the ACT score matters and has predictive value. But if you bring this up, someone will remember the person with a 30 who flunked out, and someone with a 22 who made it and claim that means that the “ACT is meaningless”. Psst: that isn’t true.

So yes, there are smokers who live a long time, there are those who drive while texting who don’t get into accidents, etc. But smoking does harm longevity and driving while texting increases one’s risk of having an accident.

Application to Illinois Football Illinois football is starting MANY true freshmen and, well, the record so far is grim (2 wins over weaker non-conference opposition, followed by 5 straight losses against “power 5” caliber opposition (USF isn’t “power 5” but they are an undefeated, ranked team). And prospects for another win this season are grim, with 2 Top 10 teams (Wisconsin, Ohio State) and 3 improved teams (Indiana, Purdue, Northwestern) left to play.

So the PR department is playing this “the future is bright” angle:

And yes, the team is playing a lot of freshmen.

But: how good is that class? I went on ESPN and looked at how the Big Ten 2017 recruiting classes were ranked:

Top 10: Ohio State, Michigan
10-25: Penn State, Maryland, Nebraska
26-39: Michigan State
40-49: Wisconsin, Iowa, Northwestern
50-59: Rutgers, Illinois, Indiana, Minnesota, Purdue.

So, based on talent, we *might* be able to hang with Rutgers, Indiana, Minnesota and Purdue, youth or no youth.

Now yes, measuring recruiting is tough to do, and there is always that individual “lightly regarded” recruit who blossoms into an NFL player. It does happen..individually. But a team composed of lightly regarded recruits is rarely, if ever, successful.

Workout notes: yesterday, wet 10K walk (untimed). today: weights. Pull ups were a struggle, so I did a couple of 5-5 sets then 2 sets of 10, one of 7-3 (50 total). (switched grip), usual PT, incline presses: 10 x 135, 4 x 160, 6 x 150, military (dumbbell: 10 x 50, 10 x 45) 10 x 180 machine (90 each arm), rows: 3 sets of 10 x 110. Then a chilly 5K walk outside.

October 25, 2017 Posted by | college football, education, football, science, statistics, walking, weight training | , , | Leave a comment

Statistical inference and the morning weight room

I know that this is far from perfect. But for a couple of years, the university had some smaller than average classes. And yes, the gym was more empty at 6 am.
Today: there were more people than usual in the gym at 6 am (start of classes). But that isn’t the only factor: our university is also tearing down buildings and replacing them with updated ones (yes, badly needed upgrades). That reduces the number of available classrooms, hence we have more afternoon/late afternoon classes than before.

So more students plus “being in class in the afternoon” means “more people in they gym” in the morning. Nevertheless, I got through the routine (weights only) in 42 minutes; then added 20 minutes of skips and legs then walked 4 outside.

Social media: it is interesting. In one case, somebody thought he was “calling me out” when, in fact, I was arguing about langue and not the concept. In another case, a Trump supporter refused to read anything from the mainstream media because…well, the election projections were wrong.

Note: the polls did pretty well with the national popular vote; even the state polls in the battle ground states were not that far off..it is just that several were off by a little bit IN THE SAME DIRECTION (which Nate Silver said was a real possibility). The polls weren’t bad, but some (not all) of the inferences from the polls were. But try explaining that to someone whose mind is already made up.

I’ve learned to say “ok, I’ll leave your company for others to enjoy”.

And yes, I’ve had to do that with people who vote the same way that I do. Statements like “no, Bernie Sanders would not have won” or “Hillary Clinton really isn’t that good of a campaigner; Bill Clinton and Barack Obama were a lot better” or “yes, the Russians did spread disinformation but there is no evidence that they hacked the voting machines themselves” have earned me both ire and blocks on Twitter.

No big loss though.

Workout notes:
hip hikes, toe raises, rotator cuff
pull ups: 5 sets of 10: ok.
incline: 10 x 135, 8 x 150, 4 x 160 (decent hip placement)
military: (standing, with dumbbells) 10 x 50, 10 x 45, 20 x 40
rows (Hammer) 3 sets of 10 x 200

The above took 42 minutes.

rope skips: 34, 50, 50 (last two sets: ended at 50 voluntarily). I am getting better.
goblet squats: 5 x 50 (window sill), 10 x 50, 10 x 53 (kettle), 5 x 70 (20 inch box)
4 mile walk in Bradley Park; kind of sluggish. Very good weather though; nice and cool.

August 23, 2017 Posted by | 2016, social/political, statistics, walking, weight training | | Leave a comment

A bit of statistics

Ok, how can we draw statistical inference when we cannot run a controlled experiments? After all, correlation and causation are not the same. This is a useful guide as to the how and when. Basically: is the correlation strong, and is there some “plausible reason” for such a correlation? This paper lists 7 points.

Simpson’s paradox You can see a discussion here.

Think of it this way: say 1000 women and 1000 men apply for admission to graduate school. 656 men get admitted, whereas only 260 women get admitted. Does this mean that things are biased against women?

But then we see that there are two very different graduate programs. The very selective graduate program admitted 8 percent of all male applicants but 10 percent of all women applicants. The other graduate program..the “easy to get into” program admitted 90 percent of female applicants and 80 percent of all male applicants. So: we see that the women outdid the men in both programs. Yet, we also see that 800 women applied to the “difficult to get into program” and only 200 men did. On the other hand, 800 men applied to the easy program but only 200 women did.

Check it out: women: 800*.1 =80 admits to the hard program, 200*.9 = 180 admits to the easy program, so 260 total admits. Men: 200*.08 = 16 admits to the hard program, 800*.8 = 640 admits to the easy program, or 656 total admits.

This isn’t just some “trick” either. When social scientists analysed the “stand your ground” defense law in Florida, they found that whites were more likely to be convicted than non-whites. BUT this was because whites were more likely to be accused of assaulting a white victim; it turns out that the probability of prosecution was higher if the victim was white than if the victim was non-white. You can see the details here.

workout notes: 4 mile walk after weights: rotator cuff, 5 sets of 10 pull ups, bench press: 10 x 135, 5 x 185 (strong), 10 x 170, incline: 10 x 135 (very easy), military: 10 x 50 standing, 20 x 50 seated supported, 10 x 200 machine, rows: 3 sets of 10 x 50 single arm. head stand, 2 sets each of 10 yoga leg lifts, 12 twist crunch.

November 29, 2016 Posted by | science, social/political, statistics, walking, weight training | Leave a comment

Pre-election Sunday….

Ok, the time for spinning is over and what do the numbers say? Here are the betting lines:

6novodds

They range from 3/10 to 1/5 for Clinton, with most at 1/4. This is a slight change from last night, but not much of a change.

Here is Upshot’s list of models:

upshotmodellist6nov

And here are several prediction maps (I’ve put the source in green lettering). This is the list (from most favorable to Trump to most favorable to Clinton)

Election Projection: 284
Fivethirtyeight (Nate Silver) 293
Electoral vote: 317
Benchmark Politics: 322
Predictionwise 323
Princeton (Sam Wang) 323
Upshot (New York Times) 326

I’ve put together the maps, and labeled the source in green.

Some notes: Benchmark uses more data than just polling (e. g. economic indicators, history) and Predictionwise factors in betting lines for each state. And of course, each model factors the various polls a bit differently (e. g., how does one weight older polls? What track record does that polling outfit have? Is it a “likely voter” model or a “registered voter” model?)

But if you notice, the projected Electoral College count doesn’t vary that much; much of the dispute is in the “confidence interval”. Nate Silver’s model has a wider confidence interval (which can vary from a narrow Trump win to a Clinton landslide) and Sam Wang’s has a narrower confidence interval; I talk about this a bit more here.

mapsuselection6november

November 6, 2016 Posted by | political/social, politics, politics/social, statistics | | 2 Comments