six thousand miles of road, 600 miles of subway track, 400 miles of bike lanes and a half a mile of tram track, if you've ever been to roosevelt island.
3d printing conference new york, these are the numbers that make upthe infrastructure of new york city. these are the statisticsof our infrastructure. they're the kind of numbers you can findreleased in reports by city agencies. for example, the departmentof transportation will probably tell you
how many miles of road they maintain. the mta will boast how many milesof subway track there are. most city agencies give us statistics. this is from a report this year from the taxi and limousine commission, where we learn that there's about13,500 taxis here in new york city. pretty interesting, right? but did you ever think aboutwhere these numbers came from? because for these numbers to exist,someone at the city agency
had to stop and say, hmm, here's a numberthat somebody might want want to know. here's a numberthat our citizens want to know. so they go back to their raw data, they count, they add, they calculate, and then they put out reports, and those reportswill have numbers like this. the problem is, how do they knowall of our questions? we have lots of questions. in fact, in some ways there's literallyan infinite number of questions
that we can ask about our city. the agencies can never keep up. so the paradigm isn't exactly working,and i think our policymakers realize that, because in 2012, mayor bloombergsigned into law what he called the most ambitious and comprehensiveopen data legislation in the country. in a lot of ways, he's right. in the last two years,the city has released 1,000 datasets on our open data portal, and it's pretty awesome.
so you go and look at data like this, and instead of just countingthe number of cabs, we can start to ask different questions. so i had a question. when's rush hour in new york city? it can be pretty bothersome.when is rush hour exactly? and i thought to myself,these cabs aren't just numbers, these are gps recordersdriving around in our city streets recording each and every ride they take.
there's data there,and i looked at that data, and i made a plot of the average speed oftaxis in new york city throughout the day. you can see that from about midnightto around 5:18 in the morning, speed increases, and at that point,things turn around, and they get slower and slower and sloweruntil about 8:35 in the morning, when they end up at around11 and a half miles per hour. the average taxi is going 11 and a halfmiles per hour on our city streets, and it turns out it stays that way for the entire day.
(laughter) so i said to myself, i guessthere's no rush hour in new york city. there's just a rush day. makes sense. and this is importantfor a couple of reasons. if you're a transportation planner,this might be pretty interesting to know. but if you want to get somewhere quickly, you now know to set your alarm for4:45 in the morning and you're all set. new york, right? but there's a story behind this data.
this data wasn'tjust available, it turns out. it actually came from something calleda freedom of information law request, or a foil request. this is a form you can find on thetaxi and limousine commission website. in order to access this data,you need to go get this form, fill it out, and they will notify you, and a guy named chris whongdid exactly that. chris went down, and they told him, "just bring a brand new hard drivedown to our office,
leave it here for five hours,we'll copy the data and you take it back." and that's where this data came from. now, chris is the kind of guywho wants to make the data public, and so it ended up online for all to use,and that's where this graph came from. and the fact that it exists is amazing.these gps recorders -- really cool. but the fact that we have citizenswalking around with hard drives picking up data from city agenciesto make it public -- it was already kind of public,you could get to it, but it was "public," it wasn't public.
and we can do better than that as a city. we don't need our citizenswalking around with hard drives. now, not every datasetis behind a foil request. here is a map i made with the mostdangerous intersections in new york city based on cyclist accidents. so the red areas are more dangerous. and what it shows is firstthe east side of manhattan, especially in the lower area of manhattan,has more cyclist accidents. that might make sense
because there are more cyclistscoming off the bridges there. but there's other hotspots worth studying. there's williamsburg.there's roosevelt avenue in queens. and this is exactly the kind of datawe need for vision zero. this is exactly what we're looking for. but there's a storybehind this data as well. this data didn't just appear. how many of you guys know this logo? yeah, i see some shakes.
have you ever tried to copyand paste data out of a pdf and make sense of it? i see more shakes. more of you tried copying and pastingthan knew the logo. i like that. so what happened is, the datathat you just saw was actually on a pdf. in fact, hundreds and hundredsand hundreds of pages of pdf put out by our very own nypd, and in order to access it,you would either have to copy and paste for hundreds and hundreds of hours,
or you could be john krauss. john krauss was like, i'm not going to copy and paste this data.i'm going to write a program. it's called the nypd crash data band-aid, and it goes to the nypd's websiteand it would download pdfs. every day it would search;if it found a pdf, it would download it and then it would runsome pdf-scraping program, and out would come the text, and it would go on the internet,and then people could make maps like that.
and the fact that the data's here,the fact that we have access to it -- every accident, by the way,is a row in this table. you can imagine how many pdfs that is. the fact that wehave access to that is great, but let's not release it in pdf form, because then we're having our citizenswrite pdf scrapers. it's not the best useof our citizens' time, and we as a city can do better than that. now, the good news is thatthe de blasio administration
actually recently released this dataa few months ago, and so now we canactually have access to it, but there's a lot of datastill entombed in pdf. for example, our crime datais still only available in pdf. and not just our crime data,our own city budget. our city budget is only readableright now in pdf form. and it's not just usthat can't analyze it -- our own legislatorswho vote for the budget also only get it in pdf.
so our legislators cannot analyzethe budget that they are voting for. and i think as a city we can doa little better than that as well. now, there's a lot of datathat's not hidden in pdfs. this is an example of a map i made, and this is the dirtiest waterwaysin new york city. now, how do i measure dirty? well, it's kind of a little weird, but i looked at the levelof fecal coliform, which is a measurement of fecal matterin each of our waterways.
the larger the circle,the dirtier the water, so the large circles are dirty water,the small circles are cleaner. what you see is inland waterways. this is all data that was sampledby the city over the last five years. and inland waterways are,in general, dirtier. that makes sense, right? and the bigger circles are dirty.and i learned a few things from this. number one: never swim in anythingthat ends in "creek" or "canal." but number two: i also foundthe dirtiest waterway in new york city,
by this measure, one measure. in coney island creek, which is notthe coney island you swim in, luckily. it's on the other side. but coney island creek, 94 percentof samples taken over the last five years have had fecal levels so high that it would be against state lawto swim in the water. and this is not the kind of factthat you're going to see boasted in a city report, right? it's not going to bethe front page on nyc.gov.
you're not going to see it there, but the fact that we can getto that data is awesome. but once again, it wasn't super easy, because this data was noton the open data portal. if you were to go to the open data portal, you'd see just a snippet of it,a year or a few months. it was actually on the departmentof environmental protection's website. and each one of these links is an excelsheet, and each excel sheet is different. every heading is different:you copy, paste, reorganize.
when you do you can make mapsand that's great, but once again, we can do better than thatas a city, we can normalize things. and we're getting there, becausethere's this website that socrata makes called the open data portal nyc. this is where 1,100 data setsthat don't suffer from the things i just told you live, and that number is growing,and that's great. you can download data in any format,be it csv or pdf or excel document. whatever you want,you can download the data that way.
the problem is, once you do, you will find that each agencycodes their addresses differently. so one is street name,intersection street, street, borough, address, building,building address. so once again, you're spending time,even when we have this portal, you're spending timenormalizing our address fields. and that's not the best useof our citizens' time. we can do better than that as a city. we can standardize our addresses,
and if we do,we can get more maps like this. this is a map of fire hydrantsin new york city, but not just any fire hydrants. these are the top 250 grossing firehydrants in terms of parking tickets. so i learned a few things from this map,and i really like this map. number one, just don't parkon the upper east side. just don't. it doesn't matter whereyou park, you will get a hydrant ticket. number two, i found the two highestgrossing hydrants in all of new york city, and they're on the lower east side,
and they were bringing in over55,000 dollars a year in parking tickets. and that seemed a little strangeto me when i noticed it, so i did a little digging and it turns outwhat you had is a hydrant and then something calleda curb extension, which is like a seven-footspace to walk on, and then a parking spot. and so these cars came along,and the hydrant -- "it's all the way over there, i'm fine," and there was actually a parking spotpainted there beautifully for them.
they would park there, and the nypddisagreed with this designation and would ticket them. and it wasn't just mewho found a parking ticket. this is the googlestreet view car driving by finding the same parking ticket. so i wrote about this on my blog,on i quant ny, and the dot responded, and they said, "while the dot has not receivedany complaints about this location, we will review the roadway markingsand make any appropriate alterations."
and i thought to myself,typical government response, all right, moved on with my life. but then, a few weeks later,something incredible happened. they repainted the spot, and for a second i thought i sawthe future of open data, because think about what happened here. for five years, this spot was beingticketed, and it was confusing, and then a citizen found something,they told the city, and within a few weeks the problem was fixed.
it's amazing. and a lot of peoplesee open data as being a watchdog. it's not, it's about being a partner. we can empower our citizensto be better partners for government, and it's not that hard. all we need are a few changes. if you're foiling data, if you're seeing your databeing foiled over and over again, let's release it to the public, that'sa sign that it should be made public. and if you're a government agencyreleasing a pdf,
let's pass legislation that requires youto post it with the underlying data, because that datais coming from somewhere. i don't know where, but it'scoming from somewhere, and you can release it with the pdf. and let's adopt and sharesome open data standards. let's start with our addresseshere in new york city. let's just startnormalizing our addresses. because new york is a leader in open data. despite all this, we are absolutelya leader in open data,
and if we start normalizing things,and set an open data standard, others will follow. the state will follow,and maybe the federal government, other countries could follow, and we're not that far off from a timewhere you could write one program and map information from 100 countries. it's not science fiction.we're actually quite close. and by the way, who are weempowering with this? because it's not just john kraussand it's not just chris whong. there are hundreds of meetupsgoing on in new york city right now,
active meetups. there are thousands of peopleattending these meetups. these people are going after workand on weekends, and they're attending these meetupsto look at open data and make our city a better place. groups like betanyc, who just last weekreleased something called citygram.nyc that allows you to subscribeto 311 complaints around your own home,or around your office. you put in your address,you get local complaints.
and it's not just the tech communitythat are after these things. it's urban planners likethe students i teach at pratt. it's policy advocates, it's everyone, it's citizens from a diverseset of backgrounds. and with some small, incremental changes, we can unlock the passionand the ability of our citizens
to harness open dataand make our city even better, whether it's one dataset,or one parking spot at a time. thank you.
(applause)
