Research notes

Using Python to access tweets from the command line

Here at the Harvard of the Plains, I teach a class in digital product development I like to call Programming as an Act of Journalism. Lots of people ask me about it and I’m always a bit cagey about it because, to be frank, I’m still kind of making it up as I go. My course goals could actually fill an entire degree, so I spend a lot of time pushing and pulling against my wants and needs for the class. But the basic outline is I take College of Journalism students who know nothing about code and product development and we build a prototype of a product they invent in 16 weeks. And the first eight weeks of hands-on classes are spent doing intro to programming type work. 

I’ve started developing lessons I call Small Wins for the class so they can see stuff working on the screen. Here’s the first: a lesson in using Python to access the Twitter API. It’s the follow up class to basic Python (variables, strings, integers, functions and lists). The goal is to talk about using libraries, introduce programmatic thinking and data structure and, well, see some stuff happen in front of their eyes.

In the class, I have students install a virtual machine and put Ubuntu Linux on that. Why? Lots of reasons, but the main ones are that I want to expose students to a new environment while at the same time keeping it compartmentalized so they don’t fear “messing up their computer” by working in the terminal. So this walkthrough assumes you’re using Ubuntu.

1. Install pip, a package management library for Python.

$ sudo apt-get install python-pip python-dev build-essential 

2. Install python-twitter, a Python wrapper around the Twitter API that makes accessing the API breathtakingly simple (really!).

$ sudo pip install python-twitter

Start Python

$ python

Create an instance of the Twitter API class, which creates an object with several methods that we’re going to be using. Think of it like buying a Swiss Army knife. You get the knife and a whole bunch of tools that go with it. You just have to unfold them to use them.

>>> import twitter

>>> api = twitter.Api()

5. Pick a Twitter user to gather their tweets. For this example, we’ll use Stephen Colbert (@StephenAtHome). Then we’ll use one of our tools in the Twitter API class — GetUserTimeline, which does what you think it does.

>>> statuses = api.GetUserTimeline('StephenAtHome')

6. Congrats. You now have Stephen Colbert’s latest tweets. Stand in appreciation of what all the python-twitter library has done for you. It created an http request, sent that request to the properly formatted API URL, ingested the response, converted the structured JSON file it returned into a Python dictionary and returned it to you as an object for you to use. What’s that? You’re not excited? You want to see them? Okay, fine.

>>> for tweet in statuses:

...     print tweet

Yikes. What is that? It’s all the data that comes with each tweet. Take a look at it all. Background colors, times, dates, all kinds of stuff. Well, that won’t do. Maybe we just want to see the text of the tweet. If you look at the output, you’ll see a pattern in the output. You’ll see things like “protected”: false and “screen_name”: “StephenAtHome”. Those are called key/value pairs. The key is the name of the attribute — like an ID or the text of the tweet — and the value is what it sounds like it is.

7. In our case, we want to see the text of the tweet. To do that, you have to address the key. In Python, we do that with dot notation. What does that mean? It means you can pick the key in an object and display it by adding a dot after your object and then the key name. So like this: nameofobject DOT name of key

>>> for tweet in statuses:

...     print tweet.text

Exercise:

— Looking at the keys, what would you have to do to see if each tweet was favorited?

8. Okay, one users tweets are nice, but what if I want to see tweets about a subject, not a person? For that, we’ll use a different function in the python-twitter library, GetSearch:

>>> tweets = api.GetSearch("politifact", per_page=100, page=1)

Exercise:

— Using what you’ve learned, how would you print out the text of the tweets gathered in step 8?

Thoughts, suggestions, criticisms? The comment box awaits you. Want to go further with this? Here’s the python-twitter code and documentation. 

A completely arbitrary list of takeaways from two unconferences

This past weekend, I attended Spark Camp: Data, an unconference in Austin focused on using data to tell stories, whatever they may be. A month before, I was at News Foo, an unconference at Arizona State University that brings technologists and journalists together to talk about … whatever they want to talk about regarding the future of news. Both conferences included a lot of chatter about journalism schools and what they should be doing. People I talked to were all fascinated to hear I teach programming and data at a journalism school.

At both, listening to this discussion going on, I came away with some random thoughts about journalism school curricula, programmer-journalists and the future. Here they are in a completely unnecessary and arbitrary list.

• The number of things Journalism is asking its journalism schools to teach could fill three degrees plus a couple of minors. Business, law, economics, entrepreneurship, computer science, data science, and also all the journalism fundamentals. We have no idea what The Future is, other than that it’s wildly different from the past, so we’re tossing everything into What Journalism Schools Should Be Teaching and the list is starting to look a little silly. Especially when you consider we have 40 credit hours to work with.
• I view this as a challenge, not a lament.
• You are number 114 on the list of people who have asked me if I have any students who are budding journalist-developers ready to start busting out apps in your shop. You are also deep on the list of people telling me you’re looking for people and having a hard time finding them. There’s 10 fish in this pond right now and everyone has a line in the water.
• The number of students I have seen who are budding journalist-developers ready to start busting out apps in your shop: 0. Why? Comes down to passion. I haven’t seen that student take what we talked about in class and run off on their own. It seems they’re still waiting for something. I don’t know what that is.
• Where are these future journalist-developers who will Save Us All? What can we do to find them? I don’t know. I think more about it every day.
• I think the problem with finding these students starts with reward structures. Students are told from even before they walk on campus that being a journalist means Being a Good Writer, Being a Good Editor, Being a Good Photographer. No one is telling them they could be an application developer, or a data journalist, or a media entrepreneur. Or if they have heard it, that voice is getting drowned out by traditionalists. A disturbing amount of time, the traditionalists drowning those students out are other students. Until we can attach a reward to this — until it cracks the consciousness of students that there are jobs in this path — I think we’ll continue to struggle. 
• I still believe you can teach journalists to be programmers and people like that will be vital to the Future of Journalism, but I’m also starting to think more and more about what a journalism minor for a computer-science major would look like. 
• A potential archetype of a journalist-developer student? That kid who messed around with programming in high school and loved it while having a blast on the student paper. But they came to college and thought they had to get a CS or similar degree because that’s what They told them. You know They. They tell people a lot of crap. I’m looking for that techie carrying regret in their heart for not pursuing journalism. I can unburden that regret. And get you a job.
• The pipeline of techie/journo students to internships to jobs is a problem that is going to be with us for a while. There’s not enough students right now. Plain and simple. And I worry that because of this supply problem, the internships and jobs will go away. And just when that happens, we’ll solve the supply problem and have the reverse.
• Problems I would love to have to talk about: What is the career path for a developer in a newsroom? There isn’t one right now. Who will be the first to hire a developer as an assistant managing editor or above? I ache for the day we have to discuss this instead of the scarcity of talent. I long for the day when we have to debate turning over editorial strategy to someone who came into the newsroom to build apps. That will be a great day.

A word of advice for Code Year participants

A whole herd of people are learning to program this year through free weekly lessons via email from Code Year. Some, like me, are interested in new lessons and approaches. Some people are starting from scratch. I have a word of advice to you, one I need to follow myself, as you get started on your Code Year.

Unplug.

Turn off Tweetdeck. Shut down IM. Turn off your email notifier. No one has liked your status in the last minute, so don’t check. Unplug. 

Why? Because you’ll learn nothing in 20-30 second bursts between distractions. Learning to code, like many things in life, requires you to focus for extended periods of time. It requires to you really burrow in on details. This is probably a level of detail you may not be accustomed to. It’s probably concepts you aren’t familiar with. You need uninterrupted time to focus on the task at hand. 

In short, you aren’t as good at multitasking as you think you are. So don’t do it. Unplug, focus, and really learn something.

Want an experiment in how distracted you are? Get into the environment you intend to work on your Code Year exercises in. Turn on the stuff you normally have on. I’ve got Spotify, an IM client and Tweetdeck on myself right now, plus my Google notifier and my phone handy. Turn all the stuff on that you’d normally have running.

Now read this. It’s a fantastic essay on how important solitude is to proper thinking. See how long you can go before you’re distracted.

Me? Two paragraphs. It took me over an hour to read that one essay because oops, got an email and now someone @ replied me on Twitter and awwww, cute baby on Facebook and here’s a text about getting dinner tonight. And on and on and on and on.

It was shocking to me how my social/work habits had ruined my ability to concentrate on a thing and read it. Really read it. At times, the urge to check email was physical. I could feel myself getting uncomfortable because what if I did get an email? What if? Well, I better check. 

And so I’m making a change. At points in my day now, when I need to focus on something, I’m opting into a total digital blackout. Goodbye email notifier. So long Twitter. Shhhhh now Spotify. It’s thinking time. They’ll be there in an hour. And I’m not an on-call brain surgeon, so no one is going to die if I don’t get that text.

It’ll wait. It can all wait. It’s thinking time. Time to focus.

Data journalism class description: your thoughts?

I’m teaching a data journalism/investigative reporting class for the first time this spring. I’ve got the class pretty well mapped out — I know what I’m going to teach — but I’m struggling with a course description. Here’s what I’ve got. Fellow data nerds, what say you?

Every day, more of our lives is becoming digital and more of that life is getting stored in a database somewhere. With a historic explosion of data about everything going on right now, reporters need the skills to analyze and understand data to then write the stories hidden in the information. Gone are the days when a reporter could grab a notebook and say “I suck at math.” If that’s what you aspire to, leave now. Don’t want to be that reporter? Welcome. Data journalism harnesses the tools of the data analyst and uses them to do investigative reporting. We’re going to get our hands dirty with spreadsheets, databases, maps, some basic stats and, time permitting, some stuff I’ll call “serious future s**t.” And in the end, we’ve got a project to produce. So buckle up and hold on.

Thinking out loud: The management wisdom of Battlestar Galactica

I’m going to News Foo in Phoenix in a few weeks, and I’m thinking of proposing an Ignite talk there called the Management Wisdom of Battlestar Galactica. I’m a huge fan of the re-imagined Battlestar Galactica series that was on Syfy. Besides having lead a rag-tag fleet of the last humans left alive in a Cylon holocaust, I think Admiral William Adama would have been a pretty decent project manager. Here’s what I’ve come up with as project management wisdom from the Admiral and the show:

1. Sine Qua Non. Means “without which not” or the must-have thing that makes everything else possible. I’ve argued that a good news app focuses on one thing and does it really well and that if you can’t say what that one thing is clearly, you don’t have a project.
2. “It’s not enough to survive … One has to be worthy of survival.” If you want to do something special, you have to commit to it. This is as much about project groups as it is projects. It can be summed up thusly: Clock punchers should GTFO.
3. “Then grab your gun and bring in the cat.” If you believe in what you are doing, then come prepared to fight for it. Stand your ground. Make an argument. Fight to make your project better. 
4. “Sooner or later, the day comes when you can’t hide from the things that you’ve done anymore.” There’s getting the job done and getting the job done right. Any corners you cut now will come back to you. Get it done, get it online, get people using it, iterate often, but know your technical debt will come due one day. 
5. “Sometimes you have to roll a hard six.” A hard six is two threes on a pair of six-sided dice in the game of craps. It’s one of the highest odds rolls in the game, but with higher risk comes higher reward. If you aren’t gambling big, why are you playing?

Thoughts? I kinda dig the SciFi movie explains Thing Not Related To Science Fiction genre of lightning talk. I’ve given a talk called The Matrix Explains the Current News Business several times and it works pretty well. Not sure about this one. Let me know if you have a thought.

My other idea? What can the Zen Buddhist concept of the Beginner’s Mind tell us about the future of journalism.

Help me refine an incomplete idea

The McCormick Foundation and the Poynter Institute for Media Studies are funding specialized reporting institutes in 2012 and are taking applications now. I saw this and had an idea for one, but I’m not convinced it’s fully baked. Help me out by adding your suggestions in the comments below.

The idea (the short version):

Reporting for News Apps: Getting, Cleaning, Vetting, Analyzing and Visualizing Data to Tell Stories on the Web.

The slightly longer version:

Done right, news applications require a combination of skills, from investigative reporting to data literacy to information design principles to programming. A reporter working on a news app could face open records challenges, dirty data, questions about validity and accuracy, the formation of analytical approaches and the need to know when geographic data should be a map or not. And that’s before the first line of code hits the internet. There’s a lot to learn — and a lot to learn from. This specialized reporting institute would focus on the challenges specific to reporting for news apps, how the steps can be improved, how other fields within and outside of journalism have tackled these problems and how the results of the reporting-for-news-apps process can be extended to other parts of journalism. 

Limitations

• McCormick/Poynter have specific goals they wish to fulfill with their conferences. Full list here.
• Specifically, they want the institute to focus on a topic. The first topic that came to my mind was education data, specifically school test score data that every state has and many news organizations produce apps around. Thoughts?
• They place an emphasis on the diversity of the conference, in type of journalist who attends (e.g. broadcast, print, online), type of outlet (newspaper, ethnic media, independent) and the speakers. Technology conferences have a well-earned “white dude” problem. Thoughts on how we reach beyond the core of nerdy dudes who do this now and beyond the proto-nerd who might be interested in attending?

I think this is solid idea for a conference, but I’m not sure it’s fully baked. So, instead of me sitting here wondering what else I should put in the application I’m going to submit by the deadline of Nov. 15, I’m going to ask you. What else should I put in there?