Blog

Download you twitter feed using Ruby on Rails with OAuth

Jonathon Horsman Sep 15 10 comments

Twitter provides an easy to use API allowing you to display your tweets on another web site or inside another application.

This article explains how to do this in a Ruby on Rails application using the new OAuth authentication mechanism.

It assumes you already have a Ruby on Rails application. I’m not going to go into tedious OAuth implementation details, just the results!

Register your application on the Twitter developer site

Create your application on Twitter, go to http://dev.twitter.com and click Register an app. Fill in the details giving it a name such as “My site tweet feed” and make it read only. The values you enter don’t really matter since no one else will be using your application, so long as a basic description and application website are set.

Copy your consumer key and consumer secret

After registration you should see your Twitter application settings page:

Twitter API settings page

Take note of the Consumer key and Consumer secret values, you will need to copy these into your application later.

Copy your access token

Click the My Access Token link on the right.

Twitter Client Token settings page

Note the Access Token and Access Token Secret, you will also need to copy these later.

Now you’re ready to modify your Ruby on Rails application to connect to the Twitter API.

Install the gem

There are a number of Ruby gems available for connecting to the API. I’ve had success with Grackle so that’s what we’ll use.

Install the Grackle gem:

gem install grackle

Create a table for storing tweets

Create a migration which will create a table for storing your tweets:

script/generation migration CreateTweets

Edit that migration, creating the table with 2 fields:

class CreateTweets < ActiveRecord::Migration
  def self.up
    create_table :tweets do |t|
      t.string :content
      t.datetime :created
    end
  end

  def self.down
    drop_table :tweets
  end
end

Create a model for the new table

Create a Tweet model, and add a method for downloading and storing the tweets:

require 'grackle'

class Tweet < ActiveRecord::Base

  MY_APPLICATION_NAME = "arctickiwi"
  
  """Connect to the Twitter API and pull down the latest tweets"""
  def self.get_latest
    tweets = client.statuses.user_timeline? :screen_name => MY_APPLICATION_NAME # hit the API
    tweets.each do |t|
      created = DateTime.parse(t.created_at)
      # create the tweet if it doesn't already exist
      unless Tweet.exists?(["created=?", created])
        Tweet.create({:content => t.text, :created => created })
       end
    end
  end
  
  private
  def self.client
    Grackle::Client.new(:auth=>{
      :type=>:oauth,
      :consumer_key=>'copy_from_twitter_consumer_key',
      :consumer_secret=>'copy_from_twitter_consumer_secret',
      :token=>"copy_from_twitter_access_token",
      :token_secret=>"copy_from_twitter_access_token_seceret"
    })

  end
end

Make sure you update your MY_APPLICATION_NAME, unless you want to get our tweets. Also enter your consumer key, consumer secret, token and token secret.

See if it works

You should now be able to download your latest tweets and store them in the database:

On the command line, run script/console

Tweet.get_latest

You should see a whole lot of text flash past the screen as the client hits the API, downloads the tweets and sticks them in the database.

If you get an error like this:

Grackle::TwitterError: get http://api.twitter.com/1/statuses/user_timeline.json?screen_name=arctickiwi => 
401: {"request":"/1/statuses/user_timeline.json?screen_name=arctickiwi","error":"Invalid / expired Token"}

it means one of your keys or secrets is wrong.

If it worked and you’re still in the console, you can see all your tweets now in the database:

Tweet.all

Display your tweets

As with any other model in Rails you can do whatever you want with them, like display them on a web page.

You will probably want links in your tweets to be real clickable links on your site, so here’s a helper method for to wrap links with anchor tags (in app/helpers/application_helper.rb)

module ApplicationHelper
  def display_content_with_links(tweet)
    tweet.content.gsub(/(http:\/\/[a-zA-Z0-9\/\.\+\-_:?&=]+)/) {|a| "<a href=\"#{a}\">#{a}</a>"}
  end
end

Create a cron job

You will want your tweets downloaded automatically all the time. I call a rake task from cron job which runs every 10 minutes for this.

In lib/tasks create a file called download_tweets.rake

task :download_tweets => :environment do
  Tweet.get_latest
end

and inside cron:

0,10,20,30,40,50 * * * * cd /path/to/my/app && RAILS_ENV=production /usr/bin/env rake download_tweets >> /path/to/my/app/log/cron.log 2>&1

I output to a log file so if something goes wrong one day I can see why.

FYI using the whenever gem is great for automatically creating cron jobs in conjunction with Capistrano

Good luck, and please leave a comment if you find this useful.

Comments //

Jon

Jon Sep 30

Any ideas how to run a cron job for a ASP.Net MVC site?

Jonathon Horsman

Jonathon Horsman Sep 30

Hi Jon

We don't use Windows or develop in ASP, but I think you want the Windows Scheduler under Control Panel.

HTH

nayak

nayak Nov 26

Hi ,
Thanks for this nice article

i am using twitter api for my site it was working fine but from past 1 week tweets are not displaying in my site when i checked the log file its showing fetching data from people which i have requested but it is not storing into my database i dont known where i am going wrong it is not displaying any error message also can u help me

Jonathon Horsman

Jonathon Horsman Nov 29

Hi Nayak

What happens when you run Tweet.get_latest in the Rails console?

I'd suggest posting your question on stackoverflow.com if you're having problems. You'll need to include your code and the log output too.

Jonno

Jon G

Jon G Jan 24

The line "unless Tweet.exists?(["created=?", created])" doesn't seem to be working for me. When I run "Tweet.get_latest" in the console, it creates the same tweet twice. Any reasons why this could be?

I think the problem is I don't really understand the breakdown of the line. What does the "created=?" (checks to see if it's already been created I'd imagine?) but then what does the ", created" do after it?

Jonathon Horsman

Jonathon Horsman Jan 24

Hi Jon

That "exists" line checks if a tweet has already been created with the same date/time (created is a timestamp), since that's how I'm uniquely identifying tweets.

However you could probably use the id_str parameter instead since that's probably more appropriate for uniquely identifying tweets.

Hope this help

Jonno

Tyler Breland

Tyler Breland Feb 10

Hi, I loved the post, but I am having an issue grabbing the profile_image_url. What I've done is just added a new migration which created an :avatar column in the Tweet table then I added :avatar => t.profile_image_url, in the create! hash. Grackle isn't pulling it, and according to the api, I am using the right method. Any thoughts? I don't see why this wouldn't work.
-Tyler

Jonathon Horsman

Jonathon Horsman Feb 11

Hi Tyler

Looks like the profile_image_url belongs to the 'user' associated with the tweet.

So in order to get the profile image URL you need to do :avatar => t.user.profile_image_url

Hope this helps

Jonno

geof

geof Jul 06

very nice tutorial! I also saw another related gem, tweetstream which looks very promising.

Stephen McCurry

Stephen McCurry Apr 13

Very helpful thanks. I used the Twitter Gem but was able to follow this example easily and adapt. It is surprising how few good examples there are of doing this basic Twitter integration.

Post a comment

  1. optional

Recent Tweets

Blog: The monumental Myspace cock-up: http://bit.ly/emgRKV
Tweeted on Friday at 09:43

Awww railsapi, delete some logs: http://bit.ly/htBNDH
Tweeted on Wednesday at 16:15