Introducing Twools – Social Media Unleashed

Twitter stopped offering RSS feeds in June this year and made it much more difficult to access your data. That’s why I built Twools- a way for you to unleash your Twitter data. You shouldn’t need to be a developer or know how to write code to be able to do this- so I have tried to make this as easy as possible.

---

---

Twitter Data in the Good Ol’ Days

It’s hard to believe that Twitter started back in 2006- that’s 7 years ago from the time I write this article. In the early days, it was very basic and despite early popularity- it was mainly popular amongst geeks, developers and other tech people. I signed up back in November 2006 despite not really understanding what the fuss was about. Well these days I think we’d be hard pushed to find someone who hasn’t heard of Twitter with 554 million users worldwide, 115 million active users every month, 135,000 new sign ups every day and 58 million tweets per day.

How did Twitter become so popular? I think a big part of it’s growth has been due to the developer community coming up with ideas on how to use Twitter and building apps for it. Not long after Twitter opened it’s doors, they released a system that allowed developers to access Twitter’s data including tweets and user information. That system was Twitter’s API- Application Programming Interface. An API helps a developer interact with an application- sending and receiving information. In the early days it was incredibly easy to access your data- it was just a case of getting a feed of your tweets or mentions which were available as RSS or XML feeds. You didn’t need to be a programmer, because you could add that feed to another app such as an RSS feed reader, Yahoo Pipes or a few years later, IFTTT.

---

Twitter Wants to Make Money

Twitter is free for end users, but of course it has to make money somehow. That somehow of course is advertising, but how could Twitter properly develop its advertising platform when it didn’t have control of the way its data was being presented? Twitter started buying up Twitter clients left, right and centre- TweetDeck being the most famous. They also started to lock down their API, meaning it was no longer possible for just anyone to access the data- you needed to be logged in or authenticated- and that raised the bar of access.

I highly recommend listening to a snippet from an interview between Emily Bell and the CEO of Twitter, Dick Costolo. He explains the thinking behind the changes in the API around 20 minutes in:

[soundcloud url=”http://api.soundcloud.com/tracks/60701848″ params=”color=ff6600&auto_play=false&show_artwork=true” width=” 100%” height=”166″ iframe=”true” /]

---

RSS & Authentication – Apparently, RSS Feeds “are infrequently used today”

When the latest version of the Twitter API (version 1.1) was launched, they announced that they were retiring RSS feeds. According to Twitter, XML, Atom, and RSS are infrequently used today. I completely disagree with that. I think the real reason why Twitter have retired RSS feeds was mainly due to the fact that they were not able to control access to their API. With the new version of the API everyone has limits on how many calls they can make every 15 minutes with your authenticated app. With RSS, because you aren’t authenticated, there is no limit. I appreciate Twitter has to keep a lid on their API- as allowing anyone to access their API with unlimited calls would be impossible.

However it is sad that the only way to access your Twitter data in the way it used to be is now firmly in the hands of developers. If you don’t know how to code and build an app that connects to the API you are out of luck,

---

Twitter and the Cloud- Some Confusion

With version 1.1 of Twitter’s API there were new “Rules of the Road- Twitter’s terms and conditions of use. Most of the terms make sense- and it is important for Twitter to make it clear how their data should be used. However there is one line in particular that has caused a lot of confusion in 1.4.a.iii:

You may provide spreadsheet or PDF files or other export functionality via non­-programmatic means, such as using a “save as” button, for up to 100,000 public Tweets and/or User Objects per user per day. Exporting Twitter Content to a datastore as a service or other cloud based service, however, is not permitted.

From the Twitter API Rules of the Road

So, we are all allowed to copy up to 100,000 public tweets every day as long as it is not via an app (I am assuming that is what it means by “via non­-programmatic means”). However what does “Exporting Twitter Content to a datastore as a service or other cloud based service” actually mean? It’s very vaugue! Does that mean we aren’t allowed to archive our tweets? That can’t be true, since Twitter offer their own archiving service where we can download all our tweets to our computers. There is no doubt, however, that it was this clause which caused the app IFTTT to remove their Twitter Triggers

---

RIP IFTTT Twitter Triggers

If you’re a regular reader of my blog, you’ll already know that I am a huge fan of the service IFTTT (If This Then That). IFTTT allows you to connect lots of different apps and services together (Twitter, LinkedIn, Flickr, Instagram, Facebook and even switches at home). A “Trigger” was something that would cause IFTTT to do something. For example “If you post a new Flickr Photo” would be a “Trigger”. An “Action” is something that IFTTT does as a result of an IFTTT Trigger. For example “…then post a Tweet”. A full example of a Trigger and an Action could be “IF you post a new Flickr Photo THEN send a Tweet”

In September 2012, IFTTT made an announcement that they they were removing Twitter Triggers. They were finally removed on September 27 and there were a lot of upset people. The reason for this, according to IFTTT was that “Twitter announced policy changes that will affect how applications and users like yourself can interact with Twitter’s data.”

I later wrote an article on how you could Get Your Twitter Triggers back using RSS. This worked by using the RSS feed for your Twitter timeline as an action in IFTTT. Unfortunately this now does not work.

We knew it was coming, but in June 2013, Twitter retired version 1 of their API together with the retirement of RSS, XML and Atom feeds. Now, once you’ve built an app that authenticates with the Twitter API, you will be given the data in JSON format. Whilst JSON is great for developers in that it is rich in data, it is not so helpful for everyone else. With so many people wanting to get their RSS feeds back, I decided to build my own solution.

---

Twools – The Solution to a Problem

Twools is my attempt at bringing back RSS feeds and so freeing your data. I have tried my best to make this as easy as possible for everyone to use. That’s quite a task, because Twitter don’t make it easy!

You install Twools on your own website. The advantage of this is that you control the way it works- and it is not very difficult to do. You just download Twools, add your Twitter App information and upload to your website. It’s that easy. Of course, you have to have a website- and it has to run PHP. That may be a deal breaker for you. However there are a huge number of websites that use PHP- WordPress is an axample- so hopefully you’ll be fine.

What Twools Can Do

You can use Twools to back up your tweets, or automate certain tasks. You could use in conjunction with IFTTT or another app to do some cool things. For example you could

• Cross post to LinkedIn using the #li hashtag
• Bookmark or cross post your Twitter Favorites
• Add new followers from a certain country or location to a Twitter list
• Get a text message when a company tweets a special offer
• Add articles that you tweet using a certain app to evernote.
• Get emailed when someone mentions your brand negatively.
• Get emailed when someone links to an article on your website even if they don’t mention you.
• Add someone mentioning a keyword to a Twitter list

What Twools Gives You

At its core, Twools gives you back your RSS feeds. However, it gives you many more RSS feeds than Twitter ever gave you. In particular, Twools gives you.

• Your home timeline
• User timeline (yours or another user’s)
• Your mentions
• Your favorites
• List timeline (from a list that you own)
• Direct messages received (disabled by default)
• Direct messages Sent (disabled by default)
• Twitter Search
• New followers
• New friends

You can then run filters on these feeds, to only return certain results. These are very powerful. For example:

Filters on Tweets

• Containing any keyword, #hashtag or @screen_name: You can filter by a single keyword, hashtag or screen name in this field. For example if you wanted to return only tweets containing the hashtag #socialmedia, you would enter that here. Or you could put a screen name (for example @iagdotme) to only return tweets that mention that user.
• Containing all keywords, #hashtags or @screen_names: You can filter by multiple keywords, hashtags or screen names in this field. This is similar to the previous field except that you can add multiple keywords, hashtags or screen names by separating them by a comma. For example you could enter “#socialmedia, @iagdotme” to only return tweets containing the hashtag #socialmedia and mentioning @iagdotme
• Containing none keywords, #hashtags or @screen_names: You can filter by multiple keywords, hashtags or screen names in this field. This is similar to the previous field except only tweets that DO NOT contain any of these keywords, hashtags or screen names will be displayed.
• Containing single #hashtag: Identical to the first field except that this specific hashtag can be removed from the feed output if selected in the final option.
• From: Only tweets from these users will be returned
• Not From: Only tweets not from these users will be returned
• Using Apps: You can filter by the app used to send the tweet. You can select multiple app names by separating by a comma. For example, to return tweets posted with either Hootsuite or Buffer you could enter “Hootsuite, Buffer” here.
• Not Using Apps: Similar to the previous field but here you can return tweets that were posted not using these apps. You can select multiple app names by separating by a comma. For example, to return tweets that were NOT posted using either Hootsuite or Buffer you could enter “Hootsuite, Buffer” here.
• Exclude Scheduling Apps: By checking this box you can exclude a variety of scheduling apps (such as Buffer, IFTTT, Social Oomph and Triberr) from the results. You can edit the list by editing the config file (see later).
• Exclude RTs: By checking this box you can exclude RTs from the results. This uses Twitters own API to only return non retweets, but also searches for RT in the tweets.
• Been Favorited: Only return tweets that have been favorited.
• Been Retweeted: Only return tweets that have been retweeted.

Filters on Tweets

• In Location: You can filter by the location of the person who tweeted the tweet. This filter looks at the location string the users’ profile.
• In Bio: You can filter a keyword from the user’s profile.
• Writes in Language: You can filter by the language that person who tweeted the tweet writes in.

Once you’ve returned the results you want, you can then do some final magic on the results. We can manipulate the output to remove hashtags, screen names or links or just give us the screen name instead of the tweet. Instead of linking to the tweet itself we could link to the URL in the tweet- and unshorten it so that we have the full URL.

There are so many possibilities, that it might take some time to get your head round it all.

Share Twools Recipes

I’ve built in a sharing facility into Twools. This way we can all share our ideas of feed recipes with each other. Basically all this does is to share the query string that Twools uses to create the feed, but it should make it easier to share and use recipes.

Here are some examples:

1. Archiving: Articles I post on Twitter using Triberr (or other app)

Triberr is a service for Bloggers to share your content with other bloggers. You basically share articles to your social networks from people within your Tribe. Triberr doesn’t offer an RSS service, so I use this to add all the articles I share on Twitter using Triberr to Evernote or Pocket. The articles I share on Triberr are ones that I think are awesome, so I do want to bookmark them for later.

This Twools recipe retrieves your tweets posted via Triberr. It removes URLs & screen names from the tweets. As well as this, it removes hashtags from the end (and removing ‘#’ from the rest). It extracts the URL and unshortens it and puts it in link field.
Show Recipe

---

2. Community Building: Return anyone who uses the hashtag #socialmedia

You could use this feed with IFTTT to automatically add someone who uses this hashtag to a list.
Show Recipe

---

3. Cross Posting: Tweets from your timeline that use the #li hashtag

Could be used to cross post to LinkedIn when using the #li hashtag
Show Recipe

---

4. Archiving & Cross Posting: Your Twitter favorites

All URLs, screen names and hashtags have been removed. URL extracted and de-shortified. Could be used for Bookmarking or cross post your Twitter Faves
Show Recipe

---

5. Community Building: Recent followers from the UK

Retrieves the screen names of all your recent followers with the keyword “UK” in their location. You could use this to automatically add your latest followers from the UK to a Twitter list using IFTTT.
Show Recipe

---

6. Monitoring: Latest people mentioning Startbucks in the UK negatively

Retrieves the latest tweets from people who have the keyword “UK” in their location who mention Starbucks with negative sentiment. You could use this with IFTTT to be sent an SMS or an email whenever someone tweets about your brand in a negative way.

Show Recipe

---

Streams – Super Simple Social Media Management Tool with Powerful Filters

RSS feeds are powerful, but sometimes you might want to have a look at the results visually in the same way you can on Twitter, TweetDeck or Hootsuite. Streams gives you this, kind of. It’s nothing fancy- and it isn’t supposed to be. It gives you a stream of the tweets with the same filters you get with the feed generator. You can then reply, favorite or retweet any of these retweets.

I use this myself quite a lot, because I find a lot of social media management tools a little lacking in filters. For example, with Twools Streams I can show the tweets from people in one of my Twitter lists excluding retweets and ones posted using a scheduling app. I also find it useful for doing a Twitter search of those people linking to an article on my blog excluding myself and ones mentioning me (since I’ll already see these in my mentions stream).

---

Keep an eye on your API usage

Twitter have locked down the number of calls you can make to their API. You have a set number of calls every 15 minutes. So to keep on top of your usage, I’ve built an API limits page in Twools which shows your limits and how many calls you have left in your current 15 minute window.

---

How do I install Twools?

If you use WordPress I highly recommend using the WordPress plugin version of Twools. With the plugin all you need to do is install it and fill in the details. See the Twools page for more details.

First of all you need to download the latest version. I’m not charging for it, but I ask that you fill in the form at the top of this page. I’d also love it if you could give me some feedback.

Twools will run on your own website. It is written in PHP so your website must have PHP installed. You don’t need to be a developer or a coder, but you do need to know how to upload files to a directory on your website. You could use an FTP programme, or perhaps you have another way.

Firstly, you will need to get Twitter access tokens. I’ve written an easy to understand article called “How to Create a Twitter App in 8 Easy Steps“.

Then you also have to edit a few lines in the config file. Please don’t be scared- it is very easy. Just open a text editor and open the config.php file up (it is in the /incs/ folder).

Firstly, you will need to set the username and password you want to secure Twools (replace “demo” and “demo” with your new username and password):

// Set username and password to protect access to Twools (RSS feeds will still not be password protected).
// Put some jibberish on the padding string for extra protection
$config['username'] = "demo";
$config['password'] = "demo";
$padding = "g`0*Y;6VOx`AIb+WBE>uKU1w;_cV-`&Y[ZgUDeZ3~jmdx]u7DGKUVN1W9~~W;kbk";

Once you’ve done that, enter those access tokens here:

/** Set access tokens here - see: https://dev.twitter.com/apps/ **/
$settings = array(
'oauth_access_token' => "",
'oauth_access_token_secret' => "",
'consumer_key' => "",
'consumer_secret' => ""
);

As well as this, you should enter your Twitter username by entering it within the quotes for each item:

// Replace by setting the default Twitter username (exclude the @ symbol)
$defaultUser = "iagdotme";

Once you’ve done this, you need to upload everything and run from your website. That’s it! There are other features you can activate such as the experimental spam filter and switch on direct message feeds. I go into more detail in the Twools documentation.

To run Twools, visit the folder on your website where you installed it. For example if you put it in a folder called “twools” and your website is example.com, you will need to visit http://example.com/twools/. Once you’ve done that, enter your username and password and you should be in.

---

Acknowledgements

A huge thanks to James Mallison for explaining how to build a simple Twitter app using v1.1 of the API and for building a fantastic PHP Twitter wrapper. This little app wouldn’t work without it!

This app also uses the Twitter Bootstrap framework for its HTML, CSS and Javascript. It means it is fully responsive and uses the latest in HTML5 and CSS3

---

Conclusion

I hope you find Twools helpful. Let me know how you get on and how we can improve it. I have lots of interesting ideas for the future. Also, share your recipes. I’ll be writing up a few more articles which will include some of them soon!