Introducing Twools – Social Media Unleashed
Twitter stopped offering RSS feeds in June this year and made it much more difficult to access your data. That’s why I built Twools- a way for you to unleash your Twitter data. You shouldn’t need to be a developer or know how to write code to be able to do this- so I have tried to make this as easy as possible.
WordPress Plugin Out Now!
To make this even easier to install and use, I’ve developed a WordPress version of Twools.
Unleash your Social Media- it’s Free!
Please fill in this form and you’ll be sent a link to download Twools. Please make sure you read this article and the Twools documentation.
Fill in this form to download twools.
Twitter Data in the Good Ol’ Days
It’s hard to believe that Twitter started back in 2006- that’s 7 years ago from the time I write this article. In the early days, it was very basic and despite early popularity- it was mainly popular amongst geeks, developers and other tech people. I signed up back in November 2006 despite not really understanding what the fuss was about. Well these days I think we’d be hard pushed to find someone who hasn’t heard of Twitter with 554 million users worldwide, 115 million active users every month, 135,000 new sign ups every day and 58 million tweets per day.
How did Twitter become so popular? I think a big part of it’s growth has been due to the developer community coming up with ideas on how to use Twitter and building apps for it. Not long after Twitter opened it’s doors, they released a system that allowed developers to access Twitter’s data including tweets and user information. That system was Twitter’s API- Application Programming Interface. An API helps a developer interact with an application- sending and receiving information. In the early days it was incredibly easy to access your data- it was just a case of getting a feed of your tweets or mentions which were available as RSS or XML feeds. You didn’t need to be a programmer, because you could add that feed to another app such as an RSS feed reader, Yahoo Pipes or a few years later, IFTTT.
Twitter Wants to Make Money
Twitter is free for end users, but of course it has to make money somehow. That somehow of course is advertising, but how could Twitter properly develop its advertising platform when it didn’t have control of the way its data was being presented? Twitter started buying up Twitter clients left, right and centre- TweetDeck being the most famous. They also started to lock down their API, meaning it was no longer possible for just anyone to access the data- you needed to be logged in or authenticated- and that raised the bar of access.
I highly recommend listening to a snippet from an interview between Emily Bell and the CEO of Twitter, Dick Costolo. He explains the thinking behind the changes in the API around 20 minutes in:
[soundcloud url=”http://api.soundcloud.com/tracks/60701848″ params=”color=ff6600&auto_play=false&show_artwork=true” width=” 100%” height=”166″ iframe=”true” /]
RSS & Authentication – Apparently, RSS Feeds “are infrequently used today”
When the latest version of the Twitter API (version 1.1) was launched, they announced that they were retiring RSS feeds. According to Twitter, XML, Atom, and RSS are infrequently used today. I completely disagree with that. I think the real reason why Twitter have retired RSS feeds was mainly due to the fact that they were not able to control access to their API. With the new version of the API everyone has limits on how many calls they can make every 15 minutes with your authenticated app. With RSS, because you aren’t authenticated, there is no limit. I appreciate Twitter has to keep a lid on their API- as allowing anyone to access their API with unlimited calls would be impossible.
However it is sad that the only way to access your Twitter data in the way it used to be is now firmly in the hands of developers. If you don’t know how to code and build an app that connects to the API you are out of luck,
Twitter and the Cloud- Some Confusion
With version 1.1 of Twitter’s API there were new “Rules of the Road- Twitter’s terms and conditions of use. Most of the terms make sense- and it is important for Twitter to make it clear how their data should be used. However there is one line in particular that has caused a lot of confusion in 1.4.a.iii:
You may provide spreadsheet or PDF files or other export functionality via non-programmatic means, such as using a “save as” button, for up to 100,000 public Tweets and/or User Objects per user per day. Exporting Twitter Content to a datastore as a service or other cloud based service, however, is not permitted.
From the Twitter API Rules of the Road
So, we are all allowed to copy up to 100,000 public tweets every day as long as it is not via an app (I am assuming that is what it means by “via non-programmatic means”). However what does “Exporting Twitter Content to a datastore as a service or other cloud based service” actually mean? It’s very vaugue! Does that mean we aren’t allowed to archive our tweets? That can’t be true, since Twitter offer their own archiving service where we can download all our tweets to our computers. There is no doubt, however, that it was this clause which caused the app IFTTT to remove their Twitter Triggers
RIP IFTTT Twitter Triggers
If you’re a regular reader of my blog, you’ll already know that I am a huge fan of the service IFTTT (If This Then That). IFTTT allows you to connect lots of different apps and services together (Twitter, LinkedIn, Flickr, Instagram, Facebook and even switches at home). A “Trigger” was something that would cause IFTTT to do something. For example “If you post a new Flickr Photo” would be a “Trigger”. An “Action” is something that IFTTT does as a result of an IFTTT Trigger. For example “…then post a Tweet”. A full example of a Trigger and an Action could be “IF you post a new Flickr Photo THEN send a Tweet”
In September 2012, IFTTT made an announcement that they they were removing Twitter Triggers. They were finally removed on September 27 and there were a lot of upset people. The reason for this, according to IFTTT was that “Twitter announced policy changes that will affect how applications and users like yourself can interact with Twitter’s data.”
I later wrote an article on how you could Get Your Twitter Triggers back using RSS. This worked by using the RSS feed for your Twitter timeline as an action in IFTTT. Unfortunately this now does not work.
We knew it was coming, but in June 2013, Twitter retired version 1 of their API together with the retirement of RSS, XML and Atom feeds. Now, once you’ve built an app that authenticates with the Twitter API, you will be given the data in JSON format. Whilst JSON is great for developers in that it is rich in data, it is not so helpful for everyone else. With so many people wanting to get their RSS feeds back, I decided to build my own solution.
Twools – The Solution to a Problem
Twools is my attempt at bringing back RSS feeds and so freeing your data. I have tried my best to make this as easy as possible for everyone to use. That’s quite a task, because Twitter don’t make it easy!
You install Twools on your own website. The advantage of this is that you control the way it works- and it is not very difficult to do. You just download Twools, add your Twitter App information and upload to your website. It’s that easy. Of course, you have to have a website- and it has to run PHP. That may be a deal breaker for you. However there are a huge number of websites that use PHP- WordPress is an axample- so hopefully you’ll be fine.
What Twools Can Do
You can use Twools to back up your tweets, or automate certain tasks. You could use in conjunction with IFTTT or another app to do some cool things. For example you could
- Cross post to LinkedIn using the #li hashtag
- Bookmark or cross post your Twitter Favorites
- Add new followers from a certain country or location to a Twitter list
- Get a text message when a company tweets a special offer
- Add articles that you tweet using a certain app to evernote.
- Get emailed when someone mentions your brand negatively.
- Get emailed when someone links to an article on your website even if they don’t mention you.
- Add someone mentioning a keyword to a Twitter list
What Twools Gives You
At its core, Twools gives you back your RSS feeds. However, it gives you many more RSS feeds than Twitter ever gave you. In particular, Twools gives you.
- Your home timeline
- User timeline (yours or another user’s)
- Your mentions
- Your favorites
- List timeline (from a list that you own)
- Direct messages received (disabled by default)
- Direct messages Sent (disabled by default)
- Twitter Search
- New followers
- New friends
You can then run filters on these feeds, to only return certain results. These are very powerful. For example:
Filters on Tweets
- Containing any keyword, #hashtag or @screen_name: You can filter by a single keyword, hashtag or screen name in this field. For example if you wanted to return only tweets containing the hashtag #socialmedia, you would enter that here. Or you could put a screen name (for example @iagdotme) to only return tweets that mention that user.
- Containing all keywords, #hashtags or @screen_names: You can filter by multiple keywords, hashtags or screen names in this field. This is similar to the previous field except that you can add multiple keywords, hashtags or screen names by separating them by a comma. For example you could enter “#socialmedia, @iagdotme” to only return tweets containing the hashtag #socialmedia and mentioning @iagdotme
- Containing none keywords, #hashtags or @screen_names: You can filter by multiple keywords, hashtags or screen names in this field. This is similar to the previous field except only tweets that DO NOT contain any of these keywords, hashtags or screen names will be displayed.
- Containing single #hashtag: Identical to the first field except that this specific hashtag can be removed from the feed output if selected in the final option.
- From: Only tweets from these users will be returned
- Not From: Only tweets not from these users will be returned
- Using Apps: You can filter by the app used to send the tweet. You can select multiple app names by separating by a comma. For example, to return tweets posted with either Hootsuite or Buffer you could enter “Hootsuite, Buffer” here.
- Not Using Apps: Similar to the previous field but here you can return tweets that were posted not using these apps. You can select multiple app names by separating by a comma. For example, to return tweets that were NOT posted using either Hootsuite or Buffer you could enter “Hootsuite, Buffer” here.
- Exclude Scheduling Apps: By checking this box you can exclude a variety of scheduling apps (such as Buffer, IFTTT, Social Oomph and Triberr) from the results. You can edit the list by editing the config file (see later).
- Exclude RTs: By checking this box you can exclude RTs from the results. This uses Twitters own API to only return non retweets, but also searches for RT in the tweets.
- Been Favorited: Only return tweets that have been favorited.
- Been Retweeted: Only return tweets that have been retweeted.
Filters on Tweets
- In Location: You can filter by the location of the person who tweeted the tweet. This filter looks at the location string the users’ profile.
- In Bio: You can filter a keyword from the user’s profile.
- Writes in Language: You can filter by the language that person who tweeted the tweet writes in.
Once you’ve returned the results you want, you can then do some final magic on the results. We can manipulate the output to remove hashtags, screen names or links or just give us the screen name instead of the tweet. Instead of linking to the tweet itself we could link to the URL in the tweet- and unshorten it so that we have the full URL.
There are so many possibilities, that it might take some time to get your head round it all.
Share Twools Recipes
I’ve built in a sharing facility into Twools. This way we can all share our ideas of feed recipes with each other. Basically all this does is to share the query string that Twools uses to create the feed, but it should make it easier to share and use recipes.
Here are some examples:
1. Archiving: Articles I post on Twitter using Triberr (or other app)
Triberr is a service for Bloggers to share your content with other bloggers. You basically share articles to your social networks from people within your Tribe. Triberr doesn’t offer an RSS service, so I use this to add all the articles I share on Twitter using Triberr to Evernote or Pocket. The articles I share on Triberr are ones that I think are awesome, so I do want to bookmark them for later.
This Twools recipe retrieves your tweets posted via Triberr. It removes URLs & screen names from the tweets. As well as this, it removes hashtags from the end (and removing ‘#’ from the rest). It extracts the URL and unshortens it and puts it in link field.
2. Community Building: Return anyone who uses the hashtag #socialmedia
You could use this feed with IFTTT to automatically add someone who uses this hashtag to a list.
3. Cross Posting: Tweets from your timeline that use the #li hashtag
Could be used to cross post to LinkedIn when using the #li hashtag
4. Archiving & Cross Posting: Your Twitter favorites
All URLs, screen names and hashtags have been removed. URL extracted and de-shortified. Could be used for Bookmarking or cross post your Twitter Faves
5. Community Building: Recent followers from the UK
Retrieves the screen names of all your recent followers with the keyword “UK” in their location. You could use this to automatically add your latest followers from the UK to a Twitter list using IFTTT.
6. Monitoring: Latest people mentioning Startbucks in the UK negatively
Retrieves the latest tweets from people who have the keyword “UK” in their location who mention Starbucks with negative sentiment. You could use this with IFTTT to be sent an SMS or an email whenever someone tweets about your brand in a negative way.
Streams – Super Simple Social Media Management Tool with Powerful Filters
RSS feeds are powerful, but sometimes you might want to have a look at the results visually in the same way you can on Twitter, TweetDeck or Hootsuite. Streams gives you this, kind of. It’s nothing fancy- and it isn’t supposed to be. It gives you a stream of the tweets with the same filters you get with the feed generator. You can then reply, favorite or retweet any of these retweets.
I use this myself quite a lot, because I find a lot of social media management tools a little lacking in filters. For example, with Twools Streams I can show the tweets from people in one of my Twitter lists excluding retweets and ones posted using a scheduling app. I also find it useful for doing a Twitter search of those people linking to an article on my blog excluding myself and ones mentioning me (since I’ll already see these in my mentions stream).
Keep an eye on your API usage
Twitter have locked down the number of calls you can make to their API. You have a set number of calls every 15 minutes. So to keep on top of your usage, I’ve built an API limits page in Twools which shows your limits and how many calls you have left in your current 15 minute window.
How do I install Twools?
If you use WordPress I highly recommend using the WordPress plugin version of Twools. With the plugin all you need to do is install it and fill in the details. See the Twools page for more details.
First of all you need to download the latest version. I’m not charging for it, but I ask that you fill in the form at the top of this page. I’d also love it if you could give me some feedback.
Twools will run on your own website. It is written in PHP so your website must have PHP installed. You don’t need to be a developer or a coder, but you do need to know how to upload files to a directory on your website. You could use an FTP programme, or perhaps you have another way.
Firstly, you will need to get Twitter access tokens. I’ve written an easy to understand article called “How to Create a Twitter App in 8 Easy Steps“.
Then you also have to edit a few lines in the config file. Please don’t be scared- it is very easy. Just open a text editor and open the config.php file up (it is in the /incs/ folder).
Firstly, you will need to set the username and password you want to secure Twools (replace “demo” and “demo” with your new username and password):
// Set username and password to protect access to Twools (RSS feeds will still not be password protected). // Put some jibberish on the padding string for extra protection $config['username'] = "demo"; $config['password'] = "demo"; $padding = "g`0*Y;6VOx`AIb+WBE>uKU1w;_cV-`&Y[ZgUDeZ3~jmdx]u7DGKUVN1W9~~W;kbk";
Once you’ve done that, enter those access tokens here:
/** Set access tokens here - see: https://dev.twitter.com/apps/ **/ $settings = array( 'oauth_access_token' => "", 'oauth_access_token_secret' => "", 'consumer_key' => "", 'consumer_secret' => "" );
As well as this, you should enter your Twitter username by entering it within the quotes for each item:
// Replace by setting the default Twitter username (exclude the @ symbol) $defaultUser = "iagdotme";
Once you’ve done this, you need to upload everything and run from your website. That’s it! There are other features you can activate such as the experimental spam filter and switch on direct message feeds. I go into more detail in the Twools documentation.
To run Twools, visit the folder on your website where you installed it. For example if you put it in a folder called “twools” and your website is example.com, you will need to visit http://example.com/twools/. Once you’ve done that, enter your username and password and you should be in.
A huge thanks to James Mallison for explaining how to build a simple Twitter app using v1.1 of the API and for building a fantastic PHP Twitter wrapper. This little app wouldn’t work without it!
I hope you find Twools helpful. Let me know how you get on and how we can improve it. I have lots of interesting ideas for the future. Also, share your recipes. I’ll be writing up a few more articles which will include some of them soon!
highly unusable website to be honest. There’s an annoying “leave me a voice message” modal frame that you need to click off to be able to read anything here, and you can’t simply scroll the documentation with the mouse.
Hi John, thanks for your comments. I agree with you about the modal frame for the voicemail app. To be honest I had forgotten about it because I never see it (it only appears once for new visitors). I’ve switched off the modal so hopefully that will make things more usable. I don’t know what the problem is with scrolling with the mouse. Are you wanting to scroll with the middle wheel? It certainly works for me. Let me know and I’ll look into it.
Even closer to launching Twools for WordPress. Here are some piccies…
Just to let you know, I have created a Twools Community on Google+. If you would like to join, here is the url – https://plus.google.com/communities/110217878433862480157
I appreciate the research and work you have put into this (I was another one who was dismayed to find that my Twitter feeds had suddenly stopped working one day). I’ll put in another vote for a WordPress plugin, although I have to say my primary response to this arrogant move by Twitter is to simply stop using their service altogether.
Thanks, Michael. I totally understand your response. There was a very interesting post on death of open APIs by Jeremy Keith- http://adactio.com/journal/6291/
As for ditching Twitter, I wish it was that easy. I am a big fan of app.net which is a Twitter look alike that is developer friendly, but the truth is, most of my audience or potential audience are on Twitter.
I am not that far from releasing the WordPress plugin. It’s still a bit rough round the edges, but I do have it working. You can add your Twitter details via a form in the WordPress dashboard settings. There are just a few bugs I need to iron out.
Looks really slick and useful.. This just shows how having good dev skills is as useful as being able to blog or do marketing in the modern world of information work.
I personally don’t use RSS, but I know enough people who do… congrats on this release!
Ian, still trying to get my head around Twools. I would be interested in a WordPress version. I might be able to install Twools but have been scared to add bits of code to my Home Directory as it is. Jeremy
Hi Jeremy. I know where you’re coming from. I think it takes a bit to get your head round, but my plan is to make this easier to use. Editing the config.php file isn’t as scary as you’d think, but I know it has put a few people off. When it comes to the WordPress plugin, it will be a lot easier. First, you won’t need to FTP it up- you’ll just install as a plugin. I could make it possible for people to edit the config.php file themselves in the WordPress editor- that would be the easy option… however I think it would be a lot easier to have a “Twools Menu” within WordPress where people can enter their details. I just need to get my head round creating an admin menu within WordPress. I think it is a case of “watch this space”.
Of course, since you’re local, I’d be happy to help you install Twools.
Hi Jeremy. Just released the WordPress plugin version!
Ian, well done! Does that mean it’s “inside” WordPress as a plugin on the index?
Hi Jeremy. It’s not in the WordPress index yet. You download it and then upload the plugin within WordPress. Eventually I’ll add it to WordPress but not quite ready for that yet.
You killed it Ian! Thanks for letting me be a part of the beta testing team and providing this to the community!
Thanks, for the encouragement, Nick. Hope it will be helpful to a lot of people!
For getting an RSS feed for twitter user timelines you can also try http://rss4twitter.appspot.com – it works fine – though there are some issues with downtime etc.. but seems to do the job
Thanks. When I started working on Twools I wasn’t aware of any other services, but it seems that other people shared my frustrations with Twitter stopping RSS feeds. I wasn’t familiar with RSS4Twitter. It looks useful. The only thing I would say would be that since Twools installs on your own website you have to worry less about downtime. Also Twools has filters and output manipulation which I find really useful.
Whatever, I am just pleased I am not the only one who thinks RSS is a great technology! 🙂
Ian, this look amazing.
Im a internet power user but your work might getting me into the follow the rules kinda “developper”. Big improvement 🙂
Im a fan of the internet working for me and twools looks just like that! Cheers for that!
I’ll give you some feedback. I do recognize some the acronyms involved (PHP, FTP) in the process, i do not fear of deleting and change some code, but somehaow i feel i need to ask someone more tech savy than me to help me. I guess what makes more noise is the “having your own server with php”. Thats a deal breaker for me cause well, i dont, and quite frankly y do not know how to do it.
The WP would be a solution? but what about the API limits on querry?
Anyways, you should be proud!
Cheers from South America!!
Twools currently does have an entry level. You do need to have a website and it does need to have PHP. Having said that you don’t really need to know what PHP is- it’s just a case of adding a few bits to a file. However, I understand that will put a few people off.
I think that is where the WordPress plugin version will come in. I hope to have a form in the admin area where you will be able to fill in all the details. That way no one will need to edit any lines of code. Also they won’t have to FTP up the files- all they will need to know how to do is install a plugin.
Your question about API limits per query- not sure I understand. As I said in the article, we all have a set number of queries every 15 minutes. That is for the app you have registered through Twitter. When you install Twools, you can see how many calls you have left. That will be the same on the WordPress version too- since it will be the same app.
OMG Ian, this is what Twitter should have provided us all with in the first place! You’ve gone way beyond what I had expected, I would have been happy if you had simply fixed the old Twitter RSS code even though I know that wasn’t possible. You seem to have taken the closure of the API as a personal mission and I think you won! You sir, are awesome. I give you a few months before some web start-up asks to buy this – I hope people are listening!!!
Question: I see where we can set the Twools username and password, and even set our Twitter username, but where do we authenticate to our Twitter accounts, where is that password stored (if anywhere), and how are you handling the encryption of the password?
Thanks so much! I think you’re right, I did see the retirement of v1 of the API and RSS feeds as the start of a personal mission. I hope you find Twools helpful! 🙂
The Twools username and password is just to protect anyone from accessing Twools on your website. Just change that to a username and password that you can remember. The login script is nothing fancy and not super-secure.
To authenticate it to your Twitter account, you need to register a Twitter app and get your access tokens as I say in the “How do I install Twools section?” above. It’s really easy to do, and I’ve written an article going through the steps- https://iag.me/socialmedia/how-to-create-a-twitter-app-in-8-easy-steps/
Once you’ve got your Consumer Key, Consumer Secret, OAuth Access Token and OAuth Access Token Secret you need to put them in the config script too.
/** Set access tokens here – see: https://dev.twitter.com/apps/ **/
$settings = array(
‘oauth_access_token’ => “”,
‘oauth_access_token_secret’ => “”,
‘consumer_key’ => “”,
‘consumer_secret’ => “”
Note, that Twools doesn’t need to store your Twitter password- it just uses the Twitter app to connect to your Twitter account. You’ll see your app listed with other ones you’ve given permission to here – https://twitter.com/settings/applications
Since Twools is installed on your website- only it can access your details- so it doesn’t have to be stored anywhere else.
Does that answer your questions?
Yes, perfectly. In fact I read that right after posting my now stupid question 🙂 This is great, installing now! Thanks again for all your hard work on this.
No problem. Don’t worry- there is a lot to read! Do let me know how you get on and if you come up with any cool recipes!
Kuwel! My mind just exploded. (No PHP though.)
Ha ha! You’re not the only one. I think my mind has exploded quite a few times over the past few weeks. Hopefully at some point I can make this into a WordPress plugin or at least make it a little easier for everyone to use. As I said in my article, I think it is a shame that you have to be a developer to have access to your Twitter data.
It looks like you’ve put a lot of work into an amazing tool Ian. Any chance that it will be a WP plugin someday? You’ll lose a lot of people with the installation process and there’s always the security questions.
Interesting one. It could be turned into a WP plugin, but it would take some work. I know what you mean about the installation process. I have tried to make it as easy as possible, but it still involves FTPing up to your website and changing a few lines of code. Some people may be frightened off that. Is this something you would feel comfortable doing yourself?
What did you mean about security questions? What specifically?
Actually, Brian. Developing a WordPress version is a brilliant idea, and having thought about it, not too difficult to do. I am going to work on it. Would that be more helpful for you?
Hi Brian, I’ve just released a WordPress plugin version of Twools. I have you to thank for making me think about doing one. Thanks!