Showing posts with label Tutorial. Show all posts
Showing posts with label Tutorial. Show all posts

Tuesday, 7 June 2016

Tools to scrape Instagram without any coding

Well I did the one list for Twitter a couple of days ago so this is a short list of tools you can use to extract data or pictures from Instagram. This one is a bit tricky. As of June 1 the API changed and some of the features may not work anymore. I quickly tried some of the tools I have laying around so I will probably add some more in the future.

odrive - Mac/Win/Lin - LINK
This used to be my go to tool when I needed photos with a certain hashtag, but now it is only possible to download the instgram pictures of your own and the once you liked. Still a nifty application as it can hook up loads of platforms to extract some data of it and can function as some kind of Dropbox.


4K Stogram - Mac/Win/Lin - LINK
Still works downloading Instagram photos with a certain hashtag or location (the last one did not work when I tried), but most importantly you do not have to log in to your own instagram account. Downloads all photos and videos to your pictures folder. There should be a handy way to also download the metadata, but have not dived too deep into this software.

Hotfolder Prints - Win - LINK
Hotfolders Prints comes with an Instagram Hashtag Monitor that takes some time (according to your settings) to download the photos with a certain hashtag. Most importantly the program also downloads the metadata of every picture into separate xml files. If you are handy with excel or other programs you can automatically import the data as a spreadsheet. The software is usable for a 15 day trial and is a bit pricy at 175 dollar.


Still have some other tools, but need to test them with the changed API, but the tools described can give you a start on scraping Instagram data/photos. 



Wednesday, 1 June 2016

Tools to scrape Twitter without any coding

On request I relist some tools that can help anyone to extract data from Twitter without any knowledge of coding. Did I miss a tool? Let me know in the comments and I will add them.


TAGS6.0 - Any OS - LINK
One of my favorite tools to scrape tweets from an account or multiple keyword search. It is a Google Sheets add-on that can update hourly so you don’t have your computer running during the sessions. I have one sheet already scraping for over a year for one keyword related to one of the companies I sometimes work.





My Twitter Scraper - Windows - LINK
A free and open source Twitter Scraper built by Software Engineering student Jason Dixon with the capability to scrape Twitter realtime. It returns username, tweet, time and location in a CSV file. Make sure you have Java 1.8 installed and get your Twitter API OAuth tokens at https://apps.twitter.com by creating a new Twitter application.




NodeXL - Windows - LINK
While a lot of people seem to have trouble with NodeXL to scrape tweets I had some success with it some years ago. NodeXL is an extension for Excel 2007 or higher (I recommend to just use Excel 2007). We probably need to wait for a fix, but sure one to keep in the loop.  


Nayoun & Gephi - Windows - LINK
Not easy to install, but you can use Nayoun & Gephi (as I describe in my other post) to live visualize different aspects of Twitter. You can export the data to a csv file in Gephi.





IFTT - Any OS - LINK
Not for huge volumes of tweets, but you can use some recipes to send certain tweets (like your own or hashtags) to a Google Sheets. Not that almost all recipes are limited on the amount of tweets they can extract at a time.


Monday, 26 October 2015

Naoyun and Gephi

One way to scrape and visualize Twitter Interactions is using Naoyun (http://matthieu-totet.fr/Koumin/tools/naoyun/), a small piece of software created by Matthieu Totet, and Gephi with the Streaming Graph plugin. A similar tool has, for example, been used to show the interactions during the Egyptian Revolution (https://www.youtube.com/watch?v=2guKJfvq4uI) over time. I personally use it to scrape data and visualize live events using some keywords or usernames. For instance the video below that shows one hour of interactions searching for #TDF (Tour de France) last year. It is a great way to identify conversations and key players on a certain subject.





To install Naoyun you have to download it at http://matthieu-totet.fr/Koumin/tools/naoyun/ and install Gephi that can be downloaded at: http://gephi.github.io/ The video below shows how to connect both programs and start live streaming Twitter interactions. The data collected consists of username, media, hashtag, link and status. Exporting is possible at the Data Labory in Gephi.




If you are having problems with running the Naoyun batch file follow this link to activate java in cmd:http://www.youtube.com/watch?v=X05JPr... (is about minecraft but applies)