Streaming API Data

Jun 9, 2013 at 8:17 PM
How do I garb the Twitter Streaming API data and save it. I mean I can see the streming API data using console app. But how do I grab it, process, it and save it in database. Is there any sample app that does that or can you guide me how to do it?
Coordinator
Jun 9, 2013 at 9:08 PM
Hi,

Here are the docs, which has a sample: https://linqtotwitter.codeplex.com/wikipage?title=Getting%20Filtered%20Statuses&referringTitle=Streaming%20Twitter%20Content. You can also download the source code and view an example in the LinqToTwitterDemo project.

That data returns as a Json formatted string. This is an area that I haven't gotten to yet, so you'll have to write code to parse the Json and process as you need. I don't have a demo for that, but there is a LinqToTwitter Stream Parser project on CodePlex and a NuGet download that might give you some ideas. There are also some discussions in this forum on the subject that you can search for that might provide some info - I remember a discussion not so long ago where someone had written some code for this.

@JoeMayo
Jun 10, 2013 at 6:02 AM
Can I implement Streaming API using Web Service or web application.


Jun 10, 2013 at 1:25 PM
One more thing, I have designed a console application-built with the sample code provided by you for Streaming API (status/filter). The only change I have made in the code is removing the counter so that the console app keeps on fetching tweets until I press a key. Now the issue, I am fetching is, after few mins the tweets stop coming and app generates an error, Can you tell me If it it happens with streaming api- or the streaming api is supposed to fetch tweets until we stop the application. Are there any constraints we need to keep in mind?


Coordinator
Jun 10, 2013 at 3:55 PM
Streaming doesn't work with Web Service or Web Application because they're stateless. e.g. you make a request, runtime instantiates new objects, streaming starts, the request completes, runtime cleans up objects (including instances of TwitterContext). If you want a continuous running stream, you'll need to run something like a Windows Service or a Cloud Service.

@JoeMayo
Coordinator
Jun 10, 2013 at 3:59 PM
Streaming API should run continuously. Twitter has good documentation on guidance and best practices for streams:

https://dev.twitter.com/docs/streaming-apis

@JoeMayo
Jun 11, 2013 at 5:26 AM
And why do the streams stop coming after sometime?


Coordinator
Jun 11, 2013 at 2:31 PM
A few tips:
  • Be sure to keep your TwitterContext instance in scope.
  • Check the callback response to see if there is an error.
  • When the stream stops, close it and use the recommended back-off strategy to reconnect.
  • You must reconnect by creating a new instance of TwitterContext.
@JoeMayo
Jun 18, 2013 at 6:50 AM
Two questions:

1] What is recommended back-off strategy?
2] By new twitter context instance, do you mean I need to use different auth keys than what I had used in initial twitter context instance?
Coordinator
Jun 18, 2013 at 10:10 PM
On back-off strategies, it's best to review Twitter's streaming API docs (link above) since they have a lot of good information. They also tell you how to keep from getting your stream turned off. I tried to provide an automated back-off strategy with earlier versions, but this takes control out of your hands and is something I shouldn't decide for you.

Reuse the same auth keys, just create a new instance of TwitterContext. Sorry, I don't have more complex samples - just trying to keep up. :)

@JoeMayo
Jun 19, 2013 at 5:54 AM
Thanks Joe. Its helpful information. I think streaming API is best left more independent, x'fering more control to developer. Your guidance in the matter is very valuable.