Streaming API Data

Jun 9, 2013 at 8:17 PM
How do I garb the Twitter Streaming API data and save it. I mean I can see the streming API data using console app. But how do I grab it, process, it and save it in database. Is there any sample app that does that or can you guide me how to do it?
Jun 9, 2013 at 9:08 PM

Here are the docs, which has a sample: You can also download the source code and view an example in the LinqToTwitterDemo project.

That data returns as a Json formatted string. This is an area that I haven't gotten to yet, so you'll have to write code to parse the Json and process as you need. I don't have a demo for that, but there is a LinqToTwitter Stream Parser project on CodePlex and a NuGet download that might give you some ideas. There are also some discussions in this forum on the subject that you can search for that might provide some info - I remember a discussion not so long ago where someone had written some code for this.

Jun 10, 2013 at 6:02 AM
Can I implement Streaming API using Web Service or web application.

Jun 10, 2013 at 1:25 PM
One more thing, I have designed a console application-built with the sample code provided by you for Streaming API (status/filter). The only change I have made in the code is removing the counter so that the console app keeps on fetching tweets until I press a key. Now the issue, I am fetching is, after few mins the tweets stop coming and app generates an error, Can you tell me If it it happens with streaming api- or the streaming api is supposed to fetch tweets until we stop the application. Are there any constraints we need to keep in mind?

Jun 10, 2013 at 3:55 PM
Streaming doesn't work with Web Service or Web Application because they're stateless. e.g. you make a request, runtime instantiates new objects, streaming starts, the request completes, runtime cleans up objects (including instances of TwitterContext). If you want a continuous running stream, you'll need to run something like a Windows Service or a Cloud Service.

Jun 10, 2013 at 3:59 PM
Streaming API should run continuously. Twitter has good documentation on guidance and best practices for streams:

Jun 11, 2013 at 5:26 AM
And why do the streams stop coming after sometime?

Jun 11, 2013 at 2:31 PM
A few tips:
  • Be sure to keep your TwitterContext instance in scope.
  • Check the callback response to see if there is an error.
  • When the stream stops, close it and use the recommended back-off strategy to reconnect.
  • You must reconnect by creating a new instance of TwitterContext.
Jun 18, 2013 at 6:50 AM
Two questions:

1] What is recommended back-off strategy?
2] By new twitter context instance, do you mean I need to use different auth keys than what I had used in initial twitter context instance?
Jun 18, 2013 at 10:10 PM
On back-off strategies, it's best to review Twitter's streaming API docs (link above) since they have a lot of good information. They also tell you how to keep from getting your stream turned off. I tried to provide an automated back-off strategy with earlier versions, but this takes control out of your hands and is something I shouldn't decide for you.

Reuse the same auth keys, just create a new instance of TwitterContext. Sorry, I don't have more complex samples - just trying to keep up. :)

Jun 19, 2013 at 5:54 AM
Thanks Joe. Its helpful information. I think streaming API is best left more independent, x'fering more control to developer. Your guidance in the matter is very valuable.