StreamingCallback Extension Method

Nov 6, 2012 at 2:46 PM

Joe,

I am having some troubles trying to use the Streaming API. Let's say I have some code like the following:

ar twitterCtx = new TwitterContext(auth);
          //  twitterCtx.StreamingUserName = "test";
          //  twitterCtx.StreamingPassword = "test";
            var searchResponse1 = 
                (from tweet in twitterCtx.Streaming
                 where tweet.Type == StreamingType.Filter &&
                       tweet.Track == "hi"
                 select tweet)
                .StreamingCallback(strm =>
                    {
                        TweetResults.Add(new Tweet { Text = strm.Content });
                    }).SingleOrDefault();

Whatever I try won't work because it can't find the StreamingCallback extension method. Do I have to add another reference or something? Could you please just give me a gentle nudge in the right direction?

Regards,

Simon

Nov 6, 2012 at 4:01 PM

Joe,

I got it to compile by changing the code to the following. I don't know why it cant find the extension method.

            // the user stream is for whoever is authenticated
            // via the Authenticator passed to TwitterContext
            var userStream = (from strm in TwitterContext.UserStream
                              where strm.Type == UserStreamType.User
                              select strm);

            TwitterExtensions.StreamingCallback(userStream, strm =>
            {
                if (strm.Status == TwitterErrorStatus.RequestProcessingException)
                    return;

                string message = string.IsNullOrEmpty(strm.Content) ? "Keep-Alive" : strm.Content;
                Debug.WriteLine(message);

                if (count++ >= 25)
                    strm.CloseStream();
            }).SingleOrDefault();

This keeps returning empty strings and never seems to see new status updates for anyone the authenticated user is following. I tried the filtered status streaming sample and I was able to see updates. The UserStream method just doesn't seem to work. Am I missing something of great importance?

Regards,

Simon

Coordinator
Nov 6, 2012 at 11:54 PM

Hi Simon,

UserStream is sometimes a little challenging to get going at first, but once everything is in-place it seems to work well. Here are a few tips:

1. Use OAuth authentication. Here's the FAQ in case you need help: http://linqtotwitter.codeplex.com/wikipage?title=LINQ to Twitter FAQ.

2. Try getting OAuth working with a simple REST API query first. i.e. do a search. That will help iron out OAuth issues before moving on to the more complex task of streams.

3. Try to use the samples from my downloadable code first because they have been tested.

4. If you encounter a lot of errors when first trying to connect, wait a while because Twitter might temporarily block your IP (security measures).

5. Follow inner exceptions to pull out all of the error information.

6, Use Fiddler for debugging the HTTP responses from Twitter.

Joe

Nov 16, 2012 at 4:47 AM

Joe,

Sorry its taken me a while to put aside some time for this. I now have a working version using the Streaming API however I have a few questions:

  1. Sometimes a tweet never arrives at all via the streaming API. Is there any way to prevent this behaviour or is the solution to just use the Search API every noe and then to ensure you are up to date?
  2. Sometimes a tweet doesn't arrive and then when I tweet again, the callback fires and it is the previous tweet. Am I doing something incorrectly that might cause this to happen?
  3. Can the streaming callback disconnect? If so, can I detect this and re-connect the callback?

Thanks heaps,

Simon

Coordinator
Nov 16, 2012 at 1:14 PM

Hi Simon,

1. There could be a couple reasons why a tweet didn't arrive. i.e. the stream disconnected and LINQ to Twitter automatically reconnected in the meantime. Maybe Twitter didn't return the tweet because of how it's internal algorithm works. I think it might be okay to run a Search occasionally, especially if the stream disconnects and you need to pick up Tweets during that timeframe.

2.  I'm not sure what's happening here. It sounds like a buffer has been holding it and doesn't flush until more tweets come in. So, if the stream was returning a steady flow of tweets, you might not notice it. LINQ to Twitter doesn't cache tweets, so this behavior might be coming from Twitter.

3. The stream can disconnect unpredictably and LINQ to Twitter attempts to automatically reconnect. Except in the case where Twitter closes the stream, which you can detect. Check out this discussion, where I added some logic to detect when Twitter closes the connection:

http://linqtotwitter.codeplex.com/discussions/391715

What I don't have is a way to notify if LINQ to Twitter is attempting to reconnect. I'm thinking that an event might be useful - what are your thoughts about that?

@JoeMayo

Nov 17, 2012 at 12:54 AM

Joe,

I think I have may have figured the duplicate issue. My bad, its not actually a duplicate its a re-tweet by someone who I don't follow. The stream still fires and I see the re-tweet. In some cases it was just looking like a duplicate because the content was the same. If I am using the following code should I be seeing re-tweets from people I don't follow?

from strm in TwitterContext.UserStream
             where strm.Type == UserStreamType.User
             select strm)
                .StreamingCallback....

I wouldn't have thought that it should have fired in that case.

As for the reconnect event, that would be a great idea.

Regards,

Simon

Coordinator
Nov 17, 2012 at 1:32 AM

Thanks for letting me know - something to keep in the back of my mind the next time someone asks that question. According to the Twitter User Streams Page, you should receive all information about a User. Apparently, that includes retweets. I've seen DMs, Favorites, Retweets by the user, and various other items on the page. It might be a good idea to review the Twitter docs too because they expect stream consumers to listen for events like deletion messages or blocks and make sure that your app preserves that state.

I was thinking more about the eventing mechanism today because I opened an issue a while back to add more instrumentation to the Streaming API. If I do events, then that will open the potential for anyone to hook up their own preferred instrumentation library. i.e. capture events and write to Enterprise Library, Log4Net, NLog, or anything else.

I'm glad it's working for you.

@JoeMayo

Nov 27, 2012 at 11:57 PM

Joe,

A couple more questions:

  1. If I'm using the StreamingCallback with a UserStream and I want to switch to Streaming so I can get all tweets that match a query string it seems to work with a new TwitterContext but not if I attempt to re-use an existing one that already registered from a UserStream callback. As usual I'm probably doing something wrong but should this work?
  2. Why does all the sample code have a counter that calls CloseStream after about 25 tweets are received via the StreamingCallback? What's the purpose of this check? Wouldn't you want to keep the stream open indefinitely? Sorry if I'm missing the point...

Simon

Coordinator
Nov 28, 2012 at 1:38 PM

I'm interested in the scenario where you weren't able to reuse a TwitterContext. i.e. was there an error somewhere or why you disconnected. BTW, did you know you could create a new class derived from TextReader and assign it to the TwitterContext.Log property to get information from LINQ to Twitter queries?

You're right - normally you wouldn't close the stream like this. However, I wanted to show how to do it and didn't want it to run forever to test the stream. Because samples are for learning the mechanics of how something works, it's impractical for them to model the real world (exceptions, logging, performance, scalability...).

@JoeMayo

Nov 28, 2012 at 8:29 PM

Ill put together a smaller app to demo. Maybe in doing so I will realise I'm doing something else horribly wrong. ;-)

Ok great, I'm not missing the point then.

Thanks again Joe, LinqToTwitter has made development so much easier and I appreciate all of your tireless efforts...