Streaming API 25% CPU in Windows Service and Console Host

Nov 5, 2013 at 4:16 PM
Edited Nov 5, 2013 at 4:35 PM
I have written a windows service (and a companion console app that hosts the service class to make debugging easier) and both seem heavy in the CPU utilization department (always between 25 and 30 %).

Should this be considered normal or is there something I can do to help this along? I'm not doing much different than the streaming sample code (I did build in the recommended twitter connection strategy).

The only odd thing I have noticed is that when running on the server, it is running in 32bit mode, even though the app was compiled as AnyCpu. There is another service project in the same solution, running on the same server, also compiled for AnyCpu which runs in 64bit just fine. The only real difference in this service is that it references LinqToTwitter and Newtonsoft.Json - but I'm fairly certain both of these are native? I can also force the app to compile to 64bit and it runs just fine in Visual Studio, so I don't get quite what's going on there...

I don't think 32 / 64 bit is related since if I comment out the Streaming API call the app does back to essentially idle, but I figured I'd mention it.

Anyway, apart from that possibility, is there anything LinqToTwitter-specific that i can do to make it more efficient? This is basically the code that runs, not much to it (I tried setting compression to both true and false and the results were not appreciably different)
var twitterCtx = GetTwitterContext();

twitterCtx.AuthorizedClient.UseCompression = false;

(from strm in twitterCtx.Streaming where strm.Type == StreamingType.Filter &&
    //strm.Language == "fr,jp,en" &&
    //strm.Track == "[!#:"//,JoeMayo,linq2twitter,microsoft,google,facebook"
    strm.Track == HASH_TAG select strm)
    .StreamingCallback(strm = > {

        if (strm.Status != TwitterErrorStatus.Success) {
            //...twitter connection strategy code....

        try {
            if (!string.IsNullOrEmpty(strm.Content) && !strm.Content.StartsWith("{\"disconnect\":")) {
                var tweet = JsonConvert.DeserializeObject < StreamingApiStatus > (strm.Content);

                //do stuff with the tweet

                lastDataRecieved = DateTime.Now; //checked below to reset the connection if no data received in a while
        } catch (Exception) {
            Console.WriteLine("Could not parse twitter response!");

Edit: Here's my connection code, not much special going on here either I think, but I thought I'd include it:
private TwitterContext GetTwitterContext()


        var auth = new SingleUserAuthorizer
            Credentials = new SingleUserInMemoryCredentials
                ConsumerKey =
                ConsumerSecret =
                TwitterAccessToken =
                TwitterAccessTokenSecret =

        return new TwitterContext(auth);
    catch (Exception ex)
        return null;

Nov 5, 2013 at 5:14 PM

Not sure what's happening with the 32/64-bit issue. I'm not surprised that you aren't seeing much CPU difference with compression because there's a tradeoff where you're processing more bytes uncompressed, but you have the overhead of compression when it's turned on. The gain that compression gives you is more network efficiency in using less bandwidth, reducing latency, and improving speed of transmission - so I would benchmark network performance when considering compression. I'm not sure of what you're doing in ProcessTweet(), but I normally use a message queue. The queue will let you minimize the amount of time you're handling message during the callback, so the thread can get back to listening for the next message on the stream. You can have another thread on the other side of the queue, pulling messages off and processing, which will help scale your application. Most of my time has been spent adding new functionality and maintenance, but I'm beginning to get more queries like this about performance. Pretty interesting.

Nov 5, 2013 at 5:38 PM
Hi Joe,

As always, thank you for the prompt response. The CPU is the same if there is tweet activity or not, and right now it's essentially "idle" most of the time and just listening, so it seems to be just the "base" processing going on?

By the way, I am still slightly amazed at how fast the entire process works. It's near instant that a tweet shows up in the callback after having entered it on the twitter site!

Also, right on about the queue and worker thread for processing tweets too. I had planned on doing that for this version but couldn't quite fit it in. It's pretty low activity right now, but in the future if we need to (let's hope so) I can add in the queue to keep the listener responsive.
Nov 5, 2013 at 6:16 PM
That's a Read on the HttpWebResponse stream. There's also some thread synchronization being managed by ManualResetEvent. In the next version of LINQ to Twitter, I'm using async with HttpClient and it's possible that there will be a big difference - perhaps in not blocking threads.
Marked as answer by FirstDivision on 11/5/2013 at 11:32 AM
Nov 5, 2013 at 6:33 PM
Sounds good to me. I'll keep an eye out for the next version.

Thanks again!