help with streaming filter?

Jan 2, 2015 at 2:20 PM
Edited Jan 7, 2015 at 12:35 PM
/
Coordinator
Jan 5, 2015 at 2:15 PM
Hi,

The first step is to read the code and do a search for the parts you don't understand. In this case:
  1. Notice that the switch statement calls DoFilterStreamAsyc on case 0.
  2. Inside of DoFilterStream, you see CancellationTokenSource, which is part of the .NET cancellation framework.
  3. Do a search for CancellationTokenSource so you can understand what that code does.
  4. Inside of DoFilterStream there is a query that starts the stream.
  5. Click the Documentation tab above, click on Making API Calls, Streaming Twitter Content, Getting Filtered Statuses and observe the documentation explaining what filters you can use in your LINQ queries, with a sample that looks like the code you copy and pasted above.
  6. Observe at the bottom of the Getting Filtered Statuses page that there is a link to the equivalent Twitter documentation. Click that link and peruse the documentation to get a better idea of how Twitter streams work and some good advice on how to use streams effectively.
  7. Notice in the DoFilterStream method that the query uses a StartAsync operator and it's parameter is a lambda. That lambda processes every tweet returned from the stream.
  8. Notice the code inside of the StartAsync lambda that checks the count. After 5 tweets, that line cancels the stream. You will understand why if you reviewed how .NET cancellation works as I indicated in #2 above.
In addition to looking for available documentation and using a search engine, another tip to understanding code is to use the Visual Studio debugger, set breakpoints, and step through the code to watch what it does.

@JoeMayo
Jan 5, 2015 at 2:54 PM
Edited Jan 8, 2015 at 8:36 AM
/
Coordinator
Jan 5, 2015 at 6:04 PM
I think Firehose requires Twitter permission, beyond normal credentials and you'll have to visit their site and request Firehose access. I have a FAQ that includes a section to help with 401 errors:

https://linqtotwitter.codeplex.com/wikipage?title=LINQ%20to%20Twitter%20FAQ&referringTitle=Documentation

@JoeMayo
Jan 6, 2015 at 8:42 AM
Edited Jan 8, 2015 at 8:36 AM
/
Jan 6, 2015 at 8:58 AM
Edited Jan 7, 2015 at 12:35 PM
/
Jan 6, 2015 at 8:59 AM
Edited Jan 7, 2015 at 12:35 PM
/
Jan 6, 2015 at 9:29 AM
Edited Jan 8, 2015 at 8:37 AM
/
Coordinator
Jan 7, 2015 at 6:10 AM
There's an answer for that in the FAQ:

https://linqtotwitter.codeplex.com/wikipage?title=LINQ%20to%20Twitter%20FAQ&referringTitle=Documentation

There's also documentation on security and OAuth Authorizers:

https://linqtotwitter.codeplex.com/wikipage?title=Securing%20Your%20Applications&referringTitle=Documentation

If there's only ever one person using the stream, the SingleUserAuthorizer is probably easiest. You should probably check out the list of authorizers in the docs to see which would be better for the technology you're using. There are demos for each authorizer in the downloadable source code, with a handful in use in the Console demo.

@JoeMayo
Jan 7, 2015 at 9:12 AM
Edited Jan 7, 2015 at 12:36 PM
How do I implement the single user auth with the webforms demo? I plan to stream rather than using the built in features so ill have to add streaming in.
Coordinator
Jan 7, 2015 at 6:14 PM
Edited Jan 7, 2015 at 6:15 PM
I think you'll have trouble with a design that uses streams in a Web app (WebForms or MVC). The reason why is that a stream is intended to be a resource you connect to and stay connected for a long time (reconnecting only as necessary). That's really it's benefit in being a very efficient way to receive real-time tweets. Now, think about the nature of a stateless Web app. A WebForms page or MVC Action has a lifetime that only exists for a single request. Once the request completes, all the associated objects and resources they own goes away and are recreated on the next request. In this case, you would be creating a new stream instance on every request. If there was only a single person using the app infrequently, you might not see many problems. However, as more people use the app simultaneously, you'll quickly run into problems because every request is creating it's own new instance of a stream. The type of errors you'll see are timeouts because of the unpredictable amount of time it takes to initiate the stream connection and connection errors from Twitter. For streams, Twitter assumes the single long-running connection and if they see activity like this, they might block your app, resulting in 401 Unauthorized errors. All of these items surface through LINQ to Twitter as exceptions.

So, depending on your requirements, here are a couple general ideas that might help you think about how you design this:
  • Web Application: Use Search queries with Web or MVC authorizors.
  • Desktop or Server Application: Use Filter stream with Application Only Authorizer.
@JoeMayo
Jan 8, 2015 at 9:53 AM
You're saying its not possible to stream-filter via a web form, is this because of limitations in place by linqtotwitter or related to the twitter api not allowing it? Using search queries with web would work, however it has a rate limit which would severely limit the functions of the app. Ideally i want a stream within a web form, is this not possible at all? I have streams working using desktop app however I specifically need it on a web form for my project. Is there absolutely no way to implement it using linqtotwitter? or maybe another twitter api? I had a look and found site streams, i dont think thats supported by linqtotwitter though-its also in beta and requires an application to gain access.

thanks,
Coordinator
Jan 9, 2015 at 4:59 AM
Correct, it isn't going to work inside of an ASP.NET application. It isn't a limitation of LINQ to Twitter or Twitter API, but a reality of working with a stateless Web application, regardless of technology. You would encounter the same architectural constraints as any other service that was doing HTTP streaming.

The rate limits are imposed by Twitter API - LINQ to Twitter is just trying to make your task easier and doesn't have any limitations beyond what the Twitter API sets.

That said, there might be some clever work-arounds. Rather than just saying "It's Impossible" (an optimistic person would say that nothing is impossible), here are a couple ideas:
  • If you had a single Web page up without closing it down, you could use Javascript to communicate directly with the Twitter API.
  • Another option is to start a stream in a server process, like a Windows Service. Then use SignalR or some other Web Sockets implementation to send the data to your Web page.
  • Twitter recently acquired Gnip and you can use their service. You'll need to pay for it, but that's another option to avoid rate limits with the REST API.
You can visit the Twitter Developer's Site to learn about their Terms of Service and limitations.

@JoeMayo