Newbie Question on Search

Nov 4, 2011 at 1:46 AM

Hi all,

I am really grateful for Linq to Twitter, it is really an outstanding implementation! I am new to the Twitter API in general, so I am trying to understand how to best collect *all* search results for a specific search term (or as many as possible) with C#. I have the following example right now to execute a search, which is working well:

var queryResultBase =
                from search in context.Search
                    search.Type == SearchType.Search
                    && search.Query == query
                    && search.ResultType == ResultType.Mixed
                select search;

So what I am trying to understand is, how do I ensure that I capture all results, from oldest to newest? I get using the MaxID and SinceID to move backwards and forwards, but ideally I would want to start with a set of the oldest results, then move forward by date until I was up to the most recent results. When I tested looping through the SinceID collection, I appear to get large gaps in my search results by day; some days look complete and others barely have any data.

I'm not sure if I am explaining this well, but any insight would be greatly appreciated. Thanks in advance!

Nov 4, 2011 at 10:35 PM


In addition to Since, SinceID, and MaxID, you also have Page and PageSize to help you work with results.  Here are all the available parameters:

I'm not sure about the consistency of the results, but I do know that Twitter will only return results for a certain period.  I'm not sure what that period of time really is and I don't recall seeing any definitive info from Twitter on how far back a search will go. 

For the consistency issue, you might want to try debugging by comparing results.  i.e. visit Twitter Search at!/search-home and enter the same query that you're using with LINQ to Twitter.  You can also use Fiddler, Ctrl+F12 tools, or Firebug to watch the HTTP traffic and see what is being sent and received.  LINQ to Twitter is designed to translate all results into an object and return the objects to you, but if you find a bug where results are not being returned, let me know.  Also, whatever filters you use in your query, LINQ to Twitter will use them in post processing the information that Twitter returns, meaning that it's possible that Twitter returns more results because it doesn't recognize the filter, but LINQ to Twitter post processing will still apply the filter before returning results to you.

Hope that gives you some ideas,


Nov 5, 2011 at 12:24 AM
Edited Nov 5, 2011 at 12:26 AM

It does help, thank you! So if I am understanding things correctly, I can only get up to 100 search results per request. So if I set PageSize to 100 and Page to 1, then I would be able to iterate the entire collection that is returned from the sample code above? The MaxID definitely works to step backwards in the timeline, and I have been testing the results with simple queries right now. It just seems odd that I have such big gaps in the timeline for results sometimes. I guess if I could tell the query to order by date that might do the trick... I will keep reading!

My next step would be to maybe utilize the stream capture... I will have to see what gives me the most success.

From what I have seen, the timeline appears to be about 9 days. I have never been able to pull anything older than that from a search. I think I read that 6-9 days is the standard somewhere.

Can't tell you enough how great this library is - really impressive!

Nov 6, 2011 at 2:27 AM

I finally get the paging, just took a little extra reading. The max search results you can get per query is 1500. Here's the quote from the API docs:

The page number (starting at 1) to return, up to a max of roughly 1500 results (based on rpp * page).

So when you are trying to get a "full" snapshot, the easiest way to do it is to set PageSize to 100 and then loop through each Page until you either a) get less than 100 records in your result collection or b) you get to page 15. Here's an example of what I did for this (apologies for the roughness, this is still just test code):

            var context = new TwitterContext();

            string query = "linq2twitter";
            int mstrCount = 0;
            int? pageID = 1;

            while (pageID != null)
                //get base set of tweets
                var result =
                from search in context.Search
                    search.Type == SearchType.Search
                    && search.Query == query
                    && search.PageSize == 100
                    && search.Page == pageID
                    && search.ResultType == ResultType.Mixed
                select search;

                foreach (var search in result)
                    foreach (var entry in search.Entries)
                        //process entry content here
                    if (search.Entries.Count < 100 || pageID >= 15)
                        pageID = null;

Hope this helps a few others...

Nov 6, 2011 at 2:38 AM

Excellent!  Thanks for sharing. :)

May 28, 2012 at 1:42 PM

Does it have to be done like how emptyfish showed? Using search.Single(), you could still get all the results at once and then iterate over the results like Joe pointed out - "LINQ to Twitter is designed to translate all results into an object and return the objects to you"

I could be missing something!?

BTW, something unrelated and this might seem like a newbie question! But, it seems I do not need authentication to use TwitterContext.Search. What are the benefits of authentication [I guess this means obtaining a oauth KEY from Twitter in this case]? 


May 28, 2012 at 7:30 PM


That isn't the only way to do it, but you do have constraints, such as a max 100 when specifying PageSize.  Here's a description of how to use MaxID and SinceID, which are supported by search queries, to page through data:

Search is one of the APIs that don't require authentication, so just instantiating a normal TwitterContext will be fine.  All of the pages in the LINQ to Twitter documentation include links to the Twitter API documentation that they match, which also specifies whether an API needs authentication or not..

The benefits of authentication, when required, are for the user.  If you were using the Twitter API to operate on behalf of a user, then they would be required to give your application permission to do so.  The LINQ to Twitter documentation has more information on the benefits of OAuth, how it work, and help for choosing the right strategy.


May 29, 2012 at 3:31 PM
Edited May 29, 2012 at 3:34 PM

Thank you Joe,

However, just to clarify - If I run a search query using the .Single method like so and specifying a 'since' and 'until' date, I wouldn't need to use the maxid and sinceid right? I hope I'm not confused! -

var twitterCtx = new TwitterContext();

var queryResults =
from search in twitterCtx.Search
where search.Type == SearchType.Search &&
search.Query == "Linq To Twitter"
search.since == specifiedFromDate
search.until == specifiedToDate
select search;

Search srch = queryResults.Single();


May 29, 2012 at 3:52 PM

The Since and Until parameters would be less precise if the Until was today.  That's because you're likely to receive additional tweets after your previous query within the same day.  SinceID and MaxID are very precise because they let you keep track of what you've received and still get new results without duplication.


Jun 1, 2012 at 8:05 PM


If I use search.until like this

search.Until == nowTime.Date


Will it bring me back results until this very second? Do I need to Omit Search.Until for achieving this?


Jun 1, 2012 at 9:30 PM

The servers for twitter are scaled out all over the place. So Time accuracy is a big issue as time zones are different.

This is also undefined, there is no documentation on this. So what twitter gives you is what we get unfortunately.

DateTime.Now will actually produce an error for you.

Go to the twitter forums if you'd like to see if you can get some answers there




Jun 20, 2014 at 1:01 PM

I am using linqtotwitter -Version 2.1.11 facing the same issue that @emptyfish was facing above.

From his code, I am pretty sure that it would start working, but I couldn't find those properties (search.Page == pageID) and the ones that @JoeMayo specified (search.since == specifiedFromDate search.until == specifiedToDate) in the version 2.1.11 of LinqtoTwitter- which is much recent.

What is happening is in some specific query, I am getting as few as 2 results only, while on!/search-home there are around 15 (default-value) or more.
        var auth = new ApplicationOnlyAuthorizer
            Credentials = new InMemoryCredentials
                ConsumerKey = "yfBzaLaPCfV0eAypLnbSzEwai",
                ConsumerSecret = "5v5CkF3XLjRdE6vrsIABr82k049xm3nXXsoEUV2CBTkCSi24UT"
        var twitterContext = new TwitterContext(auth);
        var searchResults =twitterContext.Search.Where(x => x.Query == QUERY && x.Type == SearchType.Search);

        Console.WriteLine("Total Search results : {0}", searchResults.Count());
        Console.WriteLine("Search results Until: {0}", searchResults.SingleOrDefault().Until.Date.ToString("dd/mm/yyyy"));
        Console.WriteLine("Total Search results.Statuses.Count : {0}", searchResults.SingleOrDefault().Statuses.Count());

        foreach (var item in searchResults.SingleOrDefault().Statuses)
            Console.WriteLine("\n {0} {1} {2} {3}", item.Count, item.User.Name, item.User.CreatedAt.Date.ToString("MMM dd"), item.Text);
Can you please help me with this?

Varun Shringarpure
Jul 9, 2014 at 8:55 PM
I am having the same problem as varunvns. The latest API seems to be inexplicably lacking the pagination functionality discussed in this thread, making it impossible to query more than the 100 most recent tweets for a particular search. Is there are workaround? Why was this functionality removed?