Retreiving All Direct Messages

Mar 5, 2014 at 10:55 PM
Edited Mar 6, 2014 at 12:44 AM
Hi - According to the Twitter API up to 800 DMs are stored and can be retrieved 200 at a time and am trying to do that based on the article and status-loop found here:

http://geekswithblogs.net/WinAZ/archive/2012/09/02/working-with-timelines-with-linq-to-twitter.aspx

modified for Direct Messages rather than Statuses to this:
            var messageList = new List<DirectMessage>();
            int count = 20;
            ulong sinceID = 210024050000000000;
            var directMessages =
              (from tweet in twitterCtx.DirectMessage
               where tweet.Type == DirectMessageType.SentBy
               && tweet.Count == count
               && tweet.SinceID == sinceID
               select tweet).ToList();
            ulong maxID = directMessages.Min(dm => dm.IDResponse - 1);

            messageList.AddRange(directMessages);

            do
            {
                directMessages =
               (from tweet in twitterCtx.DirectMessage
                where tweet.Type == DirectMessageType.SentBy
                && tweet.Count == count
                && tweet.SinceID == sinceID
                && tweet.MaxID == maxID
                select tweet).ToList();

                if (directMessages.Count() != 0)
                {
                    maxID = directMessages.Min(dm => dm.IDResponse - 1);
                    messageList.AddRange(directMessages);

                }
            } while (directMessages.Count() != 0 && messageList.Count < 801);

            for (int ix = 0; ix < messageList.Count; ix++)
            {
                DirectMessage dm = messageList[ix];
                uxLblResult.Text += String.Format("<BR/><BR/>{0} : {1}", ix, dm.Text);
            }
The loop is executing but Twitter stops returning data at 200 DM's

Not sure if
  1. Is there something else we need to do in the case of DM retrieval
  2. There is something wrong with our use of MaxID and SinceID
  3. The API doc wrong/out of date
And help would be greatly appreciated,
Thanks Very much!
Coordinator
Mar 25, 2014 at 11:33 PM
Hi,

It might not have anything to do with your algorithm. You might be overreaching your rate limit. There are RateLimitXxx properties on TwitterContext that LINQ to Twitter updates on every query. There's also a HelpType/RateLimits query that tells you what the rate limit is for an API (or you can look at the Twitter docs)/time window, which is normally 15 minutes.. For the DirectMessageType/SentBy query, your rate limit is 15 for every 15 minutes. Looking at the number of messages you're querying, you have the count set to 20. Consider that sometimes you don't get a full 20 messages back on every query. Then it's easy to see a situation where you've exceeded your 15 queries and only received 200 messages.

To avoid this, increase count to it's max, which is 200. That should increase the chance that you'll receive 800 without exceeding your 15 queries/15 minute rate limit window. Also, Twitter's Working with Timelines article is pretty good too:

https://dev.twitter.com/docs/working-with-timelines

@JoeMayo
Mar 26, 2014 at 2:25 AM
Hi Joe,

Thanks for your response.

Sadly changing Count to 200 had no impact.

I had read the twitter doc previously but did notice this time that any Linq2Twitter code samples I have seen leave the "sinceID" set to the arbitrary starting number (210024050000000000 in my code) whereas the Twitter doc suggests setting it after each query for efficiency.

In this case though we are not worried about DM's added since the first Query so leaving it unchanged seems right, we are trying to work our way back using only MaxID.

Some improvement by doing separate queries for SentBy and SentTo wherre I can get 200 of each, so I know they are there but nothing beyond that. If I do not specify type I get the most recent 200 only.

It is possible that one or more of the Accounts involved (Authorized Receiver or various DM Senders) is in "Direct Message Jail" having exceeded the rate limit of 200 in 24 hours I think it is. Not sure why that would make a difference but thought I'd mention it.

Also, have not seen anything in Twitter docs but wondering if there is Time Period (how old) limit on DM queries that you are aware of.

I am stumped, any suggestions at all would be most welcome!

Thanks,

Greg
Coordinator
Mar 26, 2014 at 5:40 PM
There might be a time limit. Status timeline and Search queries only go back a certain amount of time, so it seems like DM's would be the same way.

@JoeMayo