FriendshipType.FriendIDs Cursor AND Rate limit

Apr 6, 2014 at 4:45 PM
I need to use a cursor to get all of the friends ids of a user otherwise I only get the first 5000.

At this page (http://linqtotwitter.codeplex.com/wikipage?title=Listing%20Friends&ANCHOR) there is the following sample WITHOUT cursors
            var friendList =
                await
                (from friend in twitterCtx.Friendship
                 where friend.Type == FriendshipType.FriendIDs &&
                       friend.ScreenName == "JoeMayo"
                 select friend)
                .SingleOrDefaultAsync();

            if (friendList != null &&
                friendList.IDInfo != null &&
                friendList.IDInfo.IDs != null)
            {
                friendList.IDInfo.IDs.ForEach(id =>
                    Console.WriteLine("Follower ID: " + id));
            }
I have tried all kinds of variations of this code in an attempt to page through the data with no success. Any help appreciated.


Problem 2. Suppose I did get the cursor code working and the user has more than 75000 friends. I notice that the rate limit is at 15 requests. 15 requests at 5000 per page is 75000. How do I get the rest??

Thank you.
Apr 6, 2014 at 5:28 PM
you can't get the rest . you would have to wait another 15 minutes to get the rest and so on . so instead of showing the user result right a way make it every hour or so
Apr 6, 2014 at 5:35 PM
So in an hour you get the same 75000 again? How do you EVER get to the records after the first 75000? What kind of strategy do you use since they don't use a "page number" presumably because that would just be too easy? Thanks.
May 2, 2014 at 4:14 PM
I have not been working on this but the boss is now on me to get this done. In the first post of this thread I cited the sample from the documentation, but it doesn't use cursors. I need to get all of the friends ids for a user. This is what I came up with but it doesn't work. I chalk my failure to 1. I don't know this object model and 2. I have no experience with LINQ and find the syntax confusing at this point. At any rate, I need to get this working and any help is appreciated. Code snippet follows.
        static async Task ShowFriendIDsAsync(TwitterContext twitterCtx) {
            //

            Friendship friendship;
            long cursor = -1;

            long i = 0;

            do {
                friendship =
                    await
                    (from friend in twitterCtx.Friendship
                     where friend.Type == FriendshipType.FriendIDs &&
                           friend.ScreenName == "JoeMayo" &&
                           friend.Cursor == cursor
                     select friend)
                    .SingleOrDefaultAsync();

                if (friendship != null &&
                    friendship.IDInfo != null &&
                    friendship.IDInfo.IDs != null &&
                    friendship.CursorMovement != null) {

                   cursor = friendship.CursorMovement.Next;

                    friendship.IDInfo.IDs.ForEach(id =>
                        DoWork(id));
                }

            } while (cursor != 0);

            Console.WriteLine("Total: " + i.ToString());
        }
Coordinator
May 3, 2014 at 3:23 AM
That looks similar to the code in the downloadable source code, so I think you have the right idea. In the foreach, you could also add the ids you collect to a collection that holds all of the ids. Also, see the Twitter API documentation on Working with Timelines to get a feel for why and how to work with cursors.

@JoeMayo
Sep 23, 2014 at 8:26 AM
Edited Sep 23, 2014 at 8:26 AM
Hi - I'd perhaps chalk this up to a bug? I ended up with exactly the same code independently, and friendship.CursorMovement isn't being set, which means the second part of the do-while loop isn't being hit, and the cusor stays at -1... :(
Sep 23, 2014 at 8:31 AM
Cracked it! You have to test friendship.IdInfo.CursorMovement, and reset the cursor with that... (Thank heavens for the debugger, eh...?)
Sep 29, 2014 at 3:15 PM
Wow, I had given up! Thanks. Can you clarify "test friendship.IdInfo.CursorMovement, and reset the cursor with that". My Linq knowledge is weak and doesn't allow me to understand that statement very clearly. Perhaps by correcting the code I posted and then posting a working snippet? Your help or anyone else's help is greatly appreciated.

Thank you.
Sep 30, 2014 at 8:31 AM
Edited Sep 30, 2014 at 8:31 AM
Ok. It's not really a Linq issue per-se. The problem is that, when you're trying to retrieve follower or friend ids they're held in the IdInfo property of the friendship object, so that's where the CursorMovement info is stored, too. So you have to change two lines in the code example above. The first is the final test in the final if statement - change:

friendship.CursorMovement != null

to

friendship.IdInfo.CursorMovement != null


... and then in the code block that executes in the if statement, change:

cursor = friendship.CursorMovement.Next

to

cursor = friendship.IdInfo.CursorMovement.Next


Worked for me, anyhow. Hope it does for you, too.

Incidentally, I found this by using the Visual Studio debugger - it's great for actuall seeing what's going on step-by-step when the code's executing. This allowed me to look at the friendship object in Visual Studio, then expand the IdInfo object, and see that that was where the cursor information was.