As promised, we’re back with the second part of the article about processing data from Social Media. You can find our previous article on the topic by this link:

How to Extract and Process Data from Social Media. Integration with Ruby on Rails (Part 1)

In today’s post, we’re going to tell you how to organize your code in a way that adding a new social media wouldn’t cause any issues.

However, before we get to the business, let’s go through the main aspects that are worth mentioning. Since, believe us or not, you’ll certainly face them when processing data from a social media.

Access tokens

When authorizing, you’re provided with access tokens, which are used to make requests to the API of a social media. Usually, it’s either just one token or a token with a key to it. The issue is that these tokens may become invalid due to changing the password or simply expiring.

For this reason, we recommend processing token errors at all times and ask users to reconnect tokens to their accounts in social media.

Limits

All social media have their own limit for the number of requests. While some of them allow for making quite a lot of requests, the others have very strict limitations. That’s why you should look into this point beforehand.

Also, you should add the piece of logics in the app that will be able to postpone all the requests to the social media if the limits are exceeded.

Caching

Another thing you should know about social media is that they cache data all the time. As a result, you may face the situation when the number of followers in the API request may not match the actual number of followers in the social media.

If something tells you that there is an error somewhere, you can always contact the tech support.

Pagination

Social media has tons of users who generate an insane amount of content. If they used standard pagination, the user experience would leave much to be desired.

For example, while the user is reading the first 10 posts on the first page of their newsfeed, their friends have created 5 more posts. Because of that, the user would see the same posts on the second page. This is why all social media use either a cursor or boundary parameters.

You can think of a cursor as a simple page with the difference that, instead of a number, they pass a special key that the social media returned in the first request. On the last page, this key is empty, which means that there is no data to display.

Boundary parameters are usually in the form of timestamp or post id. They are passed with each request, so the user doesn’t receive the posts that are newer than specified.

As a rule, all Ruby gems support pagination out of the box. You just need to understand how it works. However, there can be certain tricks when it comes to specific social media. For example, Twitter gem pulls all followers when accessing the list of followers. If the user has more than 3000 followers (15 requests with 200 followers), you’ll exceed the limit. So, the procedure won’t yield any results.

When working with Twitter, we recommend using the following cheat sheets:

• GET friends/ids
• API reference index

Tech support

Even such giants as Twitter and Facebook have bugs. However, you can always contact tech support. Usually, every social media has its own testing console. Sometimes there is even a feature that allows for saving the request, in which you can attach a link.

For example, Facebook returns query id in the header. You can attach it to the message as well. In case you work with other social media, we recommend providing the tech support with everything necessary for them to reproduce the issue as accurate as possible.

Unfortunately, the process of solving this or that issue can be slowed down because of human factor. Sometimes the fastest way to deal with an issue is to create a new ticket for it and cross your fingers that it’ll be sent to a different person.

Organizing the code

Now goes the part that you’ve been waiting for the most: how did we make the entire thing work? Since the task was to collect data from social media, we called the processing module “grabbers.”

Here is an example of a basic grabber:

As you can see, the code is initialized using access tokens. The code takes the app keys from the configs and calls the method to initialize client API. The basic class is responsible for uniquelization of the errors from the API. This allows for having single logics to process the errors.

If there is a token error, the account of a user is marked as invalid; the user is asked to reconnect the social media. If the limit is exceeded, the procedure is repeated after some time. If there is a server error, we try do the same after a few minutes. If there is some other error, it’s logged and left as is.

Here is how grabbers for other social media look like:

The main method here is grab_endpoint, which makes a specific request that passes control to the method with the corresponding name. Also, before this, the parameters are normalized. Thus, we can pass parameters for all social media in a single format from the outside. Next, the method of the social media gem is called and the result is normalized.

Practically, all key pieces that are responsible for working with Facebook are in the same place (and, usually, in the same class). Unfortunately, the new version of Facebook’s API doesn’t work as it should. Because of that, to pull high-res images, we had to create the following workaround:

Conclusion

That’s basically it, guys. Following these steps will help you create a mobile app that allows users to view multiple social media in one place. To conclude the series, here are the main takeaways:

1. When it comes to social media data colection, APIs of all social media rely on the same principles. Keep this in mind.
2. When working with a social media API, the most important thing is to create a system of classes that receive normalized parameters and returning results in a single format.
3. All the key pieces of code responsible for processing data from a specific social media should be in one place.
4. Don’t forget to log every request to a social media. Even a slight change in the API can break your app.
5. Follow official blogs for developers. They’re the place where you can find everything about what’s changed. They can also give you some additional information about how to collect data from social media the right way.

P/S — If you liked the article, please support it with claps (you can press and hold the clap button for long claps).

Also, if you’d like to know more about the technical side of things or want to develop your own social media app, feel free to drop us a line at [email protected]

How to Extract and Process Data from Social Media. Organizing the Code (Part 2) was originally published in JetRuby on Medium, where people are continuing the conversation by highlighting and responding to this story.