As part of a broader hyper-personalization trend noticed in the user engagement space, the layouts of mobile and web applications are moving away from page-based to card-based designs. This concept is gaining wider adoption across a spectrum of platforms and applications. Facebook, for instance, used to have a running post feed, they now use card layout with clear boundaries for content. Twitter, Pinterest, LinkedIn, and other popular platforms have started adopting cards as their primary user interface (UI).

Sensing this trend early in the market, Microsoft launched the card-based framework, Adaptive Cards, in early 2017. The framework offers platform-agnostic snippets of UI, authored in JSON. Adaptive cards enable developers to exchange UI content in a common and consistent way across apps and services. When delivered to a specific app, the JSON is transformed into a native UI that automatically adapts to its surroundings. Adaptive Cards framework comes with a designer interface to assist developers to design cards on-the-fly.

In-line with Microsoft’s overall strategy to build open source technologies, they made Adaptive Cards open source, which allows anyone to use it in applications. It has renderers for multiple platforms, enabling an ecosystem of shared tooling, seamless integration between apps, and native cross-platform performance on any device. They also provide SDKs for IOS, Android, .Net, React Native, Xamarin, Bing, Luis, Outlook, and so on.

Integrating Adaptive Cards into the Microsoft application ecosystem was part of the initial Adaptive Cards rollout plan. They have come a long way; now facilitating cross-platform integrations within their ecosystem, that is you can send a card from Skype to Team, Luis to Outlook, and so on.

Given the rapid industry adoption, Microsoft is also coming up with industry standards for Cards, to define guidelines around seamless adoption to the host theme. For example, if you are using a black theme in your mail, then the card will automatically adopt this theme.

Here are some of the key benefits of Adaptive Cards:

• Adaptive Cards are simple and declarative in nature
• Cards are cross-platform compatible and easy to integrate 
• Microsoft now offers Adaptive cards templating, which allows creating, reusing, and sharing of adaptive cards. This helps in separating the data from the layout.
• Conditional formatting enables the application of simple logic at the frontend

Our journey with Adaptive Cards started last year when Imaginea became the first non-Microsoft contributor to the Adaptive Cards GitHub repository. Our team designed and built a card renderer for the React Native community. At that time, Matt Hidinger, Principal Program Manager, Microsoft had quoted “We’re pleased to see Imaginea’s commitment to the Microsoft Adaptive Cards project and to recognize them as the first external non-Microsoft contributors that we added into our GitHub repo with write access.”

We have been collaborating with the Microsoft team since then and actively contributed in defining the Adaptive Cards roadmap. Deeper understanding of the architecture gained through feature contributions has helped us adopt this technology successfully for customer projects. For a large retail major, we built a LUIS based chatbot plugin designed to be integrated with any native application, with minimal setup. We achieved ‘Multiple Canvas | Multiple Application’ by using React Native Adaptive Cards for rendering the chatbot responses in a rich and dynamic conversational interface.

While templating gives the flexibility to reuse and share a card design, we wanted to make the process of creating cards even simpler by offering an option to directly convert an image to a card. ‘Image to Card’ conversion is the new feature that we have built this year. We have used a combination of Computer Vision, Deep Learning, and Heuristic Models to understand the objects and formatting in an image. The image is then automatically converted into a card, which means, you can just upload an image and receive the card output in the JSON format effortlessly.

And the big news is here – the Microsoft team will be integrating this exciting new feature with the source code, and they will be showcasing this in the upcoming Microsoft Build Conference.

When we started this project, the first step was to identify how well a custom object recognition algorithm works on textual content. We used a FASTER RCNN model with 9000 steps and found a final loss of 0.2274 to detect objects. Here is the flow – first the object edges are detected using Canny Edge detection algorithms, then the contours are built around the objects, followed by object detection.

Once the objects and the texts are detected, the next step is to detect textual properties such as color, text size, boldness, etc. The text content of the object is extracted using Pytesseract, the text size and boldness is calculated by building contours for the text object, and PIL.quantize helps to pick the font colors.

The final step in the process is to understand the card structure. It involves the following steps:

1. Grouping the same y range images as image sets
2. Grouping the elements as choice sets based on its Ymax and Ymin differences
3. Grouping the same y range elements as column sets
4. Finally arranging the identified elements vertically [based on Ymin incrementally]

The prospect of converting an image into a working prototype with a certain level of accuracy will transform the front-end development approach. As a sequel to the Adaptive Cards ‘Image-to-Cards’ effort, we plan to replicate this approach for mobile and web application development as well. Yes, you heard it right. You just upload the image screens of your mobile application to receive a working prototype of IOS and Android apps.

We have kept the feature as open-source; you can find the source codes in our GitHub repository. If you have suggestions or would like to contribute to this project, please drop me a note at [email protected].

I will keep you posted on the progress. Stay safe!