sketch2app is an application that let users instantly convert sketches to fully functional AI applications. This blog is Part 1 of the LLM AppStudio Blog Series and introduces sketch2app

The H2O.ai team is dedicated to democratizing AI and making it accessible to everyone. One of the focus areas of our team is to simplify the adoption of data science projects by providing the fastest solutions for application developments. As Large Language Models (LLMs) continue to advance, there is a growing interest in the Apps powered by LLMs and Generative AI (Gen AI). After the release of our two incredible open-source projects, h2oGPT – the Open Source LLM model, and H2O LLM Studio – No Code LLM Model’s fine-tuning platform, in this article, we share our vision for H2O LLM AppStudio – a platform that enables users to rapidly create applications that are powered by large language models to give high-quality personalization and user experience.

One of the key components in the App Studio is the application called: sketch2app which allows users to generate interactive, web-based fully functional applications just by creating a sketch.

Sketch2App harnesses a blend of computer vision models, natural language processing, and LLMs (Large Language Models) to transform simple sketch diagrams into fully functional applications. The generated application code is not only editable and customizable but also downloadable, offering flexibility and convenience to users. To enhance its content generation capabilities, sketch2app integrates h2oGPT, enabling the creation of dynamic and engaging content within the application.

Generative AI applications can be classified into two categories. The first type involves applications that leverage Generative AI to convert user inputs, such as data, prompts, or sketches, into functional application code snippets. Examples of such applications include sketch2html, which transforms a sketch/design into HTML and CSS codes, sketch2app, which converts a sketch/design into a complete web application, notebook2app, which turns a notebook into a web application, and prompts2app, which converts a series of user prompts into a web application.

The second type of application leverages the power of LLMs or Generative AI models to provide hyper-personalization and an immersive user experience within the apps. These applications offer advanced features like context-based search, tailored conversations, and question-answering capabilities, all powered by LLMs. They are known as LLM Powered applications.

Sketch2App exemplifies the first type of application, utilizing the capabilities of generative AI, various machine learning models, and pre-trained deep learning models to create applications based on simple sketches. It also employs LLMs to a certain extent, specifically for content generation within the generated app, enhancing the overall user experience and personalization.

How it works?

The application takes a sketch as input, which is essentially an image with various properties such as layout, structure, components, box positions and coordinates, width and height, placement, and box labels. To recognize these properties, the application utilizes a series of interconnected machine-learning models in its core pipeline. During the inference process, an intermediate token called the DSL (Domain Structured Language) token is generated. The concept of the DSL token was initially introduced in the pix2code paper and has served as inspiration for sketch2app. This token contains crucial information about the sketch’s layout, coordinates, text, positions, shapes, and overall structure.

To convert the DSL tokens into the respective h2o-wave component codes, a compiler is employed. These component codes are then combined to create the web application. Additionally, the generated application is integrated with h2oGPT to facilitate content generation. This integration allows the application to generate various types of text related to the converted application. This includes but is not limited to, header text, introduction paragraphs, and content for the About page of the application.

Key Functionalities

1. Instant Conversion: Sketch to Code to App: The model chain described in the previous section enables the seamless and instant conversion of paper sketches into application code. With this streamlined process, users can quickly go from a simple sketch to a fully functional app without any delay or hassle.

2. h2oGPT Integration for Content Generation: Integrated with h2oGPT, an open-source LLM Model, this app offers advanced capabilities for content generation. It empowers users to effortlessly generate content, including quick text and paragraphs, that can be easily incorporated into the generated applications. This integration simplifies the content generation process, allowing for the rapid creation of functional prototypes. For instance, users can request h2oGPT to generate an introduction paragraph about a “Customer Churn Dashboard,” provide insights on a specific dataset, or even generate a template for a “Terms and Conditions” page.

3. Wizard-based Application Editing: The app features a user-friendly wizard that facilitates the creation and design of applications. This intuitive interface caters to designers and developers alike, enabling them to easily build functional prototypes without extensive coding knowledge. The wizard also generates text and images, further enhancing the overall application design process.

4. Built-in Code Editor: With the inclusion of a built-in code editor, users gain the ability to update or modify the generated code directly within the app. This powerful feature empowers users to fine-tune the application code according to their specific requirements. By providing the flexibility to customize and optimize the code, users can ensure that the final application is fully optimized, functional, and tailored precisely to their needs.

5. Downloadable Source Code: Sketch2App offers the convenient option to generate downloadable application codes. This capability simplifies the development process by allowing developers to export the code and seamlessly integrate it with other development tools. By providing the ability to download the source code, developers can expedite the creation of fully functional applications while maintaining compatibility with their preferred development environment.

6. Handles Complex Structures (Nested, Multi-Column, Multi-Row): The application is designed to handle even the most complex design structures. It efficiently manages intricate layouts such as multiple rows, multiple columns, and nested columns or rows. This ensures that the app can effectively accommodate diverse design requirements, providing users with the flexibility to create intricate and visually appealing interfaces.

Getting Started: A Working Example

Let’s examine an example of how a sketch can be transformed into an application using sketch2app. To begin, we launch the application and input the sketch. The sketch can be designed using any tool or software of choice. An optimal sketch typically comprises multiple boxes or cards arranged on a plane. These boxes are labeled to indicate the type of card to be created, and their dimensions (width and height) determine the corresponding size of the cards in the application. The application itself provides a comprehensive list of available labels and their respective aliases for easy reference. It is crucial to carefully position and size the cards in the sketch, as these elements are precisely mapped in the generated application, ensuring accurate representation.

Next, we trigger the sketch2app pipeline that starts processing the input image, generating the corresponding DSL token, and finally creating the application source code.

Once the source code is generated, it serves as the foundation for running the application. The generated application can be further refined and personalized using the built-in editing options within the app. These editing capabilities encompass a code editor that allows for modifications to the source code itself, a wizard-based theme editor for adjusting the application’s visual elements, and an LLM-powered content generator to enhance the app’s dynamic and engaging content.

Furthermore, both the code and the application itself can be easily downloaded, providing the flexibility to run them on alternative platforms such as local machines or servers. This feature enables users to deploy and utilize the application in various environments according to their specific needs and preferences. Whether it’s fine-tuning the code, customizing the application’s appearance, or running it on different platforms, sketch2app offers a comprehensive set of tools and options to ensure a seamless and adaptable development experience.

The integration of LLM (Large Language Models) empowers users to generate content for their applications with ease. This content generation capability encompasses various elements, including title/header text, descriptions to be added in markdown cards, report generation, synthetic data for application use, and summarization of information. By utilizing LLM models, such as the default h2oGPT, users can leverage the power of natural language processing to create high-quality and contextually relevant content.

Moreover, the application offers flexibility in terms of LLM selection. While it comes pre-configured with h2oGPT, users have the option to configure and utilize other LLM or GPT models according to their specific requirements. This allows for seamless integration with a wide range of LLM models, whether they are sourced from open-source libraries or API-based solutions.

Additionally, the application goes beyond text generation and offers the capability to generate images using text prompts. By leveraging available open-source or API-based models, users can effortlessly generate images based on specific text prompts. This feature further enhances the application’s versatility and provides a holistic content generation experience, encompassing both textual and visual elements.

Full Demo of sketch2app
Here is a full demo of using sketch2app from using a sketch drawn using the diagrams.net tool, and generated application within a minute.

H2O LLM App Studio empowers users to swiftly transform their ideas into fully functional applications. Sketch2App, a robust and adaptable app development tool, takes a sketch as input and generates the corresponding application. It offers a wide range of features, including the Generative AI Wizard, preview capabilities, built-in code editor, and logging functionalities. With Sketch2App, one can create customized applications without the need to write any code. If you would like to access or learn more about H2O LLM App Studio, please don’t hesitate to contact us at cloud-feedback@h2o.ai.