Read the Full Transcript

 

00:06

Imagine a place where you have more technological advancements happening than the product being built. And this phenomenon we can call cultural lag where rate of innovation in tech itself is much, much higher than the utilities which are consuming those technologies. 

 

00:31

And that future is basically now where we have too much technical innovation happening. Hello everyone, my name is Sandeep Singh. As you have seen my very generous introduction, I'm going to talk about how you can use these latest and greatest technologies to build something meaningful for you without being heck of Ninja Coder. 

 

01:04

So and as this talk is a lot about building LLM products using open source technologies, my talk will be little different. It's basically about making you aware of you can do pick and choose and you can decide for yourself how to get things done faster and in more effective manner. 

 

01:32

Because this domain is changing much faster than what we can imagine. Two months feel like a year in this domain because excessive amount of research and innovation happening and I'm going to explain to you how you can really make meaningful decisions for yourself and build products faster. 

 

01:56

When I say faster, it means matter of few days rather than weeks and we are going to talk about it. All right, here is the product. approximate agenda, we'll basically talk about the ecosystem of the open source LLMs tool and significant of blending between open and closed phenomena in this domain. 

 

02:18

And we'll talk about benefits of one versus other and what integration strategies you should choose and how to decide like what it means to integrate these technologies for your product and few case studies from my company which I am currently engaged with Beans.ai and I'm going to talk about few facts which will probably will surprise you that how easy it is. 

 

02:49

A little bit about me of course we already talked about it, I'll skip over it. Okay, so purpose of this slide is basically to tell you that on both sides of landscape open and closed source LLMs solutions. 

 

03:06

There's a gazillion options available. When I say gazillions, don't take it literally. It's more to just give you idea that there are more options available than what you can probably imagine. These are a few of them. 

 

03:21

If you want to take a picture of it, but basically just to give you a lot of options in open source as well as closed source or vendor specific solutions, as we know. And we're going to talk about how to leverage each side of this landscape to build your products effectively. 

 

03:40

Because nowadays, I've seen people have which is valid sort of affinity to either this or that. But in my experience, what we have done, we have built an extremely effective support bot in matter of hours. 

 

03:59

When I say hours, it's less than 8 hours, which was production grade. And we have used things which I'm going to talk about. So probably it will give you idea you don't have to be much of the coder or research house researcher to build these solutions for your company. 

 

04:18

Let's talk about it. This slide is mainly for sake of completion. Basically, it talks about open source and LLM's main benefits, why you want to choose open source, not the closed source. And of course, one of the main reason is that you can deploy whole thing on your own premise. 

 

04:39

So your data or your customer's data never goes out of your own network. And it is driven by a lot of time by compliance. For example, health care, finance, they have strict guidelines. Your data cannot flow out of your network. 

 

04:55

And then open source is a good candidate. And cost saving, you don't have to spend too much on your API calls to vendors like ChatGPT or Anthrox. through Anthropics or similar companies. No external dependency. 

 

05:11

What it means, it means, suppose you start using, say, Anthropics Cloud. And after a few months, they come up with something which doesn't suit you. Excuse me. You wouldn't be able to switch it very easily if it's highly embedded in your ecosystem. 

 

05:32

But if it's an open source, you will be able to decide for yourself. You want to replace with other open source. You'll have a higher decision making. And again, other examples are like code transparency. 

 

05:45

You know exactly what code is running for your business logic. And you have a higher control on that in case of open source. And language model customizations, in the sense one of the biggest challenges in this domain is domain adaptation. 

 

06:01

What it means that these generalized model Models are very good at talking generalized stuff about particular domain. For example, if you want to talk about the cancer, they will understand the generic terms. 

 

06:14

But if you are a, suppose a doctor or physician and you are doing some fundamental research to extend the field and if you try to talk to the NLLM about that, it won't be very effective. So what you want it to do, basically, to understand the deeper meaning of medical terminologies. 

 

06:35

And for that, you have to probably get more data and overfit so it understands your domain of medical much better than what it does in generalization, generally available APIs. So that's one of the need where open source models do much better than closed source and of the shelf vendor specific models. 

 

06:58

And of course, open source have communities like this. And don't underestimate it or take it for granted. We are really moving the mountains by open source communities like this. And thanks to H2 and Arno and Jonathan and she, of course, to build communities like this and conduct conferences like these, because you can really get very complex things done in a matter of weeks, which otherwise will take significant amount of more time. 

 

07:30

And of course, it fosters innovations because when you are part of open source community, you get to talk to a lot of folks who are doing wonderful stuff. And of course, it's boon for startup. And we are going to talk about specific use cases why it matters a lot for startups. 

 

07:47

Open source LLMs are much more useful than the closed source, especially the cost perspective. On other hand, we have closed source LLMs where we can do the similar tasks, but with different types of tools. 

 

08:05

different benefits at hand, and one of them is support and reliability. So for example, if you are something, building something which is really mission critical, and you don't have opportunity where you can really put your heads down and do the fundamental research when you hit an issue, in cases like that, probably, close source LLMs are a much more better option. 

 

08:31

Because the responsibility of doing fundamental research and extending the field is not up to you. It's for the research house of close source LLMs, like OpenAI, Google's Palm, or Cloud, Anthropic, or anything like that. 

 

08:48

And nowadays, I'm happily able to say that all close source APIs are also providing some sort of business customization, fine tuning. and instruction tuning sort of support. So that is a great thing, which was not there, say four months ago. 

 

09:06

It was not there and it's freshly provided functionality. And it's a great selling point. Of course, security and data privacy performance, they are much better because they elastically scale as per your need. 

 

09:23

You don't have to worry about the LLM ops as a previous speaker was talking. And you can integrate. Nowadays, they're providing you to integrate to your proprietary system. They're not as effective as fine tuning, but they're still helpful. 

 

09:40

And next point is the guard rails. I will take a pause here and I would tell that this is probably the most important reason why you should choose closed source LLMs. Because suppose you're deploying these chatbots to some sensitive domain. 

 

09:56

For example, for kids education domain or health care, and all of these LLMs are stochastic machines. It means they are not deterministic fully. And depending on their temperature and temporal component, they will do a different thing if asked the same question again and again. 

 

10:18

So that's a good selling point for closed source LLMs. And of course, they do continuous development because they pump in a lot of research money. And they are responsible for making commercially viable for you if you use the closed source solutions. 

 

10:37

Of course, we have talked about both parts, but let's see what you as a user of this solution have at hand. Of course, you have a number of options available and possibility of exploiting more than one solution. 

 

10:49

It means you don't have to be logged in to closed source or just the open source. You can just do the mix and match and very much what we have done in our solution. Many companies are doing to build more robust applications. 

 

11:03

And again, you can compare each LLM for your own purpose, close source or open source, on your own task, not just the benchmarks, because benchmarks are benchmarks. They are the research -oriented entities, and they have a different way of measuring things than what you might want. 

 

11:25

And they have different baselines for particular domain adaptation. Amount of fine -tuning needed is not same for different models. So you can see which needs less fine -tuning. You can choose that, or you can decide for yourself. 

 

11:40

And possibility of using a specific LLM solution for a specific task in pipeline. For example, if you have 10 tasks, you can see maybe this task will be done by the close source LLM and other tasks will be done by open source LLM. 

 

11:54

And it will depend on what sort of ensemble you want to create. A lot of options are available, and ability to pick and choose is there. So you are really at good time where you can build a great LLM app. 

 

12:09

I just want to do a quick show of hands. How many of you have really built at least one prompt engineer app or any type of LLM app here? OK, we have a few. Thanks. And rest of you, you are at just at the right conference. 

 

12:29

Because this is the place where you can really build mind -boggling innovative apps without writing a single line of code. And this app will be fully browser compatible. It will have all the bells and whistles, and many amazing things you can do without writing a single line of code, and by using all the default things. 

 

12:52

And I'm quickly going to rush over through. So first thing, what you have to do, break your LLM initiative into LLM tasks. I'm just pausing for a second. So you just marinate in that thought, which means that just decide, like for example, you want to build an e -com site, or you want to generate SQL from natural statements. 

 

13:19

Just break them in your initiative into the LLM task. And then, categorize all your tasks in which task is really mission -critical versus which task has tolerance for stochasticity and the lack of deterministic behavior. 

 

13:35

And that will guide you in choosing which LLM to use for which task and how to build your pipeline. And of course, less tolerant tasks are candidate for proprietary off -the -shelf solutions, like where you need higher quality, higher deterministic behavior, more control, temperature parameter, and so on. 

 

13:55

And the more tolerant tasks are candidate for open source with no or less fine tuning, where you can really go by even if the quality is not bang on. So we at Beans.ai, we are basically a mapping company. 

 

14:12

We are much more accurate than Google Maps, Apple Maps, no offense for apartments. It's a full blown thing where it tells exactly where to park, after parking, where to walk, where to walk inside the apartment, where it's entering, and so on. 

 

14:30

And Google Maps and other maps don't have that. But how we have built it, we have integrated a lot of innovative computer vision and machine learning and optimization algorithms. But LLM is at the center of consumption engine for the users. 

 

14:48

A lot of times, the thing which they ask, it gets interpreted by LLM Engine. And we then switch it to computer vision task, search task, or. some sort of linear optimization task. And we have used combination of various types of closed source as well as prompt engineer third party LLM solutions. 

 

15:10

And that is where mine has to see these, all the LLM solutions and set of options. It's like you're in a shopping mall and trying to buy what will work for you. And don't go with the mindset that you just have to have either ChatGPT or LLAMA or VQRA. 

 

15:29

And that's a bias I see all the time. People are either too enthusiastic about open source or they are just like more corporate feel with affinity to subscription based models. So that's where we need to innovate more. 

 

15:46

And what we do at Beans.ai, those who come from the Kaggle background, they will know there is something called the ensemble where a lot of weak learners, they work together and eventually they form the stronger learner. 

 

16:00

And that's what I have done here at Beans.ai, where what I do, I take the responses from a lot of LLMs. And then I consume them in LLMs which are open source. And I've not done much of fine tuning for my domain. 

 

16:14

And then I take them and send those responses to the closed source LLMs where millions of dollars have been spent by open AI to have a higher quality to reprocess the responses from open source solutions. 

 

16:28

And then I use those solutions in my app. So what happens? Maybe 30 calls I make to my proprietary open source LLMs. But I just make one call to highly costly. As a matter of fact, it's not that pricey. 

 

16:42

Yesterday they reduced the price. Good news, of course. So what I'm trying to convey here is that you really have to do mix and match. Don't just be heads down on one option. And we use it for automated support. 

 

16:58

bot which answers a lot of our questions which our customers have and it generates sequels, insights from dashboards, automated email consumption, response, order creations and so on. And all of this is done in pretty lightweight manner where we have not done like a lot of blockage to do the research. 

 

17:22

We basically build solutions one after another by using the combination of open source and the close source solutions. All right. Okay data enrichment is one of the use case which we use and this is very interesting. 

 

17:36

I'll give you an example here. We have apps which is used by a lot of delivery drivers and they ask a lot of questions. So I'll come to that but basically your data is limited by how much you can use for your training until you're a big corporate entity and you have like limited quantity, quality and variety. 

 

17:58

And that's where you can use LLMs to enhance all three dimensions. And for example, delivery drivers for our app, they ask, how do I mark an address not deliverable in app? And they need some steps, like go there, touch this, enter here, save, and so on, some sort of instructional question. 

 

18:19

But the same question, it can be asking much more than one way. So what I do, I send that to, say, ChatGPT, ask the above question in 20 different ways. And you get this. So you get questions like these, and all of them asking exactly same thing. 

 

18:40

And what it means, so my one candidate answer of steps to mark an address undeliverable is answer to all of these questions, which means that you have 20 times more sample data just by asking one prompt to ChatGPT. 

 

19:01

It means you can increase the quantity and the variety of your data just by making one call to ChatGPT. And that's where data enrichment comes into play. And another one is, for example, if you are not sure that you have a superset of tasks, like tasks 1, 2, 3, 4, 5, 6 here, but on particular request, maybe you will be doing maybe two or three tasks. 

 

19:26

But some human intuition is involved there. So basically, you can make one call to ChatGPT and ask out of these five tasks for this request, how many tasks are needed to be done and in which order. And it will let you know. 

 

19:39

And you can choose those two or three tasks and make the call to your open source LLM and do your purpose solving whatever you have at hand. So what I'm trying to say, dynamic behavior of selection of what to do can be automatically committed through calls to ChatGPT. 

 

19:59

And this is what we do. We have a highly dynamic behavior where we use these things. And of course, there are other use cases where you can build insomble and pipeline of the task at hand. OK, this is probably the most useful slide you're going to see in this presentation. 

 

20:19

I'm already out of time. I'll keep short. Who here have really not tried LLM Studio? OK, so good thing is a lot of you have tried. So what I'm trying to say is that H2O LLM Studio is a revolution. And how I realized, thanks to Arno and Jonathan and their team, I just had a demo in their office, LLM Studio is like eclipse for Java or pie charm for Python. 

 

20:54

It fosters the innovation. And you can create amazing LLM apps without writing single line of code. And the quality of that solution will be good enough, probably, for you. For example, the amount of time you have spent listening to this talk is enough to do a simplistic solution to real -time world problem. 

 

21:22

In less than 20 minutes, you can probably fine -tune. If you have a data ready and deploy it straight to the hugging phase. And all of these benefits, which you see on screen, it has support for low -rank adaptation. 

 

21:36

And it has a very intuitive UI. And the best part, and thanks to H2O Team, most of the hyperparameters by default work significantly better. You don't have to touch them. Just scroll them. And sometimes, if you don't know the parameter, just go and Google it. 

 

21:53

And you'll probably tell you whether increase it or decrease it and how it will affect your training. So one advice I have for all of you is that go and try LLM Studio to solve something like toy example. 

 

22:08

And it will change your perspective. Because all of us who have not really created apps, they think that these things are to be done through very researchy or developer mindset people. And this whole phenomenon, it's like a Gutenberg project. 

 

22:25

When books were starting to get printed, people thought only the very educated and elite and religious people needs to learn to read and write. But we know everybody should learn to read and write. And that is what happening with LLM apps. 

 

22:42

Everybody in the room should have mindset that they can create something which is useful for their purpose. Because in the future, the way we interact with information is going to change fundamentally. 

 

22:54

In fact, it's changing at this very moment. as we talk. So go give it a try. Sneak is a sneak peek. This is how it looks like. It's not scary. AWS like very cumbersome UI. It's very intuitive AI. And we have an amazing team who supports this. 

 

23:12

And as I was saying, you could have created a one fancy application of LLM in the same amount of time. I am just talking less than 20 minutes. OK. And that's it. I hope I passed some enthusiasm. So go and give it a try. 

 

23:28

Thank you.