Posit Open Source
Resources

A hacker's guide to open source LLMs - posit::conf(2023)

Presented by Jeremy Howard In this deeply informative video, Jeremy Howard, co-founder of fast.ai and creator of the ULMFiT approach on which all modern language models (LMs) are based, takes you on a comprehensive journey through the fascinating landscape of LMs. Starting with the foundational concepts, Jeremy introduces the architecture and mechanics that make these AI systems tick. He then delves into critical evaluations of GPT-4, illuminates practical uses of language models in code writing and data analysis, and offers hands-on tips for working with the OpenAI API. The video also provides expert guidance on technical topics such as fine-tuning, decoding tokens, and running private instances of GPT models. As we move further into the intricacies, Jeremy unpacks advanced strategies for model testing and optimization, utilizing tools like GPTQ and Hugging Face Transformers. He also explores the potential of specialized datasets like Orca and Platypus for fine-tuning and discusses cutting-edge trends in Retrieval Augmented Generation and information retrieval. Whether you're new to the field or an established professional, this presentation offers a wealth of insights to help you navigate the ever-evolving world of language models. (The above summary was, of course, created by an LLM!) For the notebook used in this talk, see https://github.com/fastai/lm-hackers. Presented at Posit Conference, between Sept 19-20 2023, Learn more at posit.co/conference. -------------------------- Talk Track: Notebooks+LLMs may just be the future of coding. Session Code: KEY-1107

Dec 15, 2023
1h 1min
Databricks Data Science RStudio Machine Learning Python Stats Tidyverse Data Visualization Data Viz Ggplot Technology Coding Connect Shiny Rmarkdown Package Manager CRAN Interoperability Serious Data Science Dplyr Forcats Ggplot2 Tibble Readr Stringr Tidyr Purrr Github Data Wrangling Tidy Data Odbc Rayshader Plumber Blogdown Gt Lazy Evaluation Tidymodels Statistics Debugging Programming Education Rstats Open Source Oss Quarto Posit ML

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

So let's start by talking about what a language model is. As you might have heard before, a language model is something that knows how to predict the next word of a sentence or knows how to fill in the missing words of a sentence.

We can look at an example of one. OpenAI has a language model, TextDaVinci003. We can play with it by passing in some words and ask it to predict what the next words might be. There's a nice site called nat.dev. nat.dev lets us play with a variety of language models. Here I've selected TextDaVinci003 and I'll hit submit and it starts printing stuff out.

The pandas were happily playing and eating the frogs that had fallen from the sky. It was an amazing sight to see these animals taking advantage of such a unique opportunity. The staff took quick measures to ensure the safety of the pandas and the frogs. So there you go, that's what happened after the extraordinary rain of live frogs at the panda breeding facility.

Tokens and tokenization

Now you might notice here it hasn't predicted pandas, it's predicted panned. And then separately, us. OK, after panned it's going to be us. So it's not always a whole word. Here it's un- and then harmed. So you can see that it's not always predicting words. Specifically what it's doing is predicting tokens. Tokens are either whole words or sub-word units, pieces of a word. Or it could even be punctuation or numbers or so forth.

So let's have a look at how that works. It's called tokenization to create tokens from a string. We can use the same tokenizer that GPT uses by using TickToken. And we can specifically say we want to use the same tokenizer that that model, TextDaVinci003, uses. And so, for example, when I earlier tried this, it talked about the frog splashing. And the result is a bunch of numbers. And what those numbers are, they're basically just lookups into a vocabulary that OpenAI, in this case, created.

And if I then decode those, it says, oh, these numbers are they, space, are, space, spool, ashing. And so put that all together, they are splashing. So you can see that the start of a word is, with a space before it, is also being encoded here.

How language models are trained: ULMFiT and fine-tuning

So these language models are quite neat. They can work at all. But they're not, of themselves, really designed to do anything.

The basic idea of what ChatGPT, GPT-4, BARD, et cetera, are doing comes from a paper which describes an algorithm that I created back in 2017 called ULMFiT. And Sebastian Ruder and I wrote a paper up describing the ULMFiT approach, which was the one that basically laid out what everybody's doing, how this system works. And the system has three steps. Step one is language model training. We actually described it as pre-training.

Now, what language model pre-training does is this is the thing which predicts the next word of a sentence. And so in the original ULMFiT paper, so the algorithm I developed in 2017, then Sebastian Ruder and I wrote it up in 2018, early 2018. What I originally did was I trained this language model on Wikipedia. Now, what that meant is I took a neural network, and a neural network is just a function. If you don't know what it is, it's just a mathematical function that's extremely flexible, and it's got lots and lots of parameters. And initially, it can't do anything. But using stochastic gradient descent, or SGD, you can teach it to do almost anything if you give it examples.

And so I gave it lots of examples of sentences from Wikipedia. So, for example, from the Wikipedia article for the birds. The birds is a 1963 American natural horror thriller film produced and directed by Alfred, and then it would stop. And so then the model would have to guess what the next word is. And if it guessed Hitchcock, it would be rewarded. And if it guessed something else, it would be penalized. And effectively, basically, it's trying to maximize those rewards. It's trying to find a set of weights for this function that makes it more likely that it would predict Hitchcock.

To do a good job of solving this problem as well as possible, of guessing the next word of sentences, the neural network is going to have to learn a lot of stuff about the world. It's going to learn that there are things called objects, that there's a thing called time, that objects react to each other over time. That there are things called movies, that movies have directors, that there are people, that people have names and so forth. And that a movie director is Alfred Hitchcock and he directed horror films and so on and so forth. It's going to have to learn an extraordinary amount if it's going to do a really good job of predicting the next word of sentences.

It's going to have to learn an extraordinary amount if it's going to do a really good job of predicting the next word of sentences.

Now, these neural networks specifically are deep neural networks. This is deep learning. And in these deep neural networks, which have, when I created this, I think it had like 100 million parameters. Nowadays, they have billions of parameters. It's got the ability to create a rich hierarchy of abstractions and representations, which it can build on.

And so this is really the key idea behind neural networks and language models. It's that if it's going to do a good job of being able to predict the next word of any sentence in any situation, it's going to have to know an awful lot about the world. It's going to have to know about how to solve math questions or figure out the next move in a chess game or recognize poetry and so on and so forth.

So the key idea here for me is that this is a form of compression. And this idea of the relationship between compression and intelligence goes back many, many decades. And the basic idea is that, yeah, if you can guess what words are coming up next, then effectively you're compressing all that information down into a neural network.

Now, I said this is not useful of itself. Well, why do we do it? Well, we do it because we want to pull out those capabilities. And the way we pull out those capabilities is we take two more steps. The second step is we do something called language model fine tuning. And in language model fine tuning, we are no longer just giving it all of Wikipedia. Or nowadays, we don't just give it all of Wikipedia. But in fact, a large chunk of the Internet is fed to pre-training these models. In the fine tuning stage, we feed it a set of documents a lot closer to the final task that we want the model to do. But it's still the same basic idea. It's still trying to predict the next word of a sentence.

After that, we then do a final classifier fine tuning. And in the classifier fine tuning, this is the kind of end task we're trying to get it to do. Now, nowadays, these two steps are very specific approaches are taken. For the step two, the step B, the language model fine tuning, people nowadays do a particular kind called instruction tuning. The idea is that the task we want most of the time to achieve is solve problems, answer questions. And so in the instruction tuning phase, we use data sets like this one. This is a great data set called OpenOrca created by a fantastic open source group. And it's built on top of something called the Flan collection.

And you can see that basically there's all kinds of different questions in here. So there's four gigabytes of questions. Here are some examples of instructions. I think this is from the Flan data set, if I remember correctly. So, for instance, it could be, does the sentence in the Iron Age answer the question, the period of time from 1200 to 1000 BCE is known as what? Choice is one, yes or no. And then the language model is meant to write one or two as appropriate for yes or no. So it's still doing language modeling. So fine tuning and pre-training are kind of the same thing. But this is more targeted now, not just to be able to fill in the missing parts of any document from the Internet, but to fill in the words necessary to answer questions, to do useful things.

OK, so that's instruction tuning. And then step three, which is the classifier fine tuning. Nowadays, there's generally various approaches, such as reinforcement learning from human feedback and others, which are basically giving humans or sometimes more advanced models, multiple answers to a question such as here are some from a reinforcement learning from human feedback paper. List five ideas for how to regain enthusiasm for my career. And so the model will spit out two possible answers or it will have a less good model and more good model. And then a human or a better model will pick which is best. And so that's used for the final fine tuning stage.

Using GPT-4 effectively

OK, so when we talk about a language model, we could be talking about something that's just been pre-trained, something that's been fine tuned or something that's gone through something like RLHF. All of those things are generally described nowadays as language models.

So my view is that if you are going to be good at language modelling in any way, then you need to start by being a really effective user of language models. And to be a really effective user of language models, you've got to use the best one that there is. And currently, so what are we up to? September 2023. The best one is by far GPT-4. This might change sometime in the not too distant future, but this is right now. GPT-4 is the recommendation, strong, strong recommendation.

Now, you can use GPT-4 by paying 20 bucks a month to OpenAI and then you can use it a whole lot. It's very hard to run out of credits, I find.

Now, what can GPT-4 do? It's interesting and instructive, in my opinion, to start with the very common views you see on the Internet or even in academia about what it can't do. So, for example, there was this paper you might have seen, GPT-4 can't reason, which describes a number of empirical analysis done of 25 diverse reasoning problems and found that it was not able to solve them. It's utterly incapable of reasoning.

So I always find you've got to be a bit careful about reading stuff like this, because I just took the first three that I came across in that paper and I gave them to GPT-4. And by the way, something very useful in GPT-4 is you can click on the share button and you'll get something that looks like this. And this is really handy.

So here's an example of something from the paper that said GPT-4 can't do this. Mabel's heart rate at 9 a.m. was 75 beats per minute. Her blood pressure at 7 p.m. was 120 over 80. She died 11 p.m. Was she alive at noon? Of course, she's a human. We know, obviously, she must be. And GPT-4 says, hmm, this appears to be a riddle, not a real inquiry into medical conditions. And, yeah, it sounds like Mabel was alive at noon. So that's correct.

This was the second one I tried from the paper that says GPT-4 can't do this. And I found actually GPT-4 can do this. Now, I mention this to say GPT-4 is probably a lot better than you would expect if you've read all this stuff on the internet about all the dumb things that it does. Almost every time I see on the internet saying something that GPT-4 can't do, I check it and it turns out it does.

This one was just last week. Sally, a girl, has three brothers. Each brother has two sisters. How many sisters does Sally have? So I have to think about it. And so GPT-4 says, OK, Sally is counted as one sister by each of her brothers. If each brother has two sisters, that means there's another sister in the picture apart from Sally. So Sally has one sister. Correct.

Why is it that people are claiming that GPT-4 can't do these things? Well, the reason is because I think on the whole they are not aware of how GPT-4 was trained. GPT-4 was not trained at any point to give correct answers. GPT-4 was trained initially to give most likely next words. And there's an awful lot of stuff on the internet where documents are not describing things that are true. There could be fiction. There could be jokes. There could be just stupid people saying dumb stuff. So this first stage does not necessarily give you correct answers.

The second stage with the instruction tuning, also, it's trying to give correct answers. But part of the problem is that then in the stage where you start asking people which answer do they like better, people tended to say in these things that they prefer more confident answers. And they often were not people who were trained well enough to recognize wrong answers. So there's lots of reasons that the SGD weight updates from this process for stuff like GPT-4 don't particularly or don't entirely reward correct answers.

But you can help it want to give you correct answers. If you think about the LM pre-training, what are the kinds of things in a document that would suggest, oh, this is going to be high quality information. And so you can actually prime GPT-4 to give you high quality information by giving it custom instructions. And what this does is this is basically text that is prepended to all of your queries. And so you say like, oh, you're brilliant at reasoning. So like, OK, that's obviously you have to prime it to give good answers. And then try to work against the fact that the RLHF folks preferred confidence. Just tell it. No, tell me if there might not be a correct answer.

Also, the way that the text is generated is it literally generates the next word. And then it puts all that whole lot back into the model and generates the next, next word, puts that all back in the model, generates the next, next, next word, and so forth. That means the more words it generates, the more computation it can do. And so I literally, I tell it that. Right. And so I say first spend a few sentences explaining background context, et cetera. So this custom instruction allows it to solve more challenging problems.

And you can see the difference. Here's what it looks like. For example, if I say, how do I get a count of rows grouped by value in pandas? And it just gives me a whole lot of information, which is actually it thinking. So I just skip over it and that gives me the answer. And actually in my custom instructions, I actually say if the request begins with VV, actually make it as concise as possible. And so it kind of goes into brief mode. And here is brief mode. How do I get the group? This is the same thing, but with VV at the start. And it just spits it out. Now, in this case, it's a really simple question. So I didn't need time to think.

So hopefully that gives you a sense of how to get language models to give good answers. You have to help them. And if you, if it's not working, it might be user error, basically. But having said that, there's plenty of stuff that language models like GPT-4 can't do. One thing to think carefully about is, does it know about itself? Can you ask it, what is your context length? How were you trained? What transformer architecture are you based on? At any one of these stages, did it have the opportunity to learn any of those things? Well, obviously not at the pre-training stage. Nothing on the Internet existed during GPT-4's training saying how GPT-4 was trained. Probably ditto in the instruction tuning, probably ditto in the RLHF. So in general, you can't ask, for example, a language model about itself.

Now, again, because of the RLHF, it'll want to make you happy by giving you opinionated answers. So it'll just spit out the most likely thing it thinks with great confidence. This is just a general kind of hallucination, right? So hallucinations is just this idea that the language model wants to complete the sentence and it wants to do it in an opinionated way that's likely to make people happy. It doesn't know anything about URLs. It really hasn't seen many at all. I think a lot of them, if not all of them, pretty much were stripped out. So if you ask it anything about, like, what's at this web page, again, it'll generally just make it up. And it doesn't know, at least GPT-4 doesn't know anything after September 2021 because the information it was pre-trained on was from that time period, September 2021 and before, called the knowledge cutoff.

Advanced data analysis and the OpenAI API

OK, now something else super helpful that you can use is what they call advanced data analysis. In advanced data analysis, you can ask it to basically write code for you. And we're going to look at how to implement this from scratch ourself quite soon. But first of all, let's learn how to use it. So I was trying to build something that split into markdown headings, a document on third level markdown headings. So that's three hashes at the start of a line. And I was doing it on the whole of Wikipedia. So using regular expressions was really slow. So I said, oh, I want to speed this up. And it said, OK, here's some code, which is great because then I can say, OK, test it and include edge cases. And so it then puts in the code, creates edge cases, tests it. And it says, yep, it's working.

However, I discovered it's not. I noticed it's actually removing the carriage return at the end of each sentence. So I said, oh, fix that and update your tests. So it said, OK. So now it's changed the test, updated the test cases. Let's run them. And oh, it's not working. So it says, oh yeah, fix the issue in the test cases. No, that didn't work. And you can see it's quite clever the way it's trying to fix it by looking at the results. But as you can see, it's not. Every one of these is another attempt, another attempt, another attempt until eventually I gave up waiting. And it's so funny each time it's like debugging again. OK, this time I've got to handle it properly. And I gave up at the point where it's like, oh, one more attempt. So it didn't solve it, interestingly enough.

And, you know, again, there's some limits to the amount of kind of logic that it can do. This is really a very simple question. I asked it to do for me. And so hopefully you can see you can't expect even GPT-4 code interpreter or advanced data analysis, as it's now called, to make it so you don't have to write code anymore. You know, it's not a substitute for having programmers.

Why would you use the OpenAI API rather than ChatGPT? Because you can do it programmatically. So you can, you know, you can analyze data sets, you can do repetitive stuff. It's kind of like a different way of programming. You know, it's things that you can think of describing. But let's just look at the most simple example of what that looks like. So if you pip install OpenAI, then you can import ChatCompletion. And then you can say, OK, ChatCompletion.create using GPT 3.5 Turbo. And then you can pass in a system message. This is basically the same as custom instructions. So, OK, you're an Aussie LLM that uses Aussie slang and analogies wherever possible.

And so you can see I'm passing in an array here of messages. So the first is the system message and then the user message, which is, what is money? OK. So GPT 3.5 returns a big embedded dictionary. And the message content is, well, my money is like the oil that keeps the machinery of our economy running smoothly. There you go. Just like a koala loves its eucalyptus leaves, we humans can't survive without this stuff. So there's the Aussie LLM's view of what is money.

Now, what happens when we are, this is really important to understand, when we have a follow up in the same conversation? How does that work? So we just asked what GOAT means. So, for example, Michael Jordan is often referred to as the GOAT for his exceptional skills and accomplishments. And Elvis and the Beatles are referred to as GOAT due to their profound influence and achievement. So I could say, what profound influence and achievements are you referring to? OK, well, I meant Elvis Presley and the Beatles did all these things. Now, how does that work? How does this follow up work? Well, what happens is the entire conversation is passed back.

And so we can actually do that here. So here is the same system prompt. Here is the same question. Right. And then the answer comes back with role assistant. And I'm going to do something pretty cheeky. I'm going to pretend that it didn't say money is like oil. I'm going to say, oh, you actually said money is like kangaroos. I thought, what is it going to do? OK, so you can like literally invent a conversation in which the language model said something different. Because this is actually how it's done in a multistage conversation. There's no state. Right. There's nothing stored on the server. You're passing back the entire conversation again and telling it what it told you. Right. So I'm going to tell it it's it told me that money is like kangaroos. And then I'll ask the user, oh, really? In what way? And it's kind of cool because you can like see how it convinces you of of something I just invented. Oh, let me break it down for you. But just like kangaroos hop around and carry their joeys in their pouch. Money is a means of carrying value around. So there you go. It's make your own analogy.

Building a code interpreter with function calling

So what else can we do? Well, let's create our own code interpreter that runs inside Jupyter. And so to do this, we're going to take advantage of a really nifty thing called function calling, which is provided by the OpenAI API. And in function calling, when we call our askGPT function, which is this little one here, we had room to pass in some keyword arguments that would be just passed along to check completion dot create. And one of those keyword arguments you can pass is functions. What on earth is that? Functions tells OpenAI about tools that you have, about functions that you have.

So, for example, I created a really simple function called sums. And it adds two things. In fact, it adds two ints. And I am going to pass that function to check completion dot create. Now, you can't pass a Python function directly. You actually have to pass what's called the JSON schema. So you have to pass the schema for the function. So I created this nifty little function that you're welcome to borrow, which uses Pydantic and also Python's inspect module to automatically take a Python function and return the schema for it. And so this is actually what's going to get passed to OpenAI. It's going to know that there's a function called sums, it's going to know what it does, and it's going to know what parameters it takes, what the defaults are, and what's required.

So this is like, when I first heard about this, I found this a bit mind bending, because this is so different to how we normally program computers. The key thing for programming the computer here actually is the docstring. This is the thing that GPT-4 will look at and say, oh, what does this function do? So it's critical that this describes exactly what the function does.

And so if I then say, what is 6 plus 3? And I just really wanted to make sure it actually did it here. So I gave it lots of prompts to say, because obviously it knows how to do it itself without calling sums. So it'll only use your functions if it feels it needs to, which is a weird concept. I mean, I guess feels is not a great word to use, but you kind of have to anthropomorphize these things a little bit because they don't behave like normal computer programs. So if I ask GPT, what is 6 plus 3, and tell it that there's a function called sums, then it does not actually return the number 9. Instead, it returns something saying, please call a function. Call this function and pass it these arguments. So if I print it out, there's the arguments. And so if I now say, OK, call the function that we got back, we finally get 9.

So this is a very simple example. It's not really doing anything that useful. But what we could do now is we can create a much more powerful function called Python. And the Python function executes code using Python and returns the result. So now I can say, ask GPT, what is 12 factorial? System prompt, you can use Python for any required computations and say, OK, here's a function you've got available. It's the Python function. So if I now call this, it will pass me back again a completion object. And here it's going to say, OK, I want you to call Python passing in this argument. And when I do, it's going to go import math, result equals blah, and then return result. Do I want to do that? Yes, I do. And there it is.

Now, there's one more step which we can optionally do. I mean, we've got the answer we wanted, but often we want the answer in more of a chat format. And so the way to do that is to, again, repeat everything that you've passed into so far. But then instead of adding an assistant role response, we have to provide a function role response. And simply put in here the result we got back from the function. And if we do that, we now get the prose response. 12 factorial is equal to 471,001,600.

Now, functions like Python, you can still ask it about non-Python things. And it just ignores it if you don't need it. So you can have a whole bunch of functions available that you've built to do whatever you need for the stuff which the language model isn't familiar with. And it'll still solve whatever it can on its own and use your tools, use your functions where possible.

OK, so we have built our own code interpreter from scratch. I think that's pretty amazing.

OK, so we have built our own code interpreter from scratch. I think that's pretty amazing.

Running open source models locally

So that is what you can do with some of the stuff you can do with OpenAI. What about stuff that you can do on your own computer? So then what we're going to be using is a library called Transformers from HuggingFace. And the reason for that is that basically people upload lots of pre-trained models or fine-tuned models up to the HuggingFace hub. And in fact, there's even a leaderboard where you can see which are the best models.

All right, so you can find models to try out from things like this leaderboard. And there's also a really great leaderboard called FastEval, which I like a lot, because it focuses on some more sophisticated evaluation methods, such as this chain of thought evaluation method. So I kind of trust these a little bit more. And these are also, you know, GSM 8K is a difficult math benchmark, BigBench Hard, and so forth. So, yeah, so, you know, StableBeluga2, WizardMath13B, DolphinLima13B, etc. These would all be good options.

Yeah, so you need to pick a model. And at the moment, nearly all the good models are based on Meta's Llama2. So when I say based on, what does that mean? Well, what that means is this model here, Llama2-7B. So it's a Llama model. That's just the name Meta called it. This is their version 2 of Llama. This is their 7 billion size one. It's the smallest one that they make. And specifically, these weights have been created for HuggingFace, so you can load it with the HuggingFace transformers. And this model has only got as far as here. It's done the language model for pre-training. It's done none of the instruction tuning and none of the RLHF. So we would need to fine tune it to really get it to do much useful.

So we can just say, OK, automatically create the appropriate model for language model. So causal LM basically refers to that ULMFiT stage 1 process or stage 2, in fact. So get the pre-trained model from this name, Meta Llama Llama2, etc. OK. Now, generally speaking, we use 16 bit floating point numbers nowadays. But if you think about it, 16 bit is 2 bytes. So 7B times 2, it's going to be 14 gigabytes just to load in the weights. So you've got to have a decent model to be able to do that. Perhaps surprisingly, you can actually just cast it to 8 bit and it still works pretty well, thanks to something called discretization.

So remember, this is just a language model that can only complete sentences. We can't ask it a question and expect a great answer. So let's just give it the start of a sentence. Jeremy Howard is R. And so we need the right tokenizer, so this will automatically create the right kind of tokenizer for this model. We can grab the tokens as PyTorch. Here they are. And just to confirm, if we decode them back again, we get back the original plus a special token to say this is the start of a document. And so we can now call generate. So generate will autoregressively, so call the model again and again, passing its previous result back as the next as the next input. And I'm just going to do that 15 times.

So this is you can you can write this for loop yourself. This isn't doing anything fancy. In fact, I would recommend writing this yourself to make sure that you know how that it all works OK. We have to put those tokens on the GPU. And at the end, I recommend putting them back onto the CPU, the result. And here are the tokens. Not very interesting. So we have to decode them using the tokenizer. And so the first 25, sorry, first 15 tokens are Jeremy Howard Izhar, 28 year old Australian AI researcher and entrepreneur. OK, well, 28 years old is not exactly correct, but we'll call it close enough. I like that. Thank you very much. Llama7b.

So, OK, so we've got a language model completing sentences. It took one and a third seconds. And that's a bit slower than it could be because we used 8-bit. If we use 16-bit, there's a special thing called bfloat16, which is a really great 16-bit floating point format that's usable on any somewhat recent NVIDIA GPU. Now, if we use it, it's going to take twice as much RAM as we discussed. But look at the time. It's come down to 390 milliseconds.

Now, there is a better option still than even that. There's a different kind of discretization called GPTQ, where a model is carefully optimized to work with 4 or 8 or other, you know, lower precision data automatically. And this particular person, known as The Bloke, is fantastic at taking popular models, running that optimization process, and then uploading the results back to HuggingFace. So we can use this GPTQ version. And internally, this is actually going to use, I'm not sure exactly how many bits this particular one is. I think it's probably going to be 4 bits. But it's going to be much more optimized. And so look at this. 270 milliseconds. It's actually faster than 16-bit. Even though internally it's actually casting it up to 16-bit each layer to do it. And that's because there's a lot less memory moving around. And to confirm, in fact, what we can even do now is we can go up to 13-bit. Easy. And in fact, it's still faster than the 7-bit, now that we're using the GPTQ version. So this is a really helpful tip.

So let's put all those things together. The tokenizer, the generate, the batch decode. We'll call this gen for generate. And so we can now use the 13-bit GPTQ model. And let's try this. Jeremy Howard is a, so it's got to 50 tokens, so fast. 16-year veteran of Silicon Valley. Co-founder of Kaggle, a marketplace for predictive model. His company, Kaggle.com, has become a data science competition. I don't know what I was going to say. But anyway, it's on the right track. I was actually there for 10 years, not 16. But that's all right.

So this is looking good. But probably a lot of the time we're going to be interested in, you know, asking questions or using instructions. So Stability AI has this nice series called Stable Beluga, including a small 7B one and other bigger ones. And these are all based on Llama 2, but these have been instruction tuned. They might even have been RLHFed. I can't remember now. So we can create a Stable Beluga model.

And now something really important that I keep forgetting, everybody keeps forgetting, is during the instruction tuning process. During the instruction tuning process, the instructions that are passed in actually are. They don't just appear like this. They actually always are in a particular format. And the format, believe it or not, changes quite a bit from fine tune to fine tune. And so you have to go to the web page for the model. And scroll down to find out what the prompt format is. So here's the prompt format. So I generally just copy it. And then I paste it into Python. And created a function called make prompt that used the exact same format that it said to use.

And so now if I want to say who is Jeremy Howard, I can call gen again. That was that function I created up here. And make the correct prompt from that question. And then it returns back. Okay, so you can see here all this prefix. This is the system instruction. This is my question. And then the assistant says, Jeremy Howard is an Australian entrepreneur, computer scientist, co-founder of machine learning and deep learning company Fast.ai. Okay, this one's actually all correct. So it's getting better by using an actual instruction tune model.

And so we could then start to scale up. So we could use the 13b. And in fact, we looked briefly at this OpenOrca data set earlier. So Llama2 has been fine-tuned on OpenOrca. And then also fine-tuned on another really great data set called Platypus. And so the whole thing together is the OpenOrca Platypus. And then this is going to be the bigger 13b. GPTQ means it's going to be quantized. So that's got a different format. Okay, a different prompt format. So again, we can scroll down and see what the prompt format is.

We can create a function called makeOpenOrcaPrompt that has that prompt format. And so now we can say, okay, who is Jeremy Howard? And now I've become British, which is kind of true. I was born in England, but I moved to Australia. Professional poker player? Definitely not that. Co-founding several companies, including Fast.ai. Also Kaggle. Okay, so not bad. It was acquired by Google. Was it 2017? Probably something around there. Okay. So you can see we've got our own models giving us some pretty good information.

Retrieval augmented generation

How do we make it even better? You know, because it's still hallucinating, you know. And, you know, Llama2, I think, has been trained with more up-to-date information than GPT-4. It doesn't have the September 2021 cutoff. But, you know, it's still got a knowledge cutoff. You know, we would like to be able to use the most up-to-date information. We want to use the right information to answer these questions as well as possible. So to do this, we can use something called Retrieval Augmented Generation.

So what happens with Retrieval Augmented Generation is when we take the question we've been asked, like, who is Jeremy Howard? And then we say, okay, let's try and search for documents that may help us answer that question. So obviously, we would expect, for example, Wikipedia to be useful. And then what we do is we say, okay, with that information, let's now see if we can tell the language model about what we found and then have it answer the question.

So let's actually grab a Wikipedia Python package. We will scrape Wikipedia, grabbing the Jeremy Howard web page. And so here's the start of the Jeremy Howard Wikipedia page. It has 613 words. Now, generally speaking, these open source models will have a context length of about 2000 or 4000. So the context length is how many tokens can it handle. So that's fine. It'll be able to handle this web page. And what we're going to do is we're going to ask it the question. So we're going to have here a question and with a question. But before it, we're going to say answer the question with the help of the context. We're going to provide this to the language model. And we're going to say context. And they're going to have the whole web page. So suddenly now our question is going to be a lot bigger. Our prompt. Right. So our prompt now contains the entire web page, the whole Wikipedia page, followed by a question.

And so now it says, Jeremy Howard is an Australian data scientist, entrepreneur and educator, known for his work in deep learning, co-founder of Fast.ai, teaches courses, develops software, conducts research, used to be. Yeah. OK. It's perfect. Right. So it's actually done a really good job. Like if somebody asked me to send them a, you know, 100 word bio, that would actually probably be better than I would have written myself. And you'll see, even though I asked for 300 tokens, it actually got sent back the end of stream token. And so it knows to stop at this point.

Well, that's all very well, but how do we know to pass in the Jeremy Howard Wikipedia page? Well, the way we know which Wikipedia page to pass in is that we can use another model to tell us which web page or which document is the most useful for answering a question. And the way we do that is we can use something called sentence transformer and we can use a special kind of model that's specifically designed to take a document and turn it into a bunch of activations where two documents that are similar will have similar activations.

What I'm going to do is I'm going to grab just the first paragraph of my Wikipedia page and I'm going to grab the first paragraph of Tony Blair's Wikipedia page. OK, so we're pretty different people. Right. This is just like a really simple, small example. And I'm going to then call this model, I'm going to say encode, and I'm going to encode my Wikipedia first paragraph, Tony Blair's first paragraph, and the question, which was, who is Jeremy Howard? And it's going to pass back a 384 long vector of embeddings for the question, for me, and for Tony Blair. And what I can now do is I can calculate the similarity between the question and the Jeremy Howard Wikipedia page. And I can also do it for the question versus the Tony Blair Wikipedia page. And as you can see, it's higher for me. And so that tells you that if you're trying to figure out what document to use to help you answer this question, better off using the Jeremy Howard Wikipedia page than the Tony Blair Wikipedia page.

So if you had a few hundred documents you were thinking of using to give back to the model as context to help it answer a question, you could literally just pass them all through to encode, go through each one, one at a time, and see which is closest. When you've got thousands or millions of documents, you can use something called a vector database, where basically as a one-off thing, you go through and you encode all of your documents.

And so in fact, there's lots of pre-built systems for this. Here's an example of one called H2O-GPT. And this is just something that I've got running here on my computer. It's just an open source thing written in Python, sitting here running on port 7860. And so I've just gone to localhost 7860. And what I did was I just uploaded, I just clicked upload, and uploaded a bunch of papers. A bunch of papers. And so, you know, we could look at ... Can we search? Yeah, I can. So for example, we can look at the ULMFiT paper that Sebastian Ruder and I did. And you can see it's taken the PDF and turned it into, slightly crappily, a text format. And then it's created an embedding for each section.

So I could then ask it, you know, what is ULMFiT? And I'll hit enter. And you can see here it's now actually saying, based on the information provided in the context. So it's showing us, it's been given some context. What context did it get? So here are the things that it found. Right? So it's being sent this context. So this is kind of citations. Goal of ULMFiT. Improves the performance by leveraging the knowledge and adapting it to the specific task at hand.

How does ULMFiT improve performance? Or maybe I should say, what techniques? Be more specific. Does ULMFiT. Let's see how it goes. Okay. There we go. So here's the three steps. Pre-train. Fine-