Introduction to Gorilla LLM Large Language Model

🦍 Gorilla: Large Language Model Connected with Massive APIs
Link: https://gorilla.cs.berkeley.edu/blogs/7_open_functions_v2.html

• Berkeley function call rankingsBerkeley function call rankings
• Online experience model:Gorilla OpenFunctions-v2 web demo
• Project details:GitHub
• Model (7B parameters) page on HuggingFace:gorilla-llm/gorilla-openfunctions-v2

1. Berkeley function call rankings

Since late 2022, large language models (LLMs) have been in the spotlight due to their powerful ability to perform general-purpose tasks. Not limited to chat applications, it has become a trend to apply these models to the development of various AI applications and software (such as Langchain, Llama Index, AutoGPT, Voyager). Models such as GPT, Gemini, Llama, Mistral and others have demonstrated their great potential through interaction with the external world, such as function calls and execution.

we launchedBerkeley Function Call Ranking (BFCL), which is the first comprehensive and executable evaluation of function calls in LLMs. Unlike previous evaluations such as the Anyscale function call dataset, we consider more forms of function calls, calls in different scenarios, and the executability of function calls. We built this dataset based on real-world application scenarios, covering most function call use cases that users are likely to encounter, such as applications in AI agents or enterprise workflows. To this end, our evaluation dataset contains rich categories covering multiple languages. At the same time, we also released the Gorilla-Openfunctions-v2 model, which is currently the most advanced open source model capable of handling function calls in multiple programming languages, including parallel and multiple function calls. In addition, we provide a special debugging feature where the model outputs an "error message" when the provided function does not meet the task requirements.

https://gorilla.cs.berkeley.edu/leaderboard#api-explorer

Berkeley function call rankings 🏆

Berkeley function call

The ranking list (BFCL) aims to comprehensively study the performance of different LLMs in function calling capabilities. It contains 2000 question-function-answer pairs in multiple programming languages ​​(Python, Java, JavaScript, REST API), covering diverse application areas and complex use cases (such as multiple function calls and parallel function calls). We also looked at function dependence detection to determine how the model reacts to poorly fitted functions (in which case an "error message" is provided). Specifically, BFCL includes various simple, parallel, multiple, executable function call scenarios and function dependency detection for 100 Java, 50 JavaScript, 70 REST API, 100 SQL, and 1680 Python.

The leaderboard shows that OpenAI's GPT-4 still leads in function call evaluation, while Gorilla OpenFunctions-v2 (from Gorilla LLM) performs almost as well. This is followed by the Mistral-medium model (from Mistral AI) and Claude-2.1 (from Anthropic). This shows that a fine-tuned open source model can achieve a level similar to that of a proprietary model on function calling tasks without the need for complex linking.

We strive to cover real-world use cases and diverse languages. In the future, we will continue to expand the testing area and explore more innovative use cases.

Performance of LLMs on the Berkeley Function Call Ranking (BFCL)

For deeper analysis and visualization of results, we provide an interactive hexagon tool for users to compare the performance of different models. We divided the tests into 9 categories, including function irrelevance detection, AST tree checking, and execution function call checking, for simple, multiple, parallel multifunctional scenarios. Through this tool, we can clearly see the performance of the model in each test. In terms of simple single function calls, the proprietary and open source models perform similarly. But when it comes to multiple and parallel function calls, the GPT family of models outperforms open source models.

Detailed analysis using the Berkeley Function Call Ranking (BFCL) hexagonal plot

Data set composition

The evaluation dataset for Gorilla OpenFunctions has been expanded from an initial 100 entries to 1900. The evaluation dataset exhibits diversity in:

• function documentation area
• Number of function documentation and function call Q&A pairs
• Data types in different programming languages

Our evaluation JSON function is scraped and generated from different website sources. We have deliberately included areas like math algebra, sports football, financial mortgages, etc. We include 40 subdomain functions in our evaluation, which allows us to understand how model performance performs in data-rich domains such as computing and cloud, as well as in niche domains such as sports and law.

Berkeley Function Call Ranking (BFCL) data composition

Assessment category 📊

We divide assessments into two main categories:

• Python: simple functions, multiple functions, parallel functions, parallel multiple functions
• Non-Python: Function dependency detection, REST API, SQL, Java, JavaScript

Python evaluation

• Simple function:This category covers the most common format: the user provides a JSON function document, and the model just calls one of the functions.
• Multiple functions:This category contains user questions that require selecting and calling a function from 2-4 JSON function documents. The model needs to choose the most appropriate function based on the context provided by the user.
• Parallel function:Parallel functions are defined as calling multiple functions simultaneously using one user query. The model needs to determine how many functions need to be called, and the question can be a single or multiple sentences.
• Parallel multiple functions:Parallel multiple functions are a combination of parallel functions and multiple functions, that is, the model is provided with multiple function documents, and the functions in each document may be called once or multiple times.

Each category has a corresponding executable category. In this section, we coded functions based on some free REST API endpoints (e.g., getting the weather) and directly calculated functions (e.g., linear regression). The executable category is designed to determine whether the function call generation can be used in actual applications.

Non-Python evaluation

In addition to the above main categories, we also included more specific categories to evaluate the model's performance in different scenarios and test its ability to cope with irrelevant questions and function documentation.

• Function Dependency Detection: This category designs a scenario where any function provided is not relevant and should not be called. We expect the model's output to be without function calls. This scenario helps us understand whether the model will produce errors when it lacks information to generate function code.
• REST API: Most API calls in the real world are REST API calls. Python mainly completes REST API calls in the python requests library through requests.get(), requests.post(), requests.delete() and other methods.
• GET request:GET requests are the most commonly used in the real world. Therefore, we included real-world GET requests to test the model's ability to generate executable REST API calls. Our evaluation includes two variants: one that requires parameters to be passed in the URL, and one that requires parameters to be put as key/value pairs in the params and/or headers of requests.get(). The model needs to decide how to call it based on the situation.
• SQL：SQL evaluation data includes our custom sql.execute function, which contains sql_keyword, table_name, columns and conditions. These parameters provide the necessary information to build a simple SQL query. Rather than specifically training a SQL model, we want to reliably build and use SQL queries via function calls. Our evaluation dataset limits the scenario to only support simple keywords such as “SELECT”, “INSERT INTO”, “UPDATE”, “DELETE”, “CREATE”.
• Java + JavaScript：Although most programming languages have the same function call format, each programming language has its own unique types. For example, C has pointer types and Java has HashMap types. The purpose of this test category is to understand how the function call model extends to not just JSON and Python types, but all language-specific types.

These categories allow us to see how different models perform in popular use cases for API calls and provide us with insights into the potential of function call models.

Evaluation Metrics 📈

We use two popular methods to evaluate the accuracy of model-generated answers: the AST checker and the execution checker. Ideally, an execution checker should be used, but since not all results are easy to execute (such as Java functions), we supplement the execution checker with an AST.

• Abstract syntax tree (AST) checker
• execution checker

AST inspection: For executable function call answers, we use an AST tree for parsing.

Example:[calculate_triangle_area(base=10, height=5)]

Analysis:Module(body=[Expr(value=List(elts=[Call(func=Name(id='calculate_triangle_area', ctx=Load()), args=[], keywords=[keyword(arg='base', value=Constant(value=10)), keyword(arg='height', value=Constant(value=5))])], ctx=Load()))], type_ignores=[]) [calculate_triangle_area(base=10, height=5)]

We extract the variables from the AST and check if each parameter is found among the possible answers and matches exactly. For each possible answer, the answers that should be accepted include:

• Boolean value:
  • We check for direct matches of Boolean values, disallowing tolerance for string versions of Boolean values.
• Integer, floating point number:
  • The answer should be unique, for example [1]
• List:
  • We check for exact matches, so any order of lists should match. [1,2,3]==[2,3,1]
• Dictionary:
  • For simplicity, we skip checking the recursive AST dictionary structure.
• String:
  • Possible dates “20th June”, “2023-06-20”, “06/20/2023”, “Jun.20,2023”
  • Possible locations ["New York City", "NYC"]
  • Anything possible ["Manchester United", "Man United", "Man U", "MUFC"]

Here are some examples of possible answers:

• {"calculate_triangle_area": {"base": [10], "height": [5], "unit": ["units", "unit"]}}
• {"predict_house_price": {"bedrooms": [3], "bathrooms": [2], "area": [1800], "location": ["San Francisco", "San Francisco, CA"]}}

This checking mechanism works for everything except executable_* and REST.

Executable checks: For executable_* and REST, we have corresponding functions that can be executed for each problem. Therefore, after the model generates answers, we will directly execute these answers. There are two types of matches:

• Deterministic executable output: We check for exact matches against the results of our human execution.
• Non-deterministic and real-world relevant executable output: We check the consistency of its response type and response JSON key to see if the value is what we expect to see.

Tips

We provide all the tips for evaluating our proprietary and open source models. For the function call model, we do not provide any system prompts, but directly enable the function call mode and place the function definition. For the chat model, we provide explicit system messages.

1. For all function call models, we directly enable function call mode and place the function definition.

2. For the chat model, we provide explicit system messages:

   1 2 3 4 5 6 7 8 9 10 11 12 13

   SYSTEM_PROMPT_FOR_CHAT_MODEL = """" 你是一个编写函数的专家。你会收到一个问题和一系列可能的函数。 根据问题，你需要进行一个或多个函数/工具调用来实现目的。 如果没有一个函数可以使用，请指出。如果给定问题缺少函数所需的参数， 也请指出。你应该只在工具调用部分返回函数调用。 """ SYSTEM_PROMPT_FOR_CHAT_MODEL = """" You are an expert in composing functions. You are given a question and a set of possible functions. Based on the question, you will need to make one or more function/tool calls to achieve the purpose. If none of the function can be used, point it out. If the given question lacks the parameters required by the function, also point it out. You should only return the function call in tools call sections. """

   1 2

   USER_MESSAGE_FOR_CHAT_MODEL = "Questions:{user_prompt}\\n这里是一系列你可以调用的JSON格式函数列表:\\n{functions}. 如果你决定返回函数调用，不得包含其他文本。" USER_MESSAGE_FOR_CHAT_MODEL = "Questions:{user_prompt}\nHere is a list of functions in JSON format that you can invoke:\n{functions}. Should you decide to return the function call(s), NO other text MUST be included."

Common mistakes

With our benchmark BFCL, we were able to identify some common mistakes LLMs make when generating function calls. These errors reveal the limitations of the current model and provide insights into how to improve it

They provide insight.

1. GPT's function documentation is difficult to format and its types are limited in real-world scenarios. For example, we need to manually convert float to number to make the function compatible with OpenAI. Additionally, numbers convey less information about precision and type consistency than floats.

   In Gorilla Openfunctions-v2, we improve the flexibility of function documentation by not limiting parameter types. In other words, users can provide Tuple, Float, or even specific types in Java such as HashMap and LinkedList.

2. GPT does not perform well in parameter scenarios that require some kind of implicit conversion. For example, when the parameters are not given directly in the user question.

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

   "Function": { "name": "finance.predict_future_value", ... "parameters": { "type": "object", "properties": { "present_value": { "type": "number", "description": "The present value of the investment." }, "annual_interest_rate": { "type": "number", "description": "The annual interest rate of the investment." }, "compounding_periods_per_year": { "type": "integer", "description": "The number of times that interest is compounded per year.", }, "time_years": { "type": "integer", "description": "The investment horizon in years." } ... } "required": ["present_value", "annual_interest_rate", "time_years"] } }

   Questions : Predict the future value of a $5000 investment with an annual interest rate of 5% in 3 years with monthly compounding.

   1 2 3 4 5 6 7 8 9 10 11

   GPT-4 output: [{ "name": "finance.predict_future_value", "parameters": { "present_value": 5000, "annual_interest_rate": 5, "compounding_periods_per_year": 12, "time_years": 3 } }]

   1 2 3 4 5 6 7 8 9 10 11

   Gorilla-openfunctions-v2 output: [{ "name": "finance.predict_future_value", "parameters": { "present_value": 5000, "annual_interest_rate": 0.05, "compounding_periods_per_year": 12, "time_years": 3 } }]

3. Chat models tend to generate malformed function calls where arguments can be extracted but cannot be executed.

   An example of the results generated by mistral-medium is as follows:solve_quadratic_equation(a=2, b=6, c=5). Through gorilla-openfunctions-v2, we can directly outputsolve_quadratic_equation(a=3, b=2, c=1), the result can be executed after receiving it.

4. REST API call inconsistencies: For example, in some cases the model may not correctly generate URLs or parameters for API calls.

Conclusion

We conducted a comprehensive and systematic evaluation of LLMs function calls via the Gorilla Open Functions leaderboard. Research shows that fine-tuned open source models can compete with proprietary models on simple function calls that do not involve complex planning and chained function calls. In addition, we launched Gorilla Open Functions v2, an open source model that helps users build AI applications through function calls and achieve json-compatible output interaction.

We hope you enjoyed this blog post. Welcome toDiscord、Twitter (#GorillaLLM)andGitHubShare your thoughts on.

If you want to quote Gorilla:

1
2
3
4
5
6

@misc{berkeley-function-calling-leaderboard,
  title={Berkeley Function Calling Leaderboard},
  author={Fanjia Yan and Huanzhi Mao and Charlie Cheng-Jie Ji and Tianjun Zhang and Shishir G. Patil and Ion Stoica and Joseph E. Gonzalez},
  howpublished={\url{https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html}},
  year={2024},
}

2. Gorilla OpenFunctions v2

Gorilla OpenFunctions-v2! Technology leadership in the open source model (SoTA), comparable to commercial models.

The latest version of Gorilla OpenFunctions, version 2, brings significant progress in large language models (LLM) for function calls in the open source community. As an upgraded replacement for the previous version, Gorilla OpenFunctions-v2 not only maintains the open source spirit, but also introduces exciting new features. This includes support for Python, Java, JavaScript, REST APIs, and more

Multiple programming languages—a first in both open and closed source models; the ability to handle multiple and parallel function calls simultaneously, and the ability to determine function dependencies. This update solidifies gorilla-openfunctions-v2’s leading position in function calling capabilities in the LLM field. And, this plug-and-play update approach allows OpenFunctions to be easily integrated into a variety of applications, from social media platforms like Instagram to delivery services like Doordash, as well as utilities including Google Calendar and Stripe.

A quick look at new features!! 🚀

The five exciting new features we are introducing in OpenFunctions-v2 include:

• Supports more data types: Gorilla Open Functions v2 now supports multiple languages and has expanded support for parameter types in function calls. For example, for Python, supported types include[string, number, boolean, list, tuple, dict, any]; Java and Javascript also support rich types. In contrast, OpenAI and many others only support JSON schema, i.e.[string, number, integer, object, array, boolean]. This native support for types means you can now use openfunctions-v2 more easily.
• Supports parallelism and multi-function: Can handle parallel and multi-function calls. In a multifunctional scenario, the user can enter multiple features when unsure which feature is most appropriate; the Gorilla model will select one or more of them (or none) in response to the user's request. In a parallel function, you can respond to user prompts by calling the same function multiple times. The Gorilla model not only supports both modes but also combines their advantages.
• Functional dependency detection: Reduce error responses when functionality or related functionality is not provided. Gorilla openfunctions v2 now automatically determines whether the functions provided to the model solve the user's problem. After recognizing this, LLM will display an error message to the user to provide more help.
• Enhanced RESTful API capabilities: Improved ability to format RESTful API calls. RESTful APIs are widely used across the web, powering many popular software services such as Slack, PayPal, and more. Our models are specially trained to handle RESTful API calls with high quality.

Quick links:

• Performance of other function call models:Berkeley function call rankings
• Online experience model:Gorilla OpenFunctions-v2 web demo
• Project details:GitHub
• Model (7B parameters) page on HuggingFace:gorilla-llm/gorilla-openfunctions-v2

Integrate OpenFunctions-v2 in your app 🔨

Using Gorilla OpenFunctions-v2 is very simple:

1. To facilitate rapid prototyping, we provide a hosted Gorilla Openfunctions-v2 model for inference use. You can also run it locally, or self-host it via HuggingFace’s page. The following example shows how to call a managed gorilla openfunctions v2 model:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

import openai

def get_gorilla_response(prompt="", model="gorilla-openfunctions-v2", functions=[]):
    openai.api_key = "EMPTY"  # 由UC Berkeley免费托管 ❤️
    openai.api_base = "<http://luigi.millennium.berkeley.edu:8000/v1>"
    try:
    completion = openai.ChatCompletion.create(
        model="gorilla-openfunctions-v2",
        temperature=0.0,
        messages=[{"role": "user", "content": prompt}],
        functions=functions,
    )
    # completion.choices[0].message.content, 函数调用的字符串格式
    # completion.choices[0].message.functionsl, 函数调用的Json格式
    return completion.choices[0]

1. Ask the model:
   波士顿和旧金山的天气怎么样？
2. Format your function calls: The model will return function calls based on your request.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

query = "波士顿和旧金山的天气怎么样？"
functions = [
    {
        "name": "get_current_weather",
        "description": "获取指定地点的当前天气",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "城市和州，比如旧金山，CA",
                },
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["location"],
        },
    }
]

1. Get your function call: The model will return a Python function call based on your request.
   This provides convenience to both developers and non-developers, allowing them to take advantage of complex functionality without writing extensive code.

Input:

1

get_gorilla_response(prompt=query, functions=[functions])

Output:

1

[get_current_weather(location='Boston, MA'), get_current_weather(location='San Francisco, CA')]

With the above example, you can leverage Gorilla OpenFunctions-v2 to generate well-formatted output, or call functions with your own definitions! You can then freely use these features in your apps and chatbots!

Note: Gorilla currently only supportsopenai==0.28.1version of the managed endpoint. We will soon be upgrading to supportopenai==1.xxversion, by thenfunctionswill betool_callsReplace.

Performance on the Berkeley function call leaderboard 🔥

We conduct a comprehensive and exhaustive evaluation on the Berkeley function call leaderboard, comparing our model against the state-of-the-art GPT-4-1106 preview as well as GPT-4 and GPT-3.5-turbo function call features. Furthermore, we compare our model with other open source models and demonstrate its superior performance. Our evaluation covered over 2000 different query and API document pairs from different domains (including travel, finance, scheduling meetings, etc.) and languages ​​(java, javascript, python, restAPI).

For an in-depth look at how our models performed in each category, see the detailed table below on the Berkeley feature call rankings. Compared to the current state-of-the-art GPT-4 function calls, Gorilla OpenFunctions-v2 performs better on simple function call categories in Python, but does not perform as well as GPT-4 on function calls involving multiple and parallel functions. This new feature remains an exciting area of ​​research for us and the entire open source community. It is worth mentioning that our model provides very stable executable function calls - these function calls are evaluated through actual execution without any intervention. As expected, the trained Gorilla model outperformed GPT-4 on function calls in programming languages ​​other than Python, such as Java, Javascript, and REST APIs. For REST APIs, our model provides a more stable output that includes all required fields, includingurl、paramsandheader, making our model ideal for immediate adoption.

The left side is generated by GPT-4, and the right side is generated by openfunctions-v2. As you can see from the error above, when GPT-4 function calls handle functions involving complex argument structures (such as dictionaries within dictionaries) with default values, the model tends to have trouble, especially in parsing the default values. Rather than being an edge case, the above example is a common paradigm for REST APIs.

OpenFunctions data composition and training 🍕

Gorilla openfunctions v2 is a 7B parameter model further trained based on the \[deepseek-coder-7b-instruct-v1.5\] large language model. To train the model, we collected a total of 65,283 question-function-answer pairs from three different sources: Python packages (19,353), Java repositories (16,586), Javascript repositories (4,245), public APIs (6,009), and command line tools from various cloud providers (19,090). The data composition is shown in the figure below.

After data collection, we performed four data augmentations to increase the diversity of our training dataset. First, we changed the function names, which is crucial to ensure that the model doesn't "remember" the API mapping. Second, we added a randomly selected, varying number of functions to make our dataset compatible with parallel functions. This way we can generate multifunctional datasets from simple functions. Third, we adopt the strategy of perturbation prompts to generate scenarios of parallel functions and extend it to include both multifunctional and parallel functions. Finally, we also include parts of the dataset where the features at input time are insufficient to solve the task, and we label these as “dependency detection” scenarios. As with most LLM training, we varied widely the degree of each data augmentation to train a robust model.

• Function name transformation: We augment the original question-feature-answer pairs by using different function names to avoid the model remembering the correlation between the function name and the question (e.g., the “uber” API for transportation).
  query + [{'name': 'func1', 'description': 'order takeout'}] -> ans1 => query + [{'name': 'func2', 'description': 'order takeout'}] -> [ans2]
• Parallel function transformation: To handle the more complex case of selecting multiple features to answer a user request, we changed the original question to ask for multiple outputs.
  query + [{'name': 'func1', 'description': 'order takeout'}] -> ans1 => query + [{'name': 'func1', 'description': 'order takeout'}, {'name': 'func2', 'description': 'get weather'}] -> [ans1]
• Multifunctional transformation: Including raw function transformations for multiple function calls in training allows the model to learn to choose which function call to use.
  query1 + [{'name': 'func1', 'description': 'order takeout'}] -> ans1 => query2 + [{'name': 'func1', 'description': 'order takeout'}] -> [ans1, ans2]
• Parallel multi-function transformation: A combination of the above parallel and multifunctional transformations.
  query1 + [{'name': 'func1', 'description': 'order takeout'}] -> ans1 => query2 + [{'name': 'func1', 'description': 'order takeout'}, {'name': 'func2', 'description': 'get weather'}] -> [ans1, ans2]
• Functional dependency detection transformation: We also included some portions of the dataset for which the features provided at input time could not solve the task. We call this "correlation detection".
  query1 + [{'name': 'func1', 'description': 'order takeout'}] -> ans1 => query2 + [{'name': 'func1', 'description': 'order takeout'}] -> [Error, the function cannot solve the question.]

After the entire data augmentation, we also performed data deduplication using Rouge score, which has become standard practice.

Conclusion

We are happy to publishgorilla-openfunctions-v2, which is a 7B parameter model trained on the Deepseek-Coder-7B-Instruct-v1.5 large language model. It takes prompts from the user and multiple API calls and returns a function with the correct parameters. OpenFunctions extends native support for Python, Java, and JavaScript, as well as parameter types in RESTful APIs. For more information, check out our blog evaluation on the Berkeley feature call rankings, and the model on our GitHub page. All results in the blog are generated usinggorilla-openfunctions-v2Generated.

Licensing:

Gorilla OpenFunctions v2 is distributed under the Apache 2.0 license. This software incorporates elements from the Deepseek model. Consequently, the licensing of Gorilla OpenFunctions v2 adheres to the Apache 2.0 license, with additional terms as outlined in Appendix A of the Deepseek license.

We hope you enjoyed this blog post. Welcome toDiscord, Twitter (#GorillaLLM) andGitHubShare your thoughts with us on .

If you want to quote Gorilla:

1
2
3
4
5
6

@inproceedings{gorilla-openfunctions-v2,
  title={Gorilla OpenFunctions v2},
  author={Charlie Cheng-Jie Ji, Huanzhi Mao, Fanjia Yan, Shishir G. Patil, Tianjun Zhang, Ion Stoica, Joseph E. Gonzalez},
  year={2024},
  howpublished={\url{https://gorilla.cs.berkeley.edu//blogs/7_open_functions_v2.html}},
}