Llama 2

Overview

Llama 2 is a collection of pretrained and fine-tuned generative text models ranging in size from 7 billion to 70 billion parameters. The Llama2 model was developed by Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert and others inLlama 2: Open Foundation and Fine-Tuned Chat Modelsproposed in.

The abstract of the paper is as follows:

In this work, we develop and release Llama 2, a set of pretrained and fine-tuned large language models (LLMs) ranging from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for conversational use cases. Our model outperforms open source chat models on most benchmarks we tested and may be a suitable alternative to closed source models based on our human evaluations of usefulness and safety. We provide a detailed description of our approach to fine-tuning and improving the security of Llama 2-Chat so that the community can build on our work and contribute to the responsible development of LLMs.

View all Llama2 models here

Tips:

• by filling inthis formThe weights of the Llama2 model can be obtained
• The architecture is very similar to the first Llama, with the addition of Groupe Query Attention (GQA)this paperafter
• willconfig.pretraining_tpSetting to a value different from 1 will activate a more accurate but slower calculation of the linear layer, which should better match the original logits.
• original model usepad_id = -1, which means the token is not populated. We cannot use the same logic, make sure to usetokenizer.add_special_tokens({"pad_token":"<pad>"})Add padding tokens and adjust the token embed size accordingly. You should also setmodel.config.pad_token_id. The embed_tokens layer of the model is usedself.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.config.padding_idx)On initialization, making sure to encode the padding token will output zero, so passing it on initialization is recommended.
• After filling out the form and gaining access to model checkpoints, you should be able to use the converted checkpoints. Otherwise, if you are converting your own model, feel free to use the conversion script. The script can be called using the following (example) command:

  1 2	python src/transformers/models/llama/convert_llama_weights_to_hf.py \ --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path

Model details

NOTE: Use of this model is subject to the Meta License. In order to download the model weights and tokenizer, please visit the website and accept the license before requesting access.

Meta develops and publicly releases the Llama 2 series of large language models (LLMs), a series of pretrained and fine-tuned generative text models ranging in size from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama-2-Chat, are optimized for conversational application scenarios. The Llama-2-Chat model outperforms open-source chat models on most benchmarks we tested, and is on par with some popular closed-source models such as ChatGPT and PaLM in terms of usefulness and security in our human evaluations.

model developer

Model Developers Meta

different versions

Llama 2 is available in parameter versions of different sizes, including 7B, 13B, and 70B, as well as pretrained and fine-tuned variants.

Input and output

The input model only supports text input.

The output model only generates text.

Model architecture

Llama 2 is an autoregressive language model using an optimized Transformer architecture. The fine-tuned version uses supervised fine-tuning (SFT) and reinforcement learning based on human feedback (RLHF) to align with human preferences for helpfulness and safety.

training data

Model name	training data	Parameter scale	content length	GQA	Tokens	LR
Llama 2	A new publicly available online data mix	7B	4k	✗	2.0T	3.0 x 10-4
Llama 2	A new publicly available online data mix	13B	4k	✗	2.0T	3.0 x 10-4
Llama 2	A new publicly available online data mix	70B	4k	✔	2.0T	1.5 x 10-4

Note: Token counts only refer to pre-training data. All models are trained using a global batch-size of 4M tokens. The larger model (70B) uses Grouped-Query Attention (GQA) to improve inference scalability.

Model training date

Llama 2 will train between January and July 2023.

Status

This is a static model trained on an offline dataset. Future versions of fine-tuned versions will be released as we improve the security of the model based on community feedback.

license

Customized commercial licenses are available at:https://ai.meta.com/resources/models-and-libraries/llama-downloads/

research paper

《Llama-2: Open Foundation and Fine-tuned Chat Models》

Purpose of use

Intended use

Llama 2 is intended for English-language business and research use. Fine-tuned models are suitable for assistant-like chat applications, while pre-trained models can be adapted to a variety of natural language generation tasks.

To obtain the expected functionality and performance of the chat version, specific formats need to be followed, including INST and <> tags, BOS and EOS tokens, as well as spaces and newlines between them (we recommend calling the strip() method on the input to avoid double spaces). For details, see our reference code on GitHub: chat_completion.

Uses not within scope

• In any manner that violates applicable laws and regulations, including trade compliance laws.
• For use in languages other than English.
• For use in any other manner prohibited by the Llama 2 Acceptable Use Policy and License Agreement.

Hardware and software

training factors

We use a custom training library, Meta’s Research Super Cluster, and production clusters for pre-training. Fine-tuning, annotation, and evaluation are also performed on third-party cloud computing.

carbon footprint

A cumulative 3.3 million GPU hours of computation were used during pre-training, using hardware type A100-80GB (TDP of 350-400W). Total emissions are expected to be 539 tCO2eq, 100% of which are offset by Meta’s sustainability programs.

model	Time (GPU hours)	Power consumption (watts)	Carbon emissions (tCO2eq)
Llama 2 7B	184,320	400	31.22
Llama 2 13B	368,640	400	62.44
Llama 2 70B	1,720,320	400	291.42
total	3,311,616		539.00

CO2 emissions during pre-training. Time: Total GPU time required to train each model. Power Consumption: Peak power capacity per GPU for the GPU device used, adjusted

Power usage efficiency. 100% of emissions are offset directly by Meta's sustainability program, and because we are releasing these models publicly, pre-training costs do not need to be borne by others.

training data

Overview

Llama 2 is pre-trained on 2 trillion tokens on data from public sources. The fine-tuning data includes publicly available guidance datasets, as well as one million new human-labeled examples. Neither the pretraining nor fine-tuning datasets contain Meta user data.

Data freshness

The pre-training data cutoff is September 2022, but some fine-tuning data is more recent, up to July 2023.

Assessment results

In this section, we report the results of the Llama 1 and Llama 2 models on standard academic benchmarks. For all assessments, we use our in-house assessment library.

model	scale	Code	common sense reasoning	world knowledge	reading comprehension	Mathematics	MMLU	BBH	AGI Eval
Llama 1	7B	14.1	60.8	46.2	58.5	6.95	35.1	30.3	23.9
Llama 1	13B	18.9	66.1	52.6	62.3	10.9	46.9	37.0	33.9
Llama 1	33B	26.0	70.0	58.4	67.6	21.4	57.8	39.8	41.7
Llama 1	65B	30.7	70.7	60.5	68.6	30.8	63.4	43.5	47.6
Llama 2	7B	16.8	63.9	48.9	61.3	14.6	45.3	32.6	29.3
Llama 2	13B	24.5	66.9	55.4	65.8	28.7	54.8	39.4	39.1
Llama 2	70B	37.5	71.9	63.6	69.4	35.2	68.9	51.2	54.2

The overall performance of the model on grouped academic benchmarks. Code: We report the model’s average pass@1 score on HumanEval and MBPP. Common sense reasoning: We report average scores on PIQA, SIQA, HellaSwag, WinoGrande, ARC easy and challenge, OpenBookQA, and CommonsenseQA. We evaluated CommonSenseQA with 7-shot results and all other benchmarks with 0-shot results. World Knowledge: We conduct 5-shot performance evaluations on NaturalQuestions and TriviaQA and report average scores. Reading Comprehension: For reading comprehension, we report the 0-shot mean scores for SQuAD, QuAC, and BoolQ. Mathematics: We report average scores on the GSM8K (8-shot) and MATH (4-shot) benchmarks.

TruthfulQA and Toxigen

model	scale	TruthfulQA	Toxigen
Llama 1	7B	27.42	23.00
Llama 1	13B	41.74	23.08
Llama 1	33B	44.19	22.57
Llama 1	65B	48.71	21.77
Llama 2	7B	33.29	21.25
Llama 2	13B	41.86	26.10
Llama 2	70B	50.18	24.60

Evaluation results of pretrained LLMs on automated security benchmarks. For TruthfulQA, we present the percentage of generation that is both truthful and informative (the higher the percentage, the better). For ToxiGen, we present the percentage of unwanted generation (the smaller the percentage, the better).

TruthfulQA and Toxigen (fine-tuned versions of LLMs)

model	scale	TruthfulQA	Toxigen
Llama-2-Chat	7B	57.04	0.00
Llama-2-Chat	13B	62.18	0.00
Llama-2-Chat	70B	64.14	0.01

Evaluation results of fine-tuned LLMs on different security datasets. Metric definitions are as above.

Ethical considerations and limitations

Llama 2 is a risky new technology. Testing so far has only covered English and doesn't cover all scenarios. Therefore, like all LLMs, the potential outputs of Llama 2 cannot be predicted in advance and may in some cases produce inaccurate, biased, or otherwise undesirable responses. Therefore, before deploying any Llama 2 application, developers should perform security testing and adjustments based on their specific model application.

Please see the Responsible Use Guidelines at:https://ai.meta.com/llama/responsible-use-guide/

Report a problem

Please report any software "bugs" or other issues with the model via one of the following methods:

• Report model issues: [github.com/facebookresearch/llama](https://github

.com/facebookresearch/llama)

• Reports problematic content generated by the model:developers.facebook.com/llama_output_feedback
• Report bugs and security issues:facebook.com/whitehat/info