Skip to content

Commit a2cd843

Browse files
rasbtrasbt
authored
Ch05 supplementary code (rasbt#81)
1 parent 861a278 commit a2cd843

34 files changed

Lines changed: 796 additions & 18 deletions

File tree

‎.github/workflows/basic-tests.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ jobs:
3333
- name: Test Selected Python Scripts
3434
run: |
3535
pytest ch04/01_main-chapter-code/tests.py
36+
pytest ch05/01_main-chapter-code/tests.py
3637
pytest appendix-A/02_installing-python-libraries/tests.py
3738
3839
- name: Validate Selected Jupyter Notebooks

‎.github/workflows/pep8-linter.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,4 +18,4 @@ jobs:
1818
- name: Install flake8
1919
run: pip install flake8
2020
- name: Run flake8 with exceptions
21-
run: flake8 . --max-line-length=120 --ignore=W504,E402,E731,C406,E741,E722,E226
21+
run: flake8 . --max-line-length=140 --ignore=W504,E402,E731,C406,E741,E722,E226

‎README.md‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,9 @@ This repository contains the code for coding, pretraining, and finetuning a GPT-
77
<br>
88
<br>
99

10-
<a href="http://mng.bz/orYv"><img src="images/cover.jpg" width="250px"></a>
10+
<a href="http://mng.bz/orYv"><img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/cover.jpg" width="250px"></a>
1111

12-
In [*Build a Large Language Model (From Scratch)*](http://mng.bz/orYv), you'll discover how LLMs work from the inside out. In this book, I'll guide you step by step through creating your own LLM, explaining each stage with clear text, diagrams, and examples.
12+
In [*Build a Large Language Model (From Scratch)*](http://mng.bz/orYv), you'll discover how LLMs work from the inside out. In this book, I'll guide you step by step through creating your own LLM, explaining each stage with clear text, diagrams, and examples.
1313

1414
The method described in this book for training and developing your own small-but-functional model for educational purposes mirrors the approach used in creating large-scale foundational models such as those behind ChatGPT.
1515

@@ -37,7 +37,7 @@ Alternatively, you can view this and other files on GitHub at [https://github.co
3737
| Ch 2: Working with Text Data | - [ch02.ipynb](ch02/01_main-chapter-code/ch02.ipynb)<br/>- [dataloader.ipynb](ch02/01_main-chapter-code/dataloader.ipynb) (summary)<br/>- [exercise-solutions.ipynb](ch02/01_main-chapter-code/exercise-solutions.ipynb) | [./ch02](./ch02) |
3838
| Ch 3: Coding Attention Mechanisms | - [ch03.ipynb](ch03/01_main-chapter-code/ch03.ipynb)<br/>- [multihead-attention.ipynb](ch03/01_main-chapter-code/multihead-attention.ipynb) (summary) <br/>- [exercise-solutions.ipynb](ch03/01_main-chapter-code/exercise-solutions.ipynb)| [./ch03](./ch03) |
3939
| Ch 4: Implementing a GPT Model from Scratch | - [ch04.ipynb](ch04/01_main-chapter-code/ch04.ipynb)<br/>- [gpt.py](ch04/01_main-chapter-code/gpt.py) (summary)<br/>- [exercise-solutions.ipynb](ch04/01_main-chapter-code/exercise-solutions.ipynb) | [./ch04](./ch04) |
40-
| Ch 5: Pretraining on Unlabeled Data | - [ch05.ipynb](ch05/01_main-chapter-code/ch05.ipynb) | [./ch05](./ch05) |
40+
| Ch 5: Pretraining on Unlabeled Data | - [ch05.ipynb](ch05/01_main-chapter-code/ch05.ipynb)<br/>- [train.py](ch05/01_main-chapter-code/train.py) (summary) <br/>- [generate.py](ch05/01_main-chapter-code/generate.py) (summary) | [./ch05](./ch05) |
4141
| Ch 6: Finetuning for Text Classification | Q2 2024 | ... |
4242
| Ch 7: Finetuning with Human Feedback | Q2 2024 | ... |
4343
| Ch 8: Using Large Language Models in Practice | Q2/3 2024 | ... |
@@ -58,4 +58,3 @@ Alternatively, you can view this and other files on GitHub at [https://github.co
5858
Shown below is a mental model summarizing the contents covered in this book.
5959

6060
<img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/mental-model.jpg" width="600px">
61-

‎appendix-A/02_installing-python-libraries/python_environment_check.ipynb‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,16 @@
11
{
22
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "c31e08b0-f551-4d67-b95e-41f49de3b392",
6+
"metadata": {},
7+
"source": [
8+
"<font size=\"1\">\n",
9+
"Supplementary code for \"Build a Large Language Model From Scratch\": <a href=\"https://www.manning.com/books/build-a-large-language-model-from-scratch\">https://www.manning.com/books/build-a-large-language-model-from-scratch</a> by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
10+
"Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
11+
"</font>"
12+
]
13+
},
314
{
415
"cell_type": "code",
516
"execution_count": 1,
@@ -45,7 +56,7 @@
4556
"name": "python",
4657
"nbconvert_exporter": "python",
4758
"pygments_lexer": "ipython3",
48-
"version": "3.10.12"
59+
"version": "3.10.6"
4960
}
5061
},
5162
"nbformat": 4,

‎appendix-A/02_installing-python-libraries/python_environment_check.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,7 @@
1-
# Sebastian Raschka, 2024
1+
# Copyright (c) Sebastian Raschka under Apache License 2.0 (see LICENSE.txt).
2+
# Source for "Build a Large Language Model From Scratch"
3+
# - https://www.manning.com/books/build-a-large-language-model-from-scratch
4+
# Code: https://github.com/rasbt/LLMs-from-scratch
25

36
import importlib
47
from os.path import dirname, join, realpath

‎appendix-A/02_installing-python-libraries/tests.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,10 @@
1+
# Copyright (c) Sebastian Raschka under Apache License 2.0 (see LICENSE.txt).
2+
# Source for "Build a Large Language Model From Scratch"
3+
# - https://www.manning.com/books/build-a-large-language-model-from-scratch
4+
# Code: https://github.com/rasbt/LLMs-from-scratch
5+
6+
# File for internal use (unit tests)
7+
18
from python_environment_check import main
29

310

‎appendix-A/03_main-chapter-code/DDP-script.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,8 @@
1+
# Copyright (c) Sebastian Raschka under Apache License 2.0 (see LICENSE.txt).
2+
# Source for "Build a Large Language Model From Scratch"
3+
# - https://www.manning.com/books/build-a-large-language-model-from-scratch
4+
# Code: https://github.com/rasbt/LLMs-from-scratch
5+
16
# Appendix A: Introduction to PyTorch (Part 3)
27

38
import torch

‎appendix-A/03_main-chapter-code/code-part1.ipynb‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,16 @@
11
{
22
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "f896245e-57c4-48fd-854f-9e43f22e10c9",
6+
"metadata": {},
7+
"source": [
8+
"<font size=\"1\">\n",
9+
"Supplementary code for \"Build a Large Language Model From Scratch\": <a href=\"https://www.manning.com/books/build-a-large-language-model-from-scratch\">https://www.manning.com/books/build-a-large-language-model-from-scratch</a> by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
10+
"Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
11+
"</font>"
12+
]
13+
},
314
{
415
"cell_type": "markdown",
516
"id": "ca7fc8a0-280c-4979-b0c7-fc3a99b3b785",

‎appendix-A/03_main-chapter-code/code-part2.ipynb‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,15 @@
11
{
22
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"<font size=\"1\">\n",
8+
"Supplementary code for \"Build a Large Language Model From Scratch\": <a href=\"https://www.manning.com/books/build-a-large-language-model-from-scratch\">https://www.manning.com/books/build-a-large-language-model-from-scratch</a> by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
9+
"Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
10+
"</font>"
11+
]
12+
},
313
{
414
"cell_type": "markdown",
515
"metadata": {

‎appendix-A/03_main-chapter-code/exercise-solutions.ipynb‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,15 @@
11
{
22
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"<font size=\"1\">\n",
8+
"Supplementary code for \"Build a Large Language Model From Scratch\": <a href=\"https://www.manning.com/books/build-a-large-language-model-from-scratch\">https://www.manning.com/books/build-a-large-language-model-from-scratch</a> by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
9+
"Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
10+
"</font>"
11+
]
12+
},
313
{
414
"cell_type": "markdown",
515
"metadata": {},

0 commit comments

Comments
 (0)