Skip to content

Commit

Permalink
add clarifying note about GELU
Browse files Browse the repository at this point in the history
  • Loading branch information
rasbt committed Jun 29, 2024
1 parent ab23ca5 commit 796f0e2
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion ch04/01_main-chapter-code/ch04.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -667,7 +667,7 @@
"metadata": {},
"source": [
"- As we can see, ReLU is a piecewise linear function that outputs the input directly if it is positive; otherwise, it outputs zero\n",
"- GELU is a smooth, non-linear function that approximates ReLU but with a non-zero gradient for negative values\n",
"- GELU is a smooth, non-linear function that approximates ReLU but with a non-zero gradient for negative values (except at approximately -0.75)\n",
"\n",
"- Next, let's implement the small neural network module, `FeedForward`, that we will be using in the LLM's transformer block later:"
]
Expand Down

0 comments on commit 796f0e2

Please sign in to comment.