{"componentChunkName":"component---src-templates-blog-js","path":"/blog/gpt2-colab-tutorial","result":{"data":{"contentfulBlogPost":{"title":"GPT-2 Colab Tutorial","publishedDate":"January 4th, 2021","body":{"json":{"nodeType":"document","data":{},"content":[{"nodeType":"paragraph","content":[{"nodeType":"text","value":"In today's tutorial, I'll walk you through how to get started with GPT-2. GPT-2 is a language model that is built by Open AI,  if you are here then chances are you haveve already heard about GPT-2 or maybe his successor GPT-3. A More in-depth explanation is ","marks":[],"data":{}},{"nodeType":"hyperlink","content":[{"nodeType":"text","value":"here","marks":[],"data":{}}],"data":{"uri":" if you are here"}},{"nodeType":"text","value":" in official docs.","marks":[],"data":{}}],"data":{}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":" For the sake of simplicity, we'll use google colab today, colab is like a more advanced jupyter notebook is ready in the cloud and waiting for you. You can even use a GPU powered application on the free tier of colab, it's truely hard to find the limits of the free tier with regular every day training tasks like that one.","marks":[],"data":{}}],"data":{}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"I'll mostly show what to do in an order by pictures, so if you don't want to write what I showed yourself(makes sense) you can go ahead and check the ","marks":[],"data":{}},{"nodeType":"hyperlink","content":[{"nodeType":"text","value":"repo","marks":[],"data":{}}],"data":{"uri":"https://github.com/eren23/gpt2-colab-tutorial"}},{"nodeType":"text","value":" for the project. This tutorial is actually based on official github ","marks":[],"data":{}},{"nodeType":"hyperlink","content":[{"nodeType":"text","value":"page","marks":[],"data":{}}],"data":{"uri":"https://github.com/openai/gpt-2"}},{"nodeType":"text","value":", yet since there are some errors with the documentation I'll just suggest you to use what is here in this tutorial instead.","marks":[],"data":{}}],"data":{}},{"nodeType":"hr","content":[],"data":{}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"First we'll start by creating a new colab project.","marks":[],"data":{}}],"data":{}},{"nodeType":"embedded-asset-block","content":[],"data":{"target":{"sys":{"space":{"sys":{"type":"Link","linkType":"Space","id":"qv0hcw26u3ts","contentful_id":"qv0hcw26u3ts"}},"id":"c5OwaSccBS4hzaaW8aqW3Yc","type":"Asset","createdAt":"2021-01-04T14:52:01.937Z","updatedAt":"2021-01-04T14:52:01.937Z","environment":{"sys":{"id":"master","type":"Link","linkType":"Environment","contentful_id":"master"}},"revision":1,"contentful_id":"5OwaSccBS4hzaaW8aqW3Yc"},"fields":{"title":{"en-US":"pynbnewnb"},"file":{"en-US":{"url":"//images.ctfassets.net/qv0hcw26u3ts/5OwaSccBS4hzaaW8aqW3Yc/511b941b93fc1146226ac783ba40ad9c/Screen_Shot_2021-01-04_at_14.18.32.png","details":{"size":473679,"image":{"width":2880,"height":1510}},"fileName":"Screen Shot 2021-01-04 at 14.18.32.png","contentType":"image/png"}}}}}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"Then we can pull the GPT-2 project by using git command.","marks":[],"data":{}}],"data":{}},{"nodeType":"embedded-asset-block","content":[],"data":{"target":{"sys":{"space":{"sys":{"type":"Link","linkType":"Space","id":"qv0hcw26u3ts","contentful_id":"qv0hcw26u3ts"}},"id":"foFepxfdifytJPD7W6Egy","type":"Asset","createdAt":"2021-01-04T14:56:54.201Z","updatedAt":"2021-01-04T14:56:54.201Z","environment":{"sys":{"id":"master","type":"Link","linkType":"Environment","contentful_id":"master"}},"revision":1,"contentful_id":"foFepxfdifytJPD7W6Egy"},"fields":{"title":{"en-US":"git-clone-gpt2"},"file":{"en-US":{"url":"//images.ctfassets.net/qv0hcw26u3ts/foFepxfdifytJPD7W6Egy/efddb8ab396a554242b51139fe1a1a4d/Screen_Shot_2021-01-04_at_14.19.08.png","details":{"size":92759,"image":{"width":2752,"height":300}},"fileName":"Screen Shot 2021-01-04 at 14.19.08.png","contentType":"image/png"}}}}}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"We can then install Tensorflow in our environment by using the comman below.","marks":[],"data":{}}],"data":{}},{"nodeType":"embedded-asset-block","content":[],"data":{"target":{"sys":{"space":{"sys":{"type":"Link","linkType":"Space","id":"qv0hcw26u3ts","contentful_id":"qv0hcw26u3ts"}},"id":"c1JvlKkNCvmbhPYfnXLbRWe","type":"Asset","createdAt":"2021-01-04T14:54:58.548Z","updatedAt":"2021-01-04T14:54:58.548Z","environment":{"sys":{"id":"master","type":"Link","linkType":"Environment","contentful_id":"master"}},"revision":1,"contentful_id":"1JvlKkNCvmbhPYfnXLbRWe"},"fields":{"title":{"en-US":"tensorflow-colab"},"file":{"en-US":{"url":"//images.ctfassets.net/qv0hcw26u3ts/1JvlKkNCvmbhPYfnXLbRWe/fa70b4e66e1b8b395b226f93a5aa7ab0/Screen_Shot_2021-01-04_at_14.22.36.png","details":{"size":530372,"image":{"width":2754,"height":956}},"fileName":"Screen Shot 2021-01-04 at 14.22.36.png","contentType":"image/png"}}}}}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"After that we need to locate our gpt-2 folder and then install the requirements, we can do that by using those 2 commands in order.","marks":[],"data":{}}],"data":{}},{"nodeType":"embedded-asset-block","content":[],"data":{"target":{"sys":{"space":{"sys":{"type":"Link","linkType":"Space","id":"qv0hcw26u3ts","contentful_id":"qv0hcw26u3ts"}},"id":"c4WOa3wYDX1taa8ZPtjLnlt","type":"Asset","createdAt":"2021-01-04T14:53:48.499Z","updatedAt":"2021-01-04T14:53:48.499Z","environment":{"sys":{"id":"master","type":"Link","linkType":"Environment","contentful_id":"master"}},"revision":1,"contentful_id":"4WOa3wYDX1taa8ZPtjLnlt"},"fields":{"title":{"en-US":"req-install-colab"},"file":{"en-US":{"url":"//images.ctfassets.net/qv0hcw26u3ts/4WOa3wYDX1taa8ZPtjLnlt/2f84ccb53d8922172275a43929fb2409/Screen_Shot_2021-01-04_at_14.23.40.png","details":{"size":95632,"image":{"width":2762,"height":530}},"fileName":"Screen Shot 2021-01-04 at 14.23.40.png","contentType":"image/png"}}}}}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"The last step before being able to run the model is to get our selection of modal, we'll choose the smallest one for the simplicity purposes.","marks":[],"data":{}}],"data":{}},{"nodeType":"embedded-asset-block","content":[],"data":{"target":{"sys":{"space":{"sys":{"type":"Link","linkType":"Space","id":"qv0hcw26u3ts","contentful_id":"qv0hcw26u3ts"}},"id":"c52RZtw67s2zz9VTPcJiEbY","type":"Asset","createdAt":"2021-01-04T14:58:31.132Z","updatedAt":"2021-01-04T14:58:31.132Z","environment":{"sys":{"id":"master","type":"Link","linkType":"Environment","contentful_id":"master"}},"revision":1,"contentful_id":"52RZtw67s2zz9VTPcJiEbY"},"fields":{"title":{"en-US":"download-model-colab-gpt2"},"file":{"en-US":{"url":"//images.ctfassets.net/qv0hcw26u3ts/52RZtw67s2zz9VTPcJiEbY/f55aebd6da72aa5140177fb2275c8a07/Screen_Shot_2021-01-04_at_14.28.00.png","details":{"size":129015,"image":{"width":2758,"height":388}},"fileName":"Screen Shot 2021-01-04 at 14.28.00.png","contentType":"image/png"}}}}}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"Since we need to test the model against our own sentences, we will need an prompt and there is no prompt provided by the colab for free tier. We'll however at this point use another module to overcome this problem.","marks":[],"data":{}}],"data":{}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"We'll use the kora module to create our console in a distinct URL.","marks":[],"data":{}}],"data":{}},{"nodeType":"embedded-asset-block","content":[],"data":{"target":{"sys":{"space":{"sys":{"type":"Link","linkType":"Space","id":"qv0hcw26u3ts","contentful_id":"qv0hcw26u3ts"}},"id":"c6St31EHpB89ZDY6eqYsnI3","type":"Asset","createdAt":"2021-01-04T15:14:22.950Z","updatedAt":"2021-01-04T15:14:22.950Z","environment":{"sys":{"id":"master","type":"Link","linkType":"Environment","contentful_id":"master"}},"revision":1,"contentful_id":"6St31EHpB89ZDY6eqYsnI3"},"fields":{"title":{"en-US":"colab-console-kora"},"file":{"en-US":{"url":"//images.ctfassets.net/qv0hcw26u3ts/6St31EHpB89ZDY6eqYsnI3/ecef1f0a5b181c8bcbedd2ef051dd6e9/Screen_Shot_2021-01-04_at_16.13.46.png","details":{"size":549556,"image":{"width":2758,"height":1076}},"fileName":"Screen Shot 2021-01-04 at 16.13.46.png","contentType":"image/png"}}}}}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"After this last block of our code has runned  succesfully  we can go ahead and play on our console by using the link from the output. Before running the command below we'll basically need to run ","marks":[],"data":{}},{"nodeType":"text","value":"cd gpt-2 ","marks":[{"type":"code"}],"data":{}},{"nodeType":"text","value":"command first so we can run our modal finally.","marks":[],"data":{}}],"data":{}},{"nodeType":"embedded-asset-block","content":[],"data":{"target":{"sys":{"space":{"sys":{"type":"Link","linkType":"Space","id":"qv0hcw26u3ts","contentful_id":"qv0hcw26u3ts"}},"id":"c6wkk4lznZBfZOMuWBHBBmF","type":"Asset","createdAt":"2021-01-04T15:00:12.156Z","updatedAt":"2021-01-04T15:00:12.156Z","environment":{"sys":{"id":"master","type":"Link","linkType":"Environment","contentful_id":"master"}},"revision":1,"contentful_id":"6wkk4lznZBfZOMuWBHBBmF"},"fields":{"title":{"en-US":"colab-gpt2-console-command"},"file":{"en-US":{"url":"//images.ctfassets.net/qv0hcw26u3ts/6wkk4lznZBfZOMuWBHBBmF/bc0d18b45a6223005810ffd210394b58/Screen_Shot_2021-01-04_at_14.28.46.png","details":{"size":20714,"image":{"width":1216,"height":56}},"fileName":"Screen Shot 2021-01-04 at 14.28.46.png","contentType":"image/png"}}}}}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"Here is my question to GPT-2 below, not the easiest task even for humans right?","marks":[],"data":{}}],"data":{}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"","marks":[],"data":{}}],"data":{}},{"nodeType":"embedded-asset-block","content":[],"data":{"target":{"sys":{"space":{"sys":{"type":"Link","linkType":"Space","id":"qv0hcw26u3ts","contentful_id":"qv0hcw26u3ts"}},"id":"c3ZLdHEqzXnZTMXcWL2DvU7","type":"Asset","createdAt":"2021-01-04T14:57:30.667Z","updatedAt":"2021-01-04T14:57:30.667Z","environment":{"sys":{"id":"master","type":"Link","linkType":"Environment","contentful_id":"master"}},"revision":1,"contentful_id":"3ZLdHEqzXnZTMXcWL2DvU7"},"fields":{"title":{"en-US":"use-model-colab-gpt2"},"file":{"en-US":{"url":"//images.ctfassets.net/qv0hcw26u3ts/3ZLdHEqzXnZTMXcWL2DvU7/5705c32562a6e65c887bfb0f6b2d9745/Screen_Shot_2021-01-04_at_14.29.29.png","details":{"size":30761,"image":{"width":1384,"height":710}},"fileName":"Screen Shot 2021-01-04 at 14.29.29.png","contentType":"image/png"}}}}}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"Results from the GPT-2 side however are quite allright to me, given that this model is just a small portion of true GPT-2 and GPT-3 even greater than the original GPT-2 I can't even decide if the results are promising or scarry :).","marks":[],"data":{}}],"data":{}},{"nodeType":"embedded-asset-block","content":[],"data":{"target":{"sys":{"space":{"sys":{"type":"Link","linkType":"Space","id":"qv0hcw26u3ts","contentful_id":"qv0hcw26u3ts"}},"id":"c5NYA2pk9AKU9bbCdHl9o3a","type":"Asset","createdAt":"2021-01-04T14:57:46.910Z","updatedAt":"2021-01-04T14:57:46.910Z","environment":{"sys":{"id":"master","type":"Link","linkType":"Environment","contentful_id":"master"}},"revision":1,"contentful_id":"5NYA2pk9AKU9bbCdHl9o3a"},"fields":{"title":{"en-US":"model-result-gpt2-colab"},"file":{"en-US":{"url":"//images.ctfassets.net/qv0hcw26u3ts/5NYA2pk9AKU9bbCdHl9o3a/8f98d3b758698c2dbb9e69d7c87eee47/Screen_Shot_2021-01-04_at_14.30.52.png","details":{"size":302400,"image":{"width":1402,"height":838}},"fileName":"Screen Shot 2021-01-04 at 14.30.52.png","contentType":"image/png"}}}}}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"","marks":[],"data":{}}],"data":{}},{"nodeType":"hr","content":[],"data":{}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"Later on the blog I might publish some more work about running GPT-2 on a container or/and how to fine tune GPT-2 model for our own needs. Until then, take care of yourselves :)","marks":[],"data":{}}],"data":{}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"","marks":[],"data":{}}],"data":{}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"","marks":[],"data":{}}],"data":{}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"","marks":[],"data":{}}],"data":{}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"","marks":[],"data":{}}],"data":{}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"","marks":[],"data":{}}],"data":{}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"","marks":[],"data":{}}],"data":{}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"","marks":[],"data":{}}],"data":{}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"","marks":[],"data":{}}],"data":{}},{"nodeType":"paragraph","content":[{"nodeType":"text","value":"","marks":[],"data":{}}],"data":{}}]}}}},"pageContext":{"slug":"gpt2-colab-tutorial"}}}