Our honest feedback of ChatGPT
by CyberTest
Posted on April 7, 2023
There is lot of talk about ChatGPT lately so we decided to test it out ourselves and provide our honest feedback and do's and don'ts.
ChatGPT is a natural language processing program that allows you to have human-like conversations with it. It has tons (multiple terabytes) of data sets from
many sources like internet, academic papers, books, articles, forums, blogs, etc. ChatGPT responses are generated based on statistical
patterns in the training data. So the responses are as accurate as the data itself.
We tested the program by asking some simple questions. The ChatGPT response came back with good enough responses. Ofcourse if you ask for something
that it has no data about it would not be able to answer, but the fact it's able to construct human like response is cool.
We then provided some code snippets to check for issues or asked to give example code how to open file and read the contents line by line.
The result was good. Few minor tweaks here and there but overall the code snippets did work. We then asked to convert 500 lines of c++ code to javascript or php
but it failed. There seems to be limit on how much response it can give back. So we broke down the code into snippets and gave it one at a time.
We then tested the converted code but there was errors. We actually had to spend some time to figure out and fix the errors. Then the response was not accurate.
From the looks of it seems that for complex stuff like converting code it may not work well but for small sample code snippets it works well.
We also found out that it was good on generating regex statements.
If you write a question first time and you don't get the accurate response, then try to tweak your question. We noticed that after
the first response it will provide more accurate response if you ask to give you more info. However if you don't get your question answered
after few tries then chances are it's not going to be able to give you the correct response. You will notice that it will keep giving same responses over and over again.
So in conclusion we think ChatGPT can be addition to Google or Bing search, StackOverflow, Wikipedia, etc. It is NOT a replacement.
For example there was questions or code snippets that we found from the Google search quicker than ChatGPT. In other cases like regex generation we found
ChatGPT worked better or at least was quicker to get it working. This makes us wonder if Google will soon add Google Search Bot feature as an alternative to ChatGPT?
Only time will show but for now you can play with it and see what works better for you when using ChatGPT.
An important note that is worth mentioning. ChatGPT reflects the work of the millions of humans who provided the actual data from many sources as mentioned above.
It is the collected and written knowledge of the human race and ChatGPT is just a program written by humans to nicely tokenize, search and output results as if a human is answering.
Many algorithms have been used with creation of ChatGPT and it is the achievement of the human engineering and everyone else who contributed to the years of data creation.
When you chat with ChatGPT in some way it feels like you are talking to millions of people. Without humans this achievement would not have been possible.
Here is our Do's and Don'ts when using ChatGPT:
------
Do's
------
- Can use it for getting example code snippets on how to use a functions or code.
- Very useful for regex generation.
- Can use for spell check or grammar check.
- Addition but NOT a replacement to StckOverflow, Forums, Blogs or search engines.
- Do validate the response with other sources for accuracy.
- Can be used for contract template generation.
- May be useful for content writing.
--------
Don'ts
--------
- The responses may not reflect real time life events. As of testing ChatGPT the data sets were up to 2021 date.
- Privacy concerns. Do not paste your company code or confidential data in ChatGPT. Consult with your legal and privacy team before using ChatGPT.
- Don't blindly trust the response. You need to validate and test it.
- Don't expect ChatGPT to create full functional program or something new. It only provides sample codes and some guidance.