This to me is why I don't use ChatGPT. Every time I use it I get answers that are questionable in validity. I'd say that 3/4 of my questions have returned incorrect answers at least partially. Maybe 10% are wildly incorrect with the remainder being partially incorrect.
So if it is wrong with MOST of the questions that I am able to validate myself, then how can I trust it on the questions that I am unable to validate myself.
The whole reason I started even doing this passive research is because I had an employee late last year who used to be a decent employee (he was never an all-star, but he got the job done to an average/satisfactory level), and started to all of a sudden perform incredibly poorly.
He was submitting code and solving problems that were just really bad. He was always just an average producer, and was always teetering on the edge during performance reviews, doing just well enough that we kept him around. But he quickly started to make mistake after mistake. Several code reviews I found really strange artifacts and comments in his code that were blatant mistakes. I confronted him about them over the course of several performance reviews and said he blamed stackoverflow "copy pasta". Eventually I actually fired him, since it was getting to a point that almost everything he submitted or produced was problematic in some way and he was burning more of my time than he was saving. So I ended up firing him.
While firing him, in front of HR he finally broke down and admitted that he has been using ChatGPT for everything and he begged us to let him stay and he would stop using it altogether. I of course didn't care at that point and we let him go. But I started to realize the increase of mistakes were all due to ChatGPT leading him astray.
That whole experiment really taught me that ChatGPT is not ready for primetime. If you blindly trusted ChatGPT you will find yourself in the wrong place most of the time. The problem is that unless you already know the answer to the question you are asking, it is very difficult to tell where chatgpt's answers might be correct and were they might be incorrect (because it is usually a mixture of both). This makes it entirely useless for asking questions that you are not comfortable validating.
Presumably you're a good manager but the way you told it makes you sound like a bad manager that didn't dig into what's going on with a poor performing employee and let them flail around until you had to fire them. There's more to that story and it's not really about ChatGPT at all.
There are other stories out there, like https://hyperbo.la/w/chatgpt-4000/ which shows it can useful and a force multiplier when used well, but it's like giving a faster car to a bad driver. It'll just result in them crashing faster. If you've got a programmer that doesn't want to program, ChatGPT can't help them be a better programmer since they don't actually want to be one!
Why’d he be so reluctant to confess to using ChatGPT? And why would he continue using it despite constant feedback that his performance was getting so much worse?
> So if it is wrong with MOST of the questions that I am able to validate myself, then how can I trust it on the questions that I am unable to validate myself.
If you’re using it to generate code, you can validate it yourself - run the code.
ChatGPT doesn't need to provide perfect AWS solutions for people without AWS experience; the onus is on the human to do the work necessary on their side to be sure of the solution it produces.
It should be thought of as a time saving tool for experts (and people willing to put the work in), not a magic button for lazy people.
The role doesn’t need any IAM RDS permissions. It needs network access. But then all of the access to the database is controlled by your standard database permissions - not IAM.
I guess there are corner cases like using IAM permissions to connect to your database which can be done. But is not typical
It still isn't tho, this will work for testing, maybe, but each lambda will be it's own connection and that will exhaust db resources real fast, you're supposed to have a pooling proxy between lambdas and RDS.
I asked ChatGPT the same question again and then followed up with:
“Is there a more efficient way to connect to the database”
It then went off the rails. It gave me generic answers about using connection pooling that your language framework provides.
Of course that doesn’t help with each Lambda being invoked separately.
I then asked “But each Lambda runs based on its own vm”
It then correctly said that while each Lambda invocation would have its own pool and couldn’t be shared, it could be shared with multiple connections during one invocation.
Which is technically correct. But not the answer I was looking for.
In all fairness, it’s the same type of answers I would expect from a junior to mid developer with limited experience with AWS. I would hire a person who could answer that well.
I try not to cheat and hint at specific issues (since that relies on prior knowledge)
I'd be surprised if even after the last prompt it wouldn't notice that. Saying "Did we miss anything" leaves it open it to re-evaluate both the implementation and the original considerations
