update: my employee is passing off ChatGPT lists as his own ideas

by Alison Green on December 23, 2024

It’s “where are you now?” month at Ask a Manager, and all December I’m running updates from people who had their letters here answered in the past.

There will be more posts than usual this week, so keep checking back throughout the day.

Remember the letter-writer whose employee was passing off ChatGPT lists as his own ideas? Here’s the update.

Last year, I wrote in about a junior engineer who was presenting ChatGPT list results as his own without understanding the results. Not surprisingly, the engineer tried doing this again. And again.

I took your advice to heart and the next time it happened, I scheduled a meeting with Dan and explained how he could use his ChatGPT results to start his investigation into the answer but I expected recommendations to be his work. He was not too responsive because this would be more work, which reinforces your point about being a personnel problem and not a technology problem. Then, my (older) boss came to me saying how great this employee was and how he had some many great ideas for this other project. I asked him if this was just a ChatGPT list and I was asked to investigate. Not surprisingly, after Dan reviewed his ChatGPT list, these ideas were completely out of scope for the project, and he wasted a good 2 weeks exploring them. I no longer had the time or energy to fight Dan’s laziness (he is not my direct report) and his manager didn’t see an issue.

This was just the tip of the iceberg of problems at this company and I had been looking for a new and more challenging engineering research environment. After nine months of interviewing (thank you for your advice), I landed a job at Big Tech Research Company that blocks all forms of AI due to security concerns. After 6 months, I find most of the people here are overly motivated with big brains. I spend my time telling them to take breaks or think of the big picture, which is a nice change. It’s a better cultural fit.

Thanks for taking my letter and all the comments. It was very helpful to put the problem into perspective. I see that you’ve had some related letters about ChatGPT, so it’s going to be a tough technology to manage as it becomes integrated into most software programs. For the record, in my personal life, I often have to write reference letters and promotional text and ChatGPT is fantastic! Super fan. I did not use it for this letter! :)

You may also like:
{ 138 comments }

{ 138 comments… read them below or add one }

  1. Not on board*

    In our office, we’ve been exploring using ChatGPT to improve on our wording for promotional materials and whatnot. It’s a great tool – not perfect, the results often need a human eye to keep it from getting nonsensical. But the ideas have to come from the person using the ChatGPT, it’s not a shortcut for producing good work.

    Reply ↓
    1. Antilles*

      ChatGPT can help produce ideas too, the problem as OP correctly notes is that it’s only a starting point to produce ideas; you still need to vet them..
      This letter/update are superficially about ChatGPT, but the reality is that Dan would have been equally a problem five years ago in 2019 or heck, twenty five years ago in 1999. It’s just that rather than ChatGPT, those letters would be about Dan relying on a Google search or some random person’s (under construction!) Angelfire website.

      Reply ↓
    2. TheBunny*

      This. I tell people to think of it like Wikipedia. Great for an initial place to start but you need to do your due diligence before your believe it.

      Reply ↓
  2. Stuart Foote*

    I know that ChatGPT has uses, and clearly AI will be around in the future and maybe improve society somewhat, but right now all the AI stuff seems epically useless. If you google something, you now need to wait until a useless AI summary that is usually either wrong or doesn’t answer the question (and in any case isn’t any better than the preview of the top result we used to get). It’s possible to ask ChatGPT questions, but you can’t rely on the answers since it will often hallucinate facts. I know it is good for programming, and has uses for reference letters and stuff like that, but overall it seems like a huge waste of energy for middling results.

    Reply ↓
    1. Audrey Puffins*

      The AI summaries are the worst. I recently asked Google about the skillset needed to climb a particular mountain; the AI summary said “this mountain is easy to climb”, but when I clicked into the website it took the info from, the full sentence was “do not be misled into thinking this mountain is easy to climb”. A very facile illustration of how you should only ever use it as a starting point and do your own work from that point, but an illustration nonetheless

      Reply ↓
      1. Oh my*

        My favourite AI summary mishap was a yarn crafts related one, specifically about the difference between worsted weight and fingering weight yarn.

        The first half of the sentence was, in fact, about yarn weights.

        After the comma, it was suddenly talking about “fingering”, the NSFW verb.

        Reply ↓
      2. TooTiredToThink*

        Seriously – I got some bloodwork back and when I googled one value that my doc flagged – Google AI said it was a normal value while also quoting the section that shows that it was in fact a low value. *smh* I’ve been seeing a lot of this type of nonsense and I hate it.

        Reply ↓
      3. Crashing into Middle Age*

        I’ve started putting -ai after all my Google searches because I don’t even want to have to scroll past that garbage.

        Reply ↓
      4. Spacewoman Spiff*

        Haha. I love that example. I was recently looking for a new winter coat that would be good to wear on especially cold days and in the midst of that, Google gave me an AI summary explaining that when it’s 17-19 degrees Fahrenheit, a light cardigan is appropriate outerwear. Just kind of astounding that AI can’t tell the difference between Fahrenheit and Celsius, given how much money and energy has gone into developing it….

        Reply ↓
    2. The Gollux, Not a Mere Device*

      If AI improves society, it won’t be ChatGPT or other large language models.

      What looks promising is some of the medical diagnostic software, used with care to avoid things like the software learning that an image including a ruler always shows disease, rather than learning the actual signs of illness in those textbook and journal illustrations of a sick patient.

      Reply ↓
      1. Daughter of Ada and Grace*

        Analytical AI, rather than Generative AI, in other words. Something that can review large quantities of data quickly and accurately (without getting bored or missing things) and identify patterns that humans can then use. Not as showy, but the potential benefits are huge.

        Reply ↓
        1. wordswords*

          Absolutely. Analytical AI has a lot of great uses and a lot of promising future ones!

          Generative AI, on the other hand… well, I’ll be polite and go with “wildly, wildly overhyped at best.”

          Reply ↓
          1. Richard Hershberger*

            Also, I suspect, not long for this world. The use cases don’t even come close to profitability. The entire premise is there will be a huge breakthrough. This is unlikely, and becoming obviously so to the point that even tech journalists are beginning to notice. The money guys will come around eventually and pull the plug.

            Reply ↓
            1. Stuart Foote*

              After seeing the wonderful Silicon Valley bets that were the metaverse and crypto take off so well, it’s certainly hard to imagine this latest tech fad could ever be overhyped and not work that well.

              Reply ↓
        2. Falling Diphthong*

          Analytical AI for large data can be really useful, and before ChatGPT was just called “machine learning algorithms.”

          Generative AI can be useful as an interim step in writing for some people. But it shouldn’t be the final version, and trying to force it in everywhere as the final version is just head scratching. (I wanted that list of Jewish months because I am doing a crossword. The month “Nisan” is not actually clarified by including that photo of a “Nissan.”)

          Reply ↓
        3. amoeba*

          There’s also other kinds of generative AI than LLMs! I’m in chemistry/biotech and for instance, AlphaFold is seriously amazing at generating protein 3D structures. There’s also all kinds of research into generating new materials/molecules/drugs, and while that’s nowhere near where some hyped companies want to make us believe it is, it’s certainly interesting and will hopefully be really useful in the future.

          Reply ↓
          1. Calamity Janine*

            honestly i would expect those to actually not be made using agreed worst practices of AI – for one thing, a black box model that can’t report on its own rules is a bad thing for the sciences. that’s a case where if the computer is able to throw pure processing power to figure out a rule, everyone would quite like to know about it!

            (also in general you could say biology’s problem is that it’s all black boxes who are too complicated to self-report all the rules they follow… the mice so far have not explained themselves no matter how we ask LOL)

            Reply ↓
      2. Red Reader the Adulting Fairy*

        Medical coder here — our AI-assisted coding system ROUTINELY does things like suggests that we code burns for the patient who saw Dr Burns or recommends every single thing listed in the “call back if you experience these side effects” list.

        Reply ↓
    3. Falling Diphthong*

      I sincerely wish this was a feature I could turn off.

      I’ve had to train myself out of glancing over the top response, since it’s now an AI summary composed by a program with no understanding of the topic whatsoever, rather than the opening paragraph of the top site for people asking this question.

      Reply ↓
      1. pmd*

        It’s a nuisance to keep typing it, but with Google you can add -ai to the search terms and it excludes the useless LLM summaries.

        Reply ↓
      2. Alf*

        I have a Firefox extension installed which hides the ai results on google – I’m sure there’s similar extensions for most browsers

        Reply ↓
  3. NotTheSameAaron*

    ChatGPT is good for reformatting text and creating recipes, but needs scrutinizing if you’re going to use it outside that.

    Reply ↓
    1. Calamity Janine*

      don’t trust it for creating recipes, either. it’s going to just copy someone else’s – but about thirty someone else’s at a time so you get mixed methods, surprise ingredients, and nonsensical instructions.

      at that point you’re better served by just going to all recipes or Alton Brown’s website and typing in the name of a dish. the absolute best case scenario, after all, is that chat gpt will be copying from a known good recipe verbatim, but without any credit given to the people who worked out how to make that recipe good – so you can’t actually go browse other recipes to your taste, for example.

      chat gpt is a nonsense engine. it does not fact check anything. it does not care about facts or if it’s giving you a good recipe. that’s not how it’s programmed. it is, instead, a source of quips like if your mozzarella cheese keeps sliding off your pizza, mix Elmer’s glue into it. if you want to trust that with your foodstuffs, i can’t stop you i guess, but keep the poison control line handy and try to only endanger yourself and not others.

      Reply ↓
        1. Calamity Janine*

          “blarney blatherer” is also a good one, though i admit i use it mainly for the alliteration. it does get the point across though – blarney is talking that sounds good until you think about it, after all… or until you find yourself adding glue to your pizza cheese wondering where you went wrong, lol

          Reply ↓
  4. Apex Mountain*

    I use ChatGPT for work multiple times per week. I’m all for caution with AI but an outright ban on all AI seems too limiting

    Reply ↓
    1. Ginger Cat Lady*

      The security issues are real. Hope your boss knows you’re using it, especially if you’re feeding proprietary company info into it.

      Reply ↓
      1. Norm Peterson*

        I work for a major university and we have a license for Microsoft’s ai product (copilot) that keeps everything out of their learning model. So it’s certainly possible to have access to an AI that can have proprietary information fed in.

        Reply ↓
          1. Rosyglasses*

            ChatGPT has a business license which is internal use only and does not learn from proprietary information. We could assume the positive; that that is what is being used.

            Reply ↓
              1. Calamity Janine*

                heck, we could even say it’s true it doesn’t learn from ADDITIONAL proprietary information…

                …but chat gpt can’t exist unless it’s already trained on a corpus that violates copyright, so it has proprietary information baked in already!

                Reply ↓
      2. Apex Mountain*

        There are so many use cases for ChatGPT in a work setting that don’t have anything to do with proprietary information though. That was why I said an outright ban doesn’t make sense to me, but evaluating each case on its own is the better way

        Reply ↓
        1. Calamity Janine*

          given how chat gpt works by their own admission, it is a machine which relies on plagiarism to work. a chat gpt that doesn’t violate copyright law is a chat gpt that doesn’t work. they outright admit this.

          additionally, as we’ve seen in AI art generation, it produces works that are not copyrightable themselves. at best you are turning large parts of your business to a source where you can’t enforce your claim on that intellectual data, because it’s not yours – it’s something that belongs to the rickety tower of authors that chat gpt has copied from. this is a major weakness that your competitors can exploit, all the while making fun of you for not actually giving a damn to put in real work (and why trust something like that with your business).

          that’s the proprietary information issue to be concerned with – chat gpt can’t produce any, and is a major intellectual copyright danger.

          Reply ↓
          1. Bitte Meddler*

            @Magic – Happy to see there’s someone who refuses to take an airplane flight, use electronics, walk on cement sidewalks or enter buildings made with cement, own any plastic goods, or drive a car!

            Reply ↓
            1. Flor*

              This is a false equivalence. Many of these things are difficult, if not impossible, to avoid in modern society (never using a sidewalk OR a car? I mean come ON.). It’s very easy to avoid actively using genAI, however. Literally everyone on the planet managed to do so until two years ago, and many of us still do.

              Reply ↓
    2. JSPA*

      when you primarily work with private data, patient data, patent data or sensitive data, you can’t afford to have blurred lines inside the company.

      For that matter, if you’re in a competitive sector (and/or publicly listed) you probably also shouldn’t want Chat GPT to invisibly link your product types plus your company’s buzz words with your long term economic forecast, or the sort of details that belong on an internal powerpoint. Even the fact that you’re putting together a big ad campaign / making dozens of text and image mockups is a form of harvestable data.

      Reply ↓
      1. Apex Mountain*

        This is all true but there are so many use cases for ChatGPT or similar that the ones I am using don’t do any of that

        Reply ↓
        1. Calamity Janine*

          if you’re using chat gpt, you’re using a machine that the company has outright stated runs on illegal activities. there’s no way to make chat gpt work without massively breaking copyright law. and, as a bonus, you can’t actually copyright what chat gpt produces – so that is additional weakness to be exploited by your competitors.

          chat gpt, for this reason, inherently has ethics and security issues.

          the fact that you are relying on this tool while not understanding how it works doesn’t make the tool okay. it just means that you’re projecting even more of an image as Someone Who Doesn’t Care About Doing Their Job Correctly. you haven’t avoided the issues, merely perpetuated them.

          that’s before we even get to “if you seek to convince your company that you can be replaced by algorithmic monkeys on typewriters, you may just succeed, even when you don’t want to”…

          Reply ↓
          1. Apex Mountain*

            You’re trying to insult me by saying I have no idea what I’m doing and that I don’t care about doing my job well.

            If you want to have a conversation, great let’s have one. But please educate yourself on the topic first – none of what you mentioned has anything to do with what I’m using it for

            Reply ↓
            1. Calamity Janine*

              well, if you’re using a system trained on an entirely limited legal corpus, then you’re using a novelty – not something that can be lumped in with appropriate use cases of chat gpt. unfortunately you do indeed need to know how chat gpt works in order to discuss the downsides. i’m sorry if you’re upset by me being the one to tell you how chat gpt works – but it doesn’t make that any less true.

              Reply ↓
              1. Apex Mountain*

                Again, the downsides you mention of chatGPT have literally nothing to do with the way I use it. I’m not upset, you’re just not fully informed

                Reply ↓
                1. Calamity Janine*

                  how do you use chat gpt in a way that completely avoids using chat gpt, a thing that the company knows and promotes as an algorithm that can’t work unless it violates copyright? the downsides are inherent limitations to how it works, much the same way that a steam locomotive has the inherent limitation of “there will be hot water involved at some point, so hot it becomes steam even”. it’s just a basic fact of how it works. chat gpt is a language learning model that cannot exist unless it’s breaking copyright law. there’s not a magical way to use it that will make it draw on only a legal corpus – chat gpt has argued in court repeatedly that they can’t do this and it doesn’t work.

    3. Czhorat*

      It REALLY depends on your job and the expectations.

      I wonder if ChatGPT-style generative AI tools are going to last or if this will be a short-lived fad due to intellectual property and security concerns.

      Reply ↓
        1. Falling Diphthong*

          ^This. And that’s more likely to be fatal.

          Violating copyright is one thing, but violating copyright and then failing to turn a profit is another.

          Reply ↓
        2. Sola Lingua Bona Lingua Mortua Est*

          Why be stingy? It’ll be a short-lived fad for all those reasons and more.

          It’s the Crystal Pepsi of computer programs.

          Reply ↓
          1. Calamity Janine*

            it’s not the first bubble that AI will have faced, but it’s certainly going to be an aggravating one to clean up after. i think my father, who has worked in this area since before i was born, is looking forward to simply retiring instead of having yet another decade or so of telling people “no, no, that’s not exactly what i do when i say i work in artificial intelligence…”

            Reply ↓
          2. Dara*

            Interestingly, Crystal Pepsi was a success – until Coke took notice and decided It Needed to Go. Crystal Pepsi was a caffeine-free, clear, lighter tasting option. So Coke took Tab – a product it didn’t care that much about – made it clear, sugar-free, and marketed it as a more “medicinal” type of product. Consumers, so used to Coke and Pepsi making their own versions of the same thing, assumed that what Coke said was true about their product must be the same as Pepsi’s, and interest tanked because consumers conflated the two in their minds.

            Reply ↓
      2. Apex Mountain*

        “It REALLY depends on your job and the expectations.”

        Yes, that’s really my point – there are a ton of things I wouldn’t use ChatGPT for, but there are some I do. There’s no blanket statement

        Reply ↓
        1. magic*

          There is a pretty important blanket statement: every time someone uses it, they’re depleting (and normalizing the depletion of) our most precious resources. How much stuff falls into the category of not important enough to justify banning usage, but somehow still worth the cost?

          Reply ↓
    4. FricketyFrack*

      I don’t mean to be rude, but are you aware of how many resources AI uses? I mean, is what you’re doing so difficult to do manually that it’s worth that? I used it once to write something that I ended up having to mostly rewrite, I just can’t imagine that mediocre-at-best results are worth the millions of gallons of fresh water and enormous amounts of electricity that it takes to run AI.

      Reply ↓
      1. magic*

        YEP. This is the most important factor, thank you for bringing it up. The environmental impact is huge and unconscionable.

        Reply ↓
        1. magic*

          A single generated short email is estimated to use a standard bottle of water. I recommend looking into it more in order to reevaluate how appropriate usage is.

          Reply ↓
          1. Calamity Janine*

            to me what truly makes it not worth it is the expense versus the product you get. if you were making something truly remarkable with no other ways to get it, that would be one thing… but it’s an inefficient use of resources for, well, a product that is mediocre. and even if you don’t care about the environment, it’s a product that chat gpt has outright stated cannot be made while following copyright law. so you get a mediocre product that you have to edit and fact-check anyway (so how much time is actually gained? why not just write a useful form letter that you can modify for additional uses and know exactly what it says, putting the time investment in up front instead of having to clean up after the computer being silly?), all for this increased expense.

            not that i would style myself as a business maven who knows all the answers, but i feel that spending more resources for a mediocre product is probably something that should be avoided. just in general, lol.

            Reply ↓
    5. EngineeringFun*

      OP here: I work on government research with very high security. We are working on an internal AI system, but for legal reasons we

      Reply ↓
  5. Hlao-roo*

    Thanks for this update! I’m glad you addressed the issue head-on with Dan. It’s too bad that neither Dan nor his manager wanted to change Dan’s performance. I’m glad you saw the big-picture situation for what it was and found a job that’s a better fit!

    Reply ↓
  6. mango chiffon*

    I find it worrisome that people use ChatGPT like they’re googling information and don’t take any time to see if that “info” is actually correct.

    Reply ↓
    1. Czhorat*

      The scary thing is that the nature of the tool means that the answers will almost always LOOK correct; looking accurate without being accurate is the worst possible result.

      Reply ↓
      1. Daughter of Ada and Grace*

        Alternately, something can be accurate without being correct, sometimes horrifyingly so. (The case I’m thinking of was someone looking for a comparison between yarn weights: sock and fingering. It correctly and accurately described sock weight yarn, then gave a description of a sex act with the same name as a yarn weight… accurate, but not correct for the context.)

        Reply ↓
      2. Bird names*

        That’s the crux, isn’t it. You still need discernment, maybe more now than ever and Dan and his boss reaaally couldn’t or didn’t want to understand that. I worry about the future, not specifically because of these tools, but the degree to which they have accelerated a trend of “how shitty and cheap can we make this before it all catches up with us” in too many places where good judgement is and remains crucial.

        Reply ↓
      3. Falling Diphthong*

        An aspect I find interesting is that it knows to give a one line plausible explanation for the answer. But it isn’t generating that one line based on any knowledge, just on some word associations. (For example, that diamond is an excellent electrical conductor because of its carbon bonds. Which is not true, but is a collection of words you might find in articles about electrical conductivity, material properties, etc.)

        Reply ↓
  7. Pastor Petty Labelle*

    ChatGPT literally cannot create anything new. It is only putting together phrases created elsewhere – often copyrighted material which ChatGPT does NOT pay to use. It also uses an incredible amount of resources for every single search.

    Instead of accepting it because it is being crammed into everything so the creators can justify the research expense, we should be pushing back against it. I wish google had an opt out of the AI summary BEFORE the search happens.

    Reply ↓
    1. Calamity Janine*

      chat gpt has repeatedly argued that not only do they not draw only on sources that are in the realm of fair use, they have stated that they CANNOT make it work while using sources available under fair use. plagiarism is an inherent part of the machine. they don’t pretend otherwise. they just whine they wanna do it anyway and that’s why they should be exempt from laws they consider inconvenient.

      people who support it should probably think about what “we know it’s illegal but we can’t make it work if it’s actually legal, so let us do it anyway!” usually leads to in terms of results.

      Reply ↓
    2. Strive to Excel*

      AMEN for the google opt out function! I’d never previously considered switching primary search engines but this succeeded in making me do it.

      Reply ↓
      1. DJ Abbott*

        What did you switch to?
        I’m guilty of accepting google AI results, but this discussion is making me more careful.

        Reply ↓
      2. Sharpie*

        Same. I hadn’t even noticed until the other day that the top result for a search in Google is now an AI generated piece of garbage. Which is not what I want, ethics aside. If I’m looking for something, I want an actual answer not some invented rubbish.

        AI is everywhere. And it stinks.

        Reply ↓
    3. Corrvin (they/them)*

      There is a Chrome extension that blocks the AI summary called “Bye, bye, Google AI”. Hope that helps! (I know there is a similar one for Edge and there’s probably one for Firefox as well.)

      Reply ↓
    4. office hobbit*

      Look up udm 14! It removes the AI summary and other bloat. You can append it to your searches yourself or run your searches through udm14 dot com.

      Reply ↓
  8. Calamity Janine*

    a caveat about the end there:

    much like AI generated art, people are becoming wise to chat gpt in personal spheres, too. you may think you’re fooling everyone. it probably is that someone thinks you’re sounding off and not like yourself at best (all the easier to be replaced immediately by any other service that can create boilerplate generic copy), and at worst, so cheap that you’d rather make the computer do it because you just don’t care.

    it’s not good writing – the biggest tell for me is that chat gpt is “purple in all the wrong places”. it tries to be dramatic as a human can but without knowing where to put that. that turns out muddled and generic at best.

    please just write an actual sentence yourself instead of reams of computer generated pablum. it will be noticed and appreciated. before you use it in your personal life, think hard if you want to actually cosign the notion of “don’t give a shit enough to do it myself so i made the computer do it lmao”. not to mention the ethical issues with AI, because these models can only exist due to plagiarism writ large – and chat gpt has repeatedly admitted this as they argue that they can’t only use texts that are fair use.

    please don’t glorify or rely on the bullshit creation engine. it creates bullshit in your business. it doesn’t actually get magically all okiedokie in your personal life.

    Reply ↓
    1. Businesstime*

      Yup. Chat GPT writing reads like what dumb people think smart people write like. Circular logic, fancy-esque phrases that amount to nearly nothing.

      Reply ↓
  9. not like a regular teacher*

    Generative AI uses massive amounts of electricity and clean water. For environmental reasons alone, we should all be using it as seldom as possible!

    Reply ↓
    1. Kenna*

      Truly – it’s baffling to me that people are using it instead of search engines, to write emails for them, to “talk” to, etc… what an absolute waste of resources, especially for something that has been proven over and over to spit out sheer nonsense fairly often! I really hope people start thinking twice before using it for frivolous things – the wastage adds up so quickly!

      Reply ↓
    2. Valancy Stirling*

      Yeah, and also, it’s impossible to train gen AI without commiting theft of intellectual property. I have many artist friends who had to stop sharing their work online because it’s the only way to keep it from training AI models.

      Reply ↓
      1. Calamity Janine*

        and sadly, text is getting scraped all the time – without any sort of consent, in defiance of the law – so it’s not like we can do any poison pill measures as have been developed for pictures… and those are still something that some algorithms are trained to defeat.

        we’re talking about the same thing that can produce a simulacrum of the Getty images logo when you ask it to make a stock photo, and the company is trying to play coy about where they trained things and how illegal that was. chat gpt has also repeatedly said they can’t make their bot work if they actually follow the law.

        it’s killing the environment, stealing, and creating a product that customers are increasingly reading as “too cheap to care about”. people don’t look successful when they use chat gpt and generative ai illustrations – they get laughed at.

        this ain’t a thing to be lauded, for all the reasons you point out and more.

        Reply ↓
    3. Insulindian Phasmid*

      Yea – I actually like the idea of using gen AI image programs to get my vague half-images into a form that I can communicate what I want to other people – it’s much easier to get across to an artist what I want with a rough draft that isn’t a stick figure. But the sheer amount of power it takes is baffling. That’s the biggest reason I don’t use it.

      Reply ↓
  10. CityMouse*

    I’m a lawyer and we see AI “hallucinations” where it will just straight make up case law. Lawyers have actually been sanctioned for submitting briefs that cited completely nonexistent cases Some of the large legal research companies are touting their AI tools that supposedly don’t do this, but I’m skeptical.

    Reply ↓
    1. Calamity Janine*

      for folks who may be curious about this, Legal Eagle on youtube has some good videos on one such case.

      they’re also great if you need to torture someone through sheer secondhand embarrassment. because the judge gave them so many times to go “aw shucks yeah sorry i used chat gpt and it invented those cases” and yet…

      Reply ↓
    2. Clisby*

      Yes, I’m a retired computer programmer and still this surprised me. When I first heard of a lawyer being sanctioned for this, I sort of assumed the AI had gathered citations that had nothing to do with the case at hand – it didn’t occur to me that the AI had manufactured citations out of thin air.

      I was even more surprised that the law firm didn’t check it. I’d expect them to check a citation that was found in Wikipedia, or mentioned in a news article, or just googling around; why would AI be different? (Also, how hard is it hand a list of citations to a legal assistant and ask them to verify them on LexisNexis?)

      Reply ↓
      1. CityMouse*

        It’s also not remotely hard to verify cites. It’s really standard to do that when reading a brief filed by opposing counsel, you never take their word on what a case says.

        Reply ↓
        1. Clisby*

          That’s exactly what I’d expect – and I think in at least one of these cases, that’s how it was caught. Someone was checking the citations and couldn’t find some of them.

          Reply ↓
          1. Calamity Janine*

            the lawyer was even given a week by the court to find his citations and didn’t actually admit he was using chat gpt to the bitter end! the Legal Eagle video about that case is just a neverending fount of secondhand embarrassment. it was bad enough for me as a layperson, and i’m sure that actual lawyers will be watching it through their fingers while cringing in their seats and whispering “oh no… oh no!!” as if they were watching a special edition of the Saw horror movies for lawyers.

            Reply ↓
    3. fhqwhgads*

      Not picking on you personally, but I really dislike how the common phrasing has been to describe LLMs making stuff up as “hallucinations”. I get that it jives with the notion of calling it “AI” and thus how that term originated, but it’s helping perpetuate its own hype.
      An LLM at its core is a “sound like a sentence a person might say” machine. It’s not at all concerned with facts. Any time it gets any info correct is because it happened to have more correct info than incorrect info, so what it generates is also correct. When it makes stuff up out of whole cloth it’s not hallucinating. It’s constructing sentences that sound like they’re a thing.

      Reply ↓
      1. CityMouse*

        To be clear, that’s the precise phrasing they used in the state bar guidance and their CLE. So the battle on that phrasing has been lost.

        Reply ↓
  11. Valancy Stirling*

    I had to fail a ton of students this year because they ask AI for book summaries instead of reading the books in question (which are assigned at the beginning of the school year), and pass off blatantly false information as if it’s fact. I had to stop assigning homework altogether – if there’s anything they need to write, they now write in class.

    It’s more time-consuming and fewer things get done, but it’s the only way to prove that they did the assignment themselves.

    Reply ↓
    2. Bird names*

      Thank you very much for taking the time to do this. While they might not appreciate it now and some might never understand why it matters, I’m glad they are required to truly engage with the material at hand. Commented about a related issue higher up, but the potential for skill loss does worry me. We’ve all seen plenty of letters about employees who blatantly copy other’s work of course, but I wonder how much this has helped increase a tolerance for shoddy subpar work, as long as it can be done quickly.

      Reply ↓
    3. Strive to Excel*

      I’ve had to cut way back on how much I let my music at work wander through new things because there’s a lot of new stuff uploaded to YouTube that I really doubt are human made. If they are, the new music creators are suffering pretty hard too.

      Reply ↓
  13. mango chiffon*

    There was an issue around hiring recently at my office where we were hiring for a higher level research position, so the candidates were often in PhD programs. Part of the interview process involved a short written project and apparently a few of the candidates that had moved to that round were returning very similarly written products. Someone decided to put it through ChatGPT because they were curious if it was AI, and it turned out that the interview candidates had put it through ChatGPT. Eventually they had to completely restart the job search and it took almost a year to hire for this position.

    Reply ↓
  14. No Longer Gig-less Data Analyst*

    Our organization has created some kind of closed ChatGPT where they have loaded in information about our company and division, and apparently whatever we feed into it doesn’t leave the org, so we can put whatever we want into it. I figure it’s only a matter of time before it gets us into trouble with confidential information, but that is a decision made much higher up the food chain than my position.

    I am admittedly someone who side-eyes AI a lot (I write books on the side), but I used it to help with my year end review and it was crazy helpful. It usually takes me forever to think about how to word my accomplishments in the buzz-wordy way that leadership tends to reward, because it doesn’t come naturally and seems like a lot of BS to me. So I typed in a very dry but thorough description of my achievements in 2024, and ChatGPT not only produced full paragraphs that sounded like the CEO had personally written them, but because it had been pre-loaded with the high-level goals and mission of the company, it listed what big picture targets my own smaller projects were related to.

    They actually invented a corporate-speak bullshit machine – what a time to be alive!

    Reply ↓
    1. Generic Name*

      This is really hilarious to me. My last company had a really onerous review process, and this would have helped so much.

      Reply ↓
  15. Pyanfar*

    Does anyone have a recommendation for an article/book/class/etc. that will give an overview of what generative AI can and can’t do, good use cases, and a brief how-to use it properly?

    Reply ↓
  16. Fotze*

    It amazes me how many people think Chat GPT is awesome because they know so little about what it outputs. Sure, it can be helpful…but my experience is that it writes with more complex wording than a natural voice, and almost exclusively writes in 4 sentence paragraphs. It can also become quite repetitive for longer pieces. I’ve even tried it for recipes, and it’s literally the same 3 ideas over and over.

    However, I think it can be a good springboard, especially if it’s given a human eye before sending. I overhead some people in the sauna yesterday saying it was great for shortening what they write to be more concise. I’ve always had the opposite problem. I’m very concise, and could never write to fill an arbitrary requirement.

    Reply ↓
    1. Falling Diphthong*

      In Richard Osman’s new book, the criminal mastermind uses it to remove any distinctive voice from his written communications. So there’s that.

      Reply ↓
      1. Calamity Janine*

        now that’s a use case i can see!

        …but also should be a big warning to anyone looking to use it in business. you want the most generic pablum possible that’s one step above lorum ipsum? you want to make sure your company has no trace of brand identity? meet your brand new hero, chat gpt…!

        Reply ↓
    2. Apex Mountain*

      “However, I think it can be a good springboard, especially if it’s given a human eye before sending.”

      That’s really all some us AI supporters are saying

      Reply ↓
  17. Strive to Excel*

    I feel like generative AI is a good replacement for things that were or could almost have been form letters before. Have to write a similar-but-not-exactly the same email to 30 peoples? Struggling to phrase “yes that actually will be a problem” into polite business language? Sure. Replacing people entirely? It’ll happen in some jobs and sector, but unlikely to do so in all of them.

    Reply ↓
    1. magic*

      Except one of those emails consumes a whole bottle of water! Like, as a person who overthinks everything and has ADHD I get the appeal. But all these minor use cases are still a big problem.

      Reply ↓
      1. Strive to Excel*

        Yes, if and only if the environmental issues are worked out! I am not a fan of generative AI in any way shape and form.

        Reply ↓
  18. LizB*

    I found an actual use case for generative AI the other day where it was genuinely helpful – I wanted to put some slightly complicated conditional formatting on an excel sheet, but could not for the life of me get the formula to work the way I wanted it to. With AI I was able to type in what I wanted in plain language and it spat out a formula that worked on the first try. I still felt icky about using it for environmental reasons but it was interesting to finally see a place in my work that it could, in fact, be useful for something instead of just a nuisance.

    Reply ↓
    1. metadata minion*

      I realize sometimes you need an answer right away, but is there some reason you can’t just ask a human for help with things like this? There are all kinds of message boards and other resources out there, and there’s probably someone who’s had a similar use case and has a formula that works.

      Reply ↓
      1. LizB*

        Fair enough, I tend to just read message boards (which I had done for about 20 minutes before I resorted to AI) and never actually post on them. I assume that if anyone else has had my use case, there will be discussion about it somewhere I can find it. I’ll rethink.

        Reply ↓
      2. Pocket Mouse*

        I do similar to LizB’s example for fixing my code in certain coding languages. If I can’t figure out my issue from my other work, tutorials, or forum searches, I’ll paste a chunk of code into ChatGPT and ask it for the fix. When I search forums, the error I’m getting may be asked about but due to entirely different problems, or may be asked about for another coding language altogether. If the question is “I’m trying to get X result, how do I do it?” it’s really tough to find the search terms that will turn up other people’s questions—and then it’s possible they’re trying to do the same thing but have an issue irrelevant to mine. And sometimes my problem is about an option or function I don’t even know about but need to include for my specific purpose, and since I don’t know about it, I can’t search for it. I avoid using AI when I can, mainly due to environmental concerns, but if it would result in hours or days of lost productivity with no guaranteed solution, sometimes that’s just not practical.

        I do appreciate the reminder in comments here to turn off AI in my search engines!

        Reply ↓
  19. Calamity Janine*

    i admit that this is mainly from me knowing inside baseball – my dad has been working in the field of AI since before i was born, and if i gave any more information, i would quickly and thoroughly doxx myself as his name appears in all sorts of referenced papers – but he loathes generative AI for good reasons. a sampling of which are…

    1. these black box models of AI have no way to self-report. they make up their own rules and then can’t tell you what they are so that you can say “wait no that’s illegal”. this is rather important for many reasons. (as a fun example, see how many “AI art filters for your pictures” will straight-up make people white, or make black people part of the scenery instead…)
    2. generative art and text models such as chat gpt are merely picking words or pixels that are most likely to follow each other. they don’t care about answers being correct, just popular.
    3. as such the text models often “hallucinate”. they invent things from whole cloth. it is incredibly easy to make them do this.
    4. it’s also very easy to get them saying the completely wrong answer. they don’t care about the logic of the situation – they literally can’t, given how those large language models work. they don’t care about the correct answer. just the popular one. and so no, chat gpt can’t tell you how many Rs are in the word strawberry, but it will confidently give you wrong answers all day long.
    5. chat gpt and AI art generation rely on mass violations of copyright law. chat gpt has outright stated repeatedly that they can’t make it work unless they do crime. that’s why they are currently getting massively sued. and yes, they’re trying the “but it doesn’t work unless you let us violate the law!” defense in court, too.
    6. because of how these models work, they always require more data – massive amounts of it. and they’ve blown past the data they’re allowed to use a long, long time ago. they’re now, sometimes accidentally, training on stuff also generated by AI. this “inbreeding” effect is already bad and will get worse as it continues. after all, as in point #1, there’s no way to look at the rules the algorithm is following directly – you just feed it more data in the right direction. it is more hungry than it can possibly be sustained on and will only increase in appetite.
    7. basically every single instance of what was defined as AI worst practices in the early 80s is being used right now, speaking of the first point. no way to self report and directly correct. no way to get a legal corpus to train on. no way to be run legally period. you get the idea. it’s about as good computer science as if you got a doctor who thinks Louis Pasteur was a hack, every good doctor goes around covered in blood and gore to show you how he does surgeries all the ti so don’t worry about it, and there’s no such thing as germ theory because you just need your humors balanced. would YOU go to that doctor? would you go to that doctor for brain surgery or other very important things? the answer is no, i would hope. so don’t trust in the equivalent in computer science.

    Reply ↓
      1. Calamity Janine*

        truly, i am an example of how a boutique and specialized neural network* can be fed a curated source of data** to produce a superior result***

        *creation of brainmeats via having a child

        **my dad’s rants about people being idiots in his field

        ***me pretty much being able to repeat said rants verbatim

        (don’t worry, i went and got my biology degree and i have gotten him back a thousand fold. mostly because a whole lot of that has been discovered since his high school course in the 70s, inevitably reporting on science at least a decade old at the youngest. i have terrified him with diagrams of cell signalling, the complete redo of the tree of life that happened based on DNA evidence in the 90s, and the full Krebs cycle. that one gets us both though. i learned it in abbreviated form for AP Bio and now i simply run away from it screaming loudly at all the chemistry, because chemistry is not a squishy enough science for me to hang with.)

        Reply ↓
  20. Calamity Janine*

    ooft, sorry, this is a nesting fail meant for Pyanfar above asking about books or articles on how to use these resources ethically! the tldr is: lmao don’t homies just don’t

    Reply ↓
    1. Calamity Janine*

      the irony of this being a double nesting fail is not lost on me.

      brb gotta go give my phone a mental breakdown so when it starts singing “daisy, daisy, give me your answer, do”, i can actually open the pod bay doors and possibly at some point there’s a giant monolith.

      Reply ↓
  21. Happy*

    How utterly depressing and yet unsurprising that these male bosses see an incompetent young man as a genius, despite LW’s direct evidence that his ideas are just wastes of time.

    I’m sorry, LW. I hope you find a better place to work, and where your insights are valued.

    Reply ↓
  22. Claire*

    Anyone who uses ChatGPT should first read about the dazzlingly huge environmental costs of doing so, in terms of water and energy use. Then decide if it’s ethical to use.

    Reply ↓
  23. Elizabeth West*

    As a creator, I despise generative AI and avoid it like the plague. It’s put journalists out of work and decimated artists’ livelihoods, not to mention the environmental costs. You can’t convince me it’s in any way ethical. Any gains are waaaaay offset by the costs in my opinion. Right now, AI is just a shiny new toy from a bunch of techbros with a raging case of dragon sickness and little awareness of anything outside their own sphere.

    OP, I hope your new job is one where guys like Dan don’t get to fail up.

    Reply ↓
  24. RidiculousPenguin*

    There’s someone at my company who was recently promoted into the C-suite. Problem is that almost all of their work is produced by ChatGPT. Until now only people in certain departments have known this, but soon the whole company will (they are in a marketing role). I feel terrible for them, but honestly the work they produce is routinely horrible, and if no one has figured this out yet that’s… problematic.

    Reply ↓

Leave a Comment

Before you comment: Please be kind, stay on-topic, and follow the site's commenting rules.
You can report an ad, tech, or typo issue here.

Subscribe to all comments on this post by RSS