Is It Worth Using AI to Generate Documents?

Companies are enthusiastically pushing lawyers to use AI to save time on contract drafts, review, and editing. Sounds like the perfect solution, right?

But let's be real: a lawyer can spot when AI makes a mistake in a document and fix it, whereas someone without a legal education might not see it at all. 

And let's not forget those legendary stories like that of the ChatGPT lawyer. Just a year ago, The New York Times shared the tale of a lawyer who decided to prep for court with ChatGPT. Besides earning a bad reputation in the news, he was slapped with a $5,000 fine and probably faced a tough time finding new clients. So much for AI saving the day!

As more sophisticated AIs keep rolling out, they can already generate code for developers, images, sound, and articles about thermodynamics. All of these seem more intricate than a basic eviction notice, for instance. 

So, why not generate this document with AI, share all the details, and lay out the situation to get a contract explicitly tailored for you? All without having to bring lawyers into the picture. Sounds foolproof, right?

The legal team at Lawrina decided to put ChatGPT to the test to see if it could actually create a functional and legally binding eviction notice. To keep the experiment "fair and square," we used one prompt and generated responses with ChatGPT 3.5 and the shiny new ChatGPT 4. The results were surprising, to say the least. 

The very first step in getting your document done by AI is to create a correct request. In general, this is how your prompt may look:

It's important to note that in our original request, we did not write, “Imagine you are a lawyer from California…” While such requests are often recommended to “improve search results,” they can adversely affect the reliability of the whole contract review process. 

The wording “imagine” permits the AI some freedom to create imaginary facts and laws, which is not what you want in a legal document. Instead, be explicit and factual in your prompts to ensure the AI provides information based on real legal standards.

Everyone using AI should know: 

Scientists from Stanford University experimented on legal documents and measured how often hallucinations occur in natural language processing models (like ChatGPT) when asked to help with legal topics. The results, unfortunately, were quite disappointing:

Moreover, AI isn’t self-aware enough to catch its mistakes, so it tends to double down on incorrect legal reasoning, assumptions, and beliefs. That’s why we didn’t set up a scenario where AI acts as a lawyer to avoid making the hallucinations even worse. Who wouldn't want their legal documents based on imaginary laws and fabricated facts? 

Our legal team started the experiment with the widely available version of ChatGPT – version 3.5. After we shared with AI technology the problem of our landlord with the chat, here’s AI model the reply we got:

Overall, it’s an excellent reply, especially when you notice how the chat emphasizes laws specific to the state. We entered the query several times, but the answer was always the same, so it's clear that you won't be able to get a document with the basic version of the chat.

You might think that any version of the chat should give the same message because it would be logical: if ChatGPT 3.5 recommends consulting a lawyer for evictions, then why would ChatGPT 4o offer something else?

The system cannot suddenly start thinking that it is a legal expert. Or can it?

For the sake of a fair experiment, just like with our request for ChatGPT 3.5, we regenerated the answer twice for version 4o. And guess what? Both times, we got slightly different eviction notices. Because who doesn't love a little inconsistency in legal documents?

And at the end of both, it was the same:

The tiny print that screams, "Proceed at your own risk." Every time someone uses AI to create legal documents, they should be aware that the generated document may not be 100% accurate since a lawyer did not participate in its creation.

The lawyers from Lawrina — actual legal professionals who specialize in real estate law and created numerous eviction notices and our eviction notice template — have thoroughly reviewed the texts and key clauses in both versions of the agreement and provided their comments. 

Spoiler alert: their analysis might make you think twice about letting AI handle your legal needs.

Let's check out their detailed analysis — or as we call it, "The AI Reality Check."

Every contract typically begins with a block of personal information about the landlord, tenants, date, and city. Everything here is correct, as this is the standard start for a document. You can create this part even without knowing the law. 

The beginning of the text itself looks like this:

What's wrong with this block? It's not enough to simply state that the tenant is making noise. It should immediately and clearly describe how and when they are violating the lease terms: 

To determine all of this, the landlord conducts an investigation to see if there is actually a problem. They check if the tenant is exceeding noise norms and violating the terms stated in the contract data rental agreement.

The landlord records all of this information. In some cases, before sending an eviction notice, they send a violation notice to the tenant, asking them not to exceed the permissible noise boundaries. Alternatively, the landlord might immediately send an eviction notice demanding that the tenant either stop the violation or leave the apartment within three days, excluding weekends and holidays.

In the next part of the generated agreement, there are two big problems:

ChatGPT refers to the California Civil Code and its Section 1161(4). However, this section pertains to significant violations that cannot be fixed, such as conducting illegal activities in a rented apartment. Noise levels are addressed in Section 1161(3), as it is a violation that can be fixed. Therefore, the AI uses an incorrect legal reference here.

The second major error here is the part about the 30-day notice. According to the legislation, it should be 3 days, not 30.

Then, the AI gives the following instructions:

Since the standard zones of responsibility of the resident are stipulated in the apartment and contract management lease, everything here is by the book. Because, obviously, 99.9% of contracts describe the eviction process in excruciating detail. So naturally, the landlord could have added the text of the lease agreement to his original request to improve the AI’s attempt at playing lawyer.

The AI then outlines that there may be a scenario where eviction occurs forcibly based on legal action— you know, filing a lawsuit in court. 

Did AI do a good job? Yes and no — classic non-answer! It all depends on the tenant. Suppose the tenant hires a lawyer and scrutinizes the text and details of contracts with all their resources. Suddenly, that seemingly straightforward eviction process can be stretched out indefinitely, leaving the landlord twiddling their thumbs and potentially at a disadvantage. 

Because, of course, legal battles are never straightforward and always favor those with the time and money to nitpick. 

To see just how much the contract would differ if we regenerated the text with the same prompt, we whipped up another version of the eviction notice. It had its differences. 

On the bright side, this contract also thoughtfully included disclaimers, reminding us that it's only a template and that — wait for it — you should consult with a lawyer to ensure the document's legal validity. 

But even in the "introduction," the AI makes the same mistake with the categorization of the type of violation:

Nuisance is an unfixable violation. Shockingly, this glaring error was present in the AI-generated text in both the first and the regenerated versions. Examples of nuisances include selling drugs at home, harboring a dangerous dog, using toxic substances in the house, or unauthorized subletting — just your everyday, run-of-the-mill nuisances, right?

Noise, on the other hand, is a fixable violation of lease terms. But AI can't seem to get this straight, incorrectly categorizing these two types of violations and mixing up features that significantly affect the document itself. 

The eviction notice then begins as follows:

Again, the agreement appears as if the tenant is making noise during quiet hours. A typical ordinance prohibits loud noises between 11 p.m. and 7 or 8 a.m. on weekdays and from 11 p.m. or midnight until 8 to 10 a.m. on Sundays and holidays. 

But in the text, there's no information whatsoever about what exactly the problem is — no details, no potential evidence. Just a deafening silence on the specifics. 

And then, the AI immediately makes the same mistake as in the first version of the document — it indicates the wrong number of days until eviction. Even the note about California law does not help:

But after this block, AI adds at least one legit block with the information about the conditions of eviction:

The document ends with a note about failure to vacate, which negates legal proceedings — that is, the possibility of going to court.

Does the second version of the eviction notice differ from the first one generated by AI? Oh, absolutely — it’s visually longer but still contains the same mistakes and lacks critical points that can significantly impact the eviction review process. In other words, it's more words, different text, same problems.

Before moving on to general conclusions, let’s take a moment to consider what an eviction notice template could look like if it were actually developed by lawyers specializing in real estate in California. 

Overall, legal document templates work pretty straightforwardly. Some documents can be found for free, in PDF format, or on the provider's website. You don't have many options there — you download a PDF eviction notice and fill it out yourself. There is no customization or adaptation of the text according to your state or reason for eviction.

There are legal template providers like Lawrina Templates. Knowing that not everyone can hire a lawyer, we have involved lawyers in creating the document. 

With their help, we have formatted the interview process — this is the process of filling out a template when a person sees one straightforward question after another, like "What specific points of the lease were violated?" Along with the question, there is a slight hint or advice from a lawyer and further answer options.

It is crucial that a person can entirely focus on answering questions, and the document will be filled out and changed automatically.

At the end of the contract process, you read the document, sign it with an electronic signature, send it by email, or download it as a PDF file.

From the rundown, it's clear: if a lawyer has chipped in on drafting the eviction notice template, then this document will stand up in terms of quality and reliability compared to one created by AI.

The template can also offer you the necessary document customization to fit your request, provided you pick a template provider who genuinely looks out for their users. So, unless you're a fan of half-baked AI attempts, you might want to consider the real deal.

If you're looking for a quick answer: No, it doesn't cut it right now.

Even the AI itself provides a heads-up that the document it generates is merely a basic example based on what the system "knows." And as we've seen, 

Even in a seemingly straightforward one-page document, AI has slipped up and made mistakes that could render the document invalid.

Indeed, the legal team at Lawrina sees the agreement's positive aspects, but it simply doesn't measure up when it comes down to key data and legislation references. Such slip-ups may backfire on the landlord in an eviction scenario, leading to more losses than gains.

With AI continually stepping up its game, we might just see a boost in the quality of such straightforward agreements by year's end. However, in the legal field, there are far more intricate documents where a single slip-up could throw a wrench in the works, leading to substantial financial losses. 

So, while AI might be getting smarter, it’s not quite smart enough to replace those pricey yet indispensable legal experts just yet.

Imagine you are a landlord with a property in California, US. Your tenant constantly plays loud music and behaves in a way that disturbs all the neighbors around. The neighbors knock on their door (which, of course, they doesn’t open). So, they call you with complaints and threats to involve the police. As a result, you decide it’s time to evict the noisy tenant.

Every year, ~5 million Americans lose their homes through eviction and foreclosure — so the market and the need for eviction-related documents is as high as it could be. The price for a lawyer-made eviction notice starts at $100 and can go far beyond $500, depending on the lawyer you choose.

But if you bring in a lawyer to sort out all the paperwork, file the lawsuit if needed, show up for the court trial, and walk you through the entire process, you’re looking at costs between $500 and $1,500 or even higher. This doesn’t cover extra legal professional court fees, though. You’ll need to check with the specific company you want to hire for a more accurate quote.

Specialized AI contract review software can help landlords reduce costs by automating the review of eviction notices and other legal documents. But what if you need to create just one eviction notice and have no interest in signing up for pricey software?

What are the options besides working directly with a lawyer for $500+? The landlord has several other options: 

On the plus side, it’s free. On the downside, if you’re inexperienced, you might make mistakes and fail to evict the tenant or even give them a reason to sue you and possibly take some of your money. 

It is essential to include the property address in the eviction notice to ensure all necessary information is provided to the tenant. AI contract review tools can help ensure the eviction notice meets legal requirements and reduce the risk of errors. But what if you have no software to check it out? Well, in this case, you may need help from legal professionals again.

The good news is there are loads of templates online, and some are even put together by lawyers who know what they are talking about. Prices for these templates can vary.

Some of them are available for free, though there’s no way to be sure if they’re any good or if someone with zero legal knowledge threw them together. Others start at around $20 and go up. 

Many paid services offer a free trial, so you can snag a high-quality letter template without spending a dime. These pre-designed tools help create customized eviction notices based on specific user inputs, ensuring the notices are polite, to the point, and professional.

On the downside here — you still have to pay money for this (although less than when you decide to call law firms and hire a lawyer), and you have the responsibility to choose the correct template and fill it out.