We drove over to look anyway. It was a stone house with big mature trees. A light snowfall made the property look like a Currier & Ives print. Our kids ran around the yard, jumping in the creek out front. We had to drive home to get our 7-year-old son into dry clothes. But in just a few minutes, we’d fallen for the place.

From the visit, I got an idea for how we might afford the property. It had a small cottage, separate from the main house, which might provide rental income that we could then use to help cover the mortgage. We still needed a large down payment, however. But I also had an idea for where to get that money. I’d borrow from myself.

Read: U.S. gains 315,000 jobs in August. Labor market still strong but shows sign of cooling.

First, I rolled an IRA into my new 401(k) plan at work. Once it was transferred, I borrowed the maximum allowed from the plan—$50,000. I’d have five years to repay the loan through automatic payroll deductions. The interest rate was the prime rate plus 1%, as I recall.

Plan loans are the most popular 401(k) feature—after the employer match, that is. At any given time, one worker in eight has a 401(k) loan outstanding. Because you’re borrowing from your own savings, you don’t need a bank’s approval. It’s also easy to apply. Often, you just fill out an online form or talk with a phone representative.

There was still one hitch, however. Borrowing from the 401(k) went against the advice of my new employer, Vanguard Group. It wasn’t a strict prohibition. Vanguard does allow loans from its 401(k) plan. But the company’s stated position was that money saved for retirement should be used only for retirement.

Read: I took Social Security at 62 and now regret it. Is there a way to increase my Social Security benefit?

This argument has real merit. It’s hard enough for many Americans to amass enough for retirement. We tend to start saving later in our careers. Many workers also don’t set aside enough each month. Why take money out of an account that may already be too small?

I knew that I was a good saver, contributing as much as I could to the plan. At the rate I was going, I didn’t think there would be a shortfall at retirement. I didn’t want to miss out on other goals. Buying a nice house in a good school district would make my work feel more rewarding.

Vanguard had other, more specific reasons to counsel workers against borrowing. The money would be “out of the market” until it was paid back. This meant I would miss out on gains if there was a run-up in stock prices. But by the same token, I might avoid a loss if share prices happened to drop while I had a loan outstanding. This was a bit of a tossup because it depended on timing.

Vanguard’s strongest argument was that some borrowers can’t repay their loans, usually because they lose their job. This can set off a financial avalanche. Any remaining balance comes due in full, usually within 60 to 90 days, depending on plan rules. If the borrower can’t make the balloon payment, the unpaid balance is subtracted from the borrower’s retirement savings. This is reported to the IRS as a taxable distribution, subject to income taxes and usually a 10% early withdrawal penalty.

Under this worst-case scenario, you could lose your job, default on the loan, lose a chunk of your savings and then owe the IRS money. Approximately $6 billion in 401(k) savings are lost this way each year, according to a 2015 estimate by researchers from Peking University, University of Pennsylvania’s Wharton School and Vanguard. Their estimate was higher than that found in previous studies.

I could imagine a black swan event like this occurring, just not to me. Like most people, I had faith in “recency”—that the current conditions I enjoyed would flow seamlessly into the future. I felt confident that my job was safe and my health would remain good.

That doesn’t always happen, of course, but everything worked out fine for us. As I look back, I realize that I’d taken a big gamble that luckily turned out OK. Yet I’d probably do the same thing all over again in the same circumstances. Like the idea of borrowing from your 401(k)? Here are four suggestions to make such loans less risky:

Borrow infrequently. I took just one loan from my 401(k) during my career. If you borrow, do it for something vitally important, and not for a luxury purchase or a vacation.

One at a time. Some 401(k) plans allow workers to have more than one loan outstanding at any given time. Those who take out two loans or more have a higher rate of defaulting. They’re often borrowing from Peter to pay Paul.

Not an emergency fund. Workers who borrow from the 401(k) to pay the rent or make a car payment could benefit from credit counseling. People who treat their 401(k) like an emergency fund are living too close to the edge.

Make sure your job is secure. Before borrowing, think carefully about your employer’s financial condition and your relationship with your boss. The most important thing by far is to not lose your job while you have a loan outstanding. If you can avoid that, things tend to work out OK. More than 90% of plan loans are repaid on time.

Learn how to shake up your financial routine at the Best New Ideas in Money Festival on Sept. 21 and Sept. 22 in New York. Join Carrie Schwab, president of the Charles Schwab Foundation.

This column first appeared on Humble Dollar. It was republished with permission.

Greg Spears is HumbleDollar’s deputy editor. Greg is also a Certified Financial Planner certificate holder.