To Pilot or Not to Pilot

We planned for 45 minutes of panel. We went almost twice that. Nobody left.

That's the best signal we could have gotten. Last week's Boston Smaht event at The Union Boston, a construction-focused incubator in South Boston, was co-hosted with the Boston chapter of the US PropTech Council as part of Boston Tech Week. The room was intimate, the turnout was solid, and the conversation flat out refused to stop.

We came in expecting to debate whether pilots are worth it. What we got instead was a much more useful conversation: how do you actually run one well, from both sides of the table.

The panel

Caroline Murray, Regional Sustainability Manager · Turner Construction

Molly Sivia, VP of Strategy & Operations · Infinityy

Ziv Levi, CEO & Co-Founder · LeanCon

Christian Reed, CEO · Reekon

Jordan Hayashi, Founder · Bizzen

Moderated by Brian Vaughn · Technify

For buyers: go in with intention, not just curiosity

The clearest thread from the owner side: a pilot is not a test drive. It is a commitment. You are putting staff time, operational bandwidth, and your organization's credibility on the line, even when the vendor is not charging you a dollar.

Caroline Murray made the point that stuck with the room all night. Turner Construction never walks into a pilot without real cost. Union labor, logistics, safety reviews, prequalification, insurance. That is a diversion from business as usual, and vendors rarely account for it when they make the ask. Buyers should name that cost internally before they say yes, not after.

The corollary is equally true. If you are not prepared to commit real resources and real attention, you are not ready to pilot. The technology can work perfectly and the pilot can still fail because your organization was never actually in the game.

For vendors: design for the reality of how buyers actually engage

Jordan Hayashi offered one of the more counterintuitive takes of the night. Bizzen builds software that users can get into, test, and work with on their own, without hand-holding from Jordan's team. The insight: watch the engagement data. When Jordan sees users actively getting into the product independently, that is when he knows there is a real opportunity to put pilot-level effort and support behind it. He is not waiting for a formal pilot to discover whether there is interest. He is using lightweight adoption signals to figure out where to invest.

Christian Reed brought a different dimension from Reekon's experience. Their tools live in the hands of the person doing the work, someone on a job site using a tape measure. But the value surfaces to a completely different person: the stakeholder who acts on the data that worker just captured. Two people, two completely different experiences of the same product. Vendors who only engage one of those relationships are going to find that the pilot looks successful to one side and irrelevant to the other.

Someone has to own the relationship, and Customer Success may not be enough

This came up organically and it hit. Pilots need a dedicated owner who can hold both sides accountable. That work usually falls to Customer Success teams, but CS is not always positioned to push back when the buyer is not holding up their end. There is a structural accountability gap there that nobody in the room had cleanly solved.

Molly Sivia framed the underlying principle well. A pilot should be a partnership, not a transaction. If the vendor is the only party truly responsible for the outcome, that is not a pilot. That is a demo with consequences.

Where do you actually pilot? More complicated than you think

One of the livelier debates of the night: which site do you choose?

The instinct for many buyers is to run pilots at their best-performing site with their most capable team. The A team. The people most likely to make anything work. But there is a real problem with that logic. The A team is probably already overwhelmed from being tapped for every new initiative. And their performance almost certainly will not reflect what the rest of your organization experiences when you try to scale it.

The other instinct is to go the opposite direction. Throw the technology at your hardest, most complex site. If it works there, it works everywhere. Several panelists pushed back on that too. If the site itself has unresolved issues, you may spend the entire pilot fixing things that have nothing to do with the technology, just trying to get the environment stable enough to test what you actually came to test.

The room did not land on a clean answer, which is probably the honest result. But the questions are worth working through before you pick a site, not after you have already committed.

Define success before you start

Ziv Levi put it plainly. Before a pilot begins, both sides need full alignment on timeline, participating stakeholders, and KPIs. Not at the first check-in. Before day one.

The gap that derails more pilots than anything else is not a bad product. It is the assumption that "the pilot succeeded" and "we are ready to scale" mean the same thing. They do not. Define what scaling actually looks like before anyone signs anything.

So where did we land?

The fact that this conversation ran well past its end time with a room full of people still asking questions is itself a signal. Everyone in that space had lived this from one side or the other. The pain is real, and the industry is clearly ready to talk about it more honestly than it usually does.

If you were there, thank you for making it what it was. If you missed it, follow Boston Smaht, the US Proptech Council and The Union Boston. We will keep the conversation going.