I was going to write a slightly academic article about the layers of data, but then the AEO Conference in Birmingham happened. Not only a great event (as usual), but I came away inspired by the “Fearless selling” session with a weekend side mission. Greg Hitchen (CEO Terrapinn), Carsten Holm (MD Diversified UK), Alison Jackson (CEO Nineteen Group), and Caitlin Read (CCO Arc Network) talked about their insights from using conversational data. This sparked an idea, and a fun weekend of building code.
As anyone who’s been on the other end of a sales call with me will have very quickly realised, I’m not a natural salesperson. I veer into detail too quickly. I talk faster when I’m excited. And I rarely explain what I actually do well.
Alison and Greg made a strong case for how conversational data was improving their sales teams performances. So I began to noodle the value of call transcripts. What data can you pull from there? And what can you do to transform this into something useful? So, over the weekend I decided to find out.
From call transcript to sales coach
For a while I’ve been recording calls, mainly to improve my follow-up emails, but realistically most of this data currently serves little purpose. However, conversations are a very rich source of data. A typical 30 min call contains 3,500 to 4,500 words. A sales team of five talking to customers for 20-30 hours a week generates anywhere between 0.7-1.4 million words each week.
“A sales team of five talking to customers for 20-30 hours a week generates anywhere between 0.7 and 1.4 million words each week”
This is a lot of data. Data about what our customers want, how they think and react. Data about how we ourselves sell, interact and perform (somewhat uncomfortably). Some of it, of course, is just white noise.
This is where my exploration of conversational data begins. What sort of data do you need to move from transcripts to personal improvement? What is the data journey to build an AI sales coach in a weekend?
What to do if you don’t know what good looks like
To build an AI sales coach I needed to understand what happens on an excellent B2B sales call. I pretty quickly decided that using myself as a benchmark might not be the best idea here. (This reveals a common modelling challenge: without enough high-quality, labelled examples of best practice, a model has no robust ground-truth against which to evaluate performance.)
What do great salespeople do on calls? If I had a large organisation, I could look at the sales calls of my most successful employees. But I don’t. Even if I did, what actually makes a salesperson effective might not just be a function of individual transcripts. It could in fact be the sequencing of multiple calls and their personal “style”. I didn’t really want to build a sales coach that told me that I would be more successful if I only developed a Scottish accent or an Irish lilt.
What I did have, however, was a highly intelligent language model – trained on books and web articles on B2B sales approaches. Something that knows the difference between SNAP, MEDDPICC, SPIN and Sandler Systems, and could come up with some clever prompts to explore the quality of my sales conversations. We could use a language model to create data about data.
As I fleshed this concept out, it became clear that there are a number of data layers needed to extract insight from raw transcripts. The code first turned conversational data into structured data. The structured data was then fed into a series of sales evaluation prompts which evaluated the data across 13 competency areas. Only once evaluation had been performed, did a further layer of prompts identify coaching feedback opportunities.
It sounds complex, but by taking the source data and transforming it in layers and using many different prompts created a rich data source which combined some smart sales advice anchored to reality. In simple terms my sales coach used data and layered prompts to become better at showing its workings.
If at first you don’t succeed, try prompting again with a different phrasing
Evaluation and coaching is not the same as learning. I didn’t want my coach to provide the same feedback every time. I wanted to get better, try new things, be encouraged and learn. With this mindset I added three other ideas into my build – two data-focused and one experience-focused.
Firstly, mindful that my style of learning is likely different from other people’s, I created a Google form which evaluated my learning style and created a data record which could be used to tailor feedback. This created a data lens through which feedback could be interpreted. While it sounds complex, the code asked me some questions, and then turned the answers into a guide as to how to communicate with me more effectively. It was interesting to see how the responses changed with different answers. Personalisation worked for me.
Secondly, I wanted the tool to not continue to remind me of my failures and rather help me learn. I might value being told that I failed to explore the commercial consequences of the customer’s problem once or twice, but after the tenth reminder I was worried I might just ignore it. So my coach needed some form of memory. What feedback had I received before? Did my behaviour change? Am I improving? The code therefore included a memory data file which was updated and referred to. I’m not quite sure how this will actually evolve but the initial “memories” were intriguing.
Thirdly, I realised I needed this to be something that I actually used. It couldn’t just sit in a platform or as a CRM entry. It needed to be something which was both useful and solved a problem for me. To this effect, I made the return path e-mail – rather than just serving feedback online.
A webhook identified that a recorded call happened in my CRM, the transcript was collected and analysed and an email was drafted and sent to me. Not only did this include coaching, but it also included a short follow-up email draft for my edits. This approach made the email a “must-open”, as it automated the “white page” problem of drafting follow up email.
It’s early days with my build. Everything is plumbed in, and I’m getting sensible feedback e-mails from past calls in a style which feels encouraging. It has decent architecture behind it, but it’s definitely not scalable. It’s also clear it’s been built over a weekend (which also included watching the very much underrated Acme vs Coyote movie). I do think however this will be something that I think is likely to be useful to me and I’m keen to see what advice it gives over the next few months.
The privacy issues
Before I wrap up, there are two privacy questions worth sitting with.
Firstly, conversational data is private data and the desire to do interesting things with this should never override confidentiality. Whenever you are sending data via an API, you’re sharing this data with a third party. Everything above ran locally (including a 28bn parameter language model) to preserve privacy. To scale what I built to encompass multiple sales people (a single transcript to call takes 20-30 mins locally) you’d need to understand the data logging, retention and commercial use clauses of your LLM provider, or build something more complex using virtual compute which you effectively “controlled” albeit being in the cloud.
Secondly, if conversational data is used there is a need to not make every interaction the final exam. People learn through making mistakes, trying new things, receiving feedback and improving. While conversational data and AI gives the ability to score and evaluate every call, there is a massive privacy impact of evaluating people in real time. Once a tool is used for surveillance and performance evaluation of employees I worry it loses all objectivity as a coach. Getting things wrong is not a failure of the learning process. It is part of the learning process.
The EU AI Act identified that any AI used for worker management is high risk and requires associated safeguards. While a manager dashboard to evaluate individual performance or call scores are inherently attractive, moving conversational data from private coach to public evaluation of on-job performance is a really dangerous step. Luckily I don’t have a manager to evaluate me, and am proud of my weekend experiment. Fingers crossed I pay attention to the advice…
