Tech Talk at PyCon JP 2024

Published on 2024-10-05

I gave a tech talk on Friday, September 27th, at PyCon JP 2024 about developing a machine learning system to reply to customer inquiries with high precision.

If you’ve interacted with a customer support chatbot, you’ve likely had a frustrating experience. Even with today’s advanced machine learning models, I feel issues like misunderstandings, irrelevant responses, and endless loops without reaching a human agent are common.

Given these issues, I discussed why we want to use AI in customer support at Mercari in the first place, how we ensure excellent UX, and how we developed our highly precise system. Our three main ingredients for a high-precision system are:

Utilizing lots of relevant metadata along with the text of the inquiry
Threshold tuning of the ML model
Using rules based on domain knowledge

I also shared the metrics we use for A/B testing and the business impact we have achieved so far.

Let me know if you have any feedback about the talk. Cheers!