Most companies track whether their people are using AI. This study asks a sharper question: how sophisticated is the use? The authors watched 713,564 real employee prompts from nearly 4,000 back-office staff across 15 functions over eight months. The results are uncomfortable reading for anyone who has bet on a generic training course.

Here is the headline. Sophistication did not improve over time, and formal AI training produced no lasting gains. That challenges how most enterprises roll out generative AI.

Why This Paper Matters

Adoption is a lagging comfort metric. A dashboard that says 60% of employees opened a chat tool tells you almost nothing about whether they are getting value. This paper measures the thing leaders actually care about: how well people use the model, not just that they touched it.

The authors built composite measures of sophistication from the prompts themselves. Prompt clarity, deliberate strategy use, and use-case diversity. That is a real, observable AI assessment for companies, drawn from conversation logs rather than surveys people fill out once and forget.

For any Chief AI Officer, this reframes the job. The bottleneck is not access and it is not a training budget. It is how expertise gets applied to the prompts in the first place. That is a talent and work-design problem, not a delivery problem.

Methodology, Explained Simply

Think of it like seasoning a dish. Anyone can watch the model answer simple questions, the equivalent of boiling water. Sophisticated use is knowing what to ask, when to press for details, and how to combine several tools into one workflow. That takes judgment, and judgment comes from knowing the domain.

The team observed 4,000 back-office employees across 15 functions for eight months. They scored every prompt on how clearly it was written, whether it deliberately used a strategy, and how many different use cases each person touched. Then they compared sophistication across seniority levels, across functions, and over time.

They also tracked what formal AI training did. Past training, hours of courses, the whole standard program. Here is the trigger: comparing employees to themselves, training showed no reliable positive effect, and heavier training even turned slightly negative.

Results and Practical Insights

Three findings matter. Senior employees used GenAI more sophisticatedly, and the gap was not small. Sophistication was highest in Strategy, Digital Innovation, and Project Management. And it did not improve with practice, nor did training move it.

I did not expect the training result to be that flat. Most of us assume a good workshop plus eight months of reps compounds into something better. This data says otherwise. The people getting real value walked in with deep domain expertise and applied it to how they used the model.

The signals that did track sophistication are cheap to observe. Employees who wrote longer first prompts and iterated more got measurably better results. You can read those from conversation logs today, without waiting for a survey or a quarterly review.

Key Takeaways for Operations and Finance Leaders

Thanks to All Authors