Currently the speed of your mobile pages does not impact your mobile rankings, however this may soon change.
Google will be updating the page speed ranking factor to specifically look at the page speed of your mobile pages when it comes to the mobile-friendly algorithm.
The issue today is that many of the ranking signals Google currently uses for mobile rankings are based on your desktop web pages, not mobile web pages. So if you have a really fast desktop web page, but the mobile version is really slow, it currently doesn’t hurt your mobile rankings.
When Google updates their mobile-friendly algorithm, they hope to add mobile-specific page speed as a factor and not rely on the desktop version.