Currently the speed of your mobile pages does not impact your mobile rankings, however this may soon change.
Google will be updating the page speed ranking factor to specifically look at the page speed of your mobile pages when it comes to the mobile-friendly algorithm.
The issue today is that many of the ranking signals Google currently uses for mobile rankings are based on your desktop web pages, not mobile web pages. So if you have a really fast desktop web page, but the mobile version is really slow, it currently doesn’t hurt your mobile rankings.
When Google updates their mobile-friendly algorithm, they hope to add mobile-specific page speed as a factor and not rely on the desktop version.
Page speed became a factor in 2011; and in 2013 Google hinted a negative factor would come to slow mobile pages. Now they are working on mobile specific page speed. While Google has said this is still in the planning phase, look for this push to come sooner rather than later.
Courtesy of SearchEngineLand