Unfortunately according to Android Police, the feature isn’t working at the moment, or at least it doesn’t seem to be working for some users. As the report points out, users have been complaining about this issue dating back to November 2017 on the Google Product Forums, but so far it doesn’t seem like there is a fix for it.
Note that recommended departure times are still visible if users were to launch the Google app, where it will also display the upcoming event. However the issue is that the notifications about departure times are not coming through which means that if you don’t check the app, you might miss it completely.
Like we said this problem dates back to last year and there is no clear fix in sight. Google’s community managers have offered some solutions but none of them seem to work, or at least not for everyone. There is a workaround in which users can load the desktop calendar UI on their phone and enable “time to leave” notifications, but this isn’t necessarily the most elegant solution and hopefully Google will have a fix soon.