How using scikit-learn in Spark could save the day

We have about 80 features and a binary class response. For interpretability purposes, we want to build a logistic regression. Further, since we have many features that are highly correlated, we need a regularization technique that trims down the features and avoids over-fitting. This is where lasso comes into action. Our data has a couple of billion records. We use Spark for almost all our analyses because of the size of our data. However, the logistic regression with L1 penality in Spark does return all the coffiecients non-zero. I tried by taking a small size sample (0.01%) in scikit-learn and it returns zero values for many of the coefficients but with the same data in Spark, lasso returns non-zero values for all the cofficients. I guess this has to do with the solver (algorithm to use in the optimization problem). Scikit-learn has a couple of solvers for lasso logistic regression. The one I am using below is SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives. Spark, on the other hand, seems to use vanila gradient descent and with our problem it is not working right.

We may want to use scikit-learn with Spark when:
1- training a model in scikit-learn takes so long
2- the machine learning algorithm we want to use does not exist in Spark but exists in scikit-learn
3- the optimization technique we want does not exists in Spark but exists in scikit-learn
4- we know scikit-learn but not Spark

when the data is imbanaced, in scikit-learn you can use buit-in class_weight parameter to penalize the dominant class. In Spark, you can create WeightCol parameter