As anyone who's hosted their own kubernetes will attest, there are a number of hoops you need to get over to get your cluster operating like a hosted Kubernetes offering such as GKE and EKS to mention a few.
First, the Kubernetes Loadbalancer service is glue code which will interact with the API to spin up cloud load balancers in whatever cloud ecosystem your cluster is running on. A luxury you may not have when self-hosting.
To go about this, people and organizations have used iptable based solutions or Nginx would sit infront of the cluster reverse proxying traffic to a nodeport. The downside of these solutions is they result in administration overhead and become a bottleneck and a single point of failure.
Fortunately, with the birth of MetalLB, these solutions are no longer necessary.
Second, which option to go with as regards presenting persistent storage will depend on budget and resources at hand. In my case, I first tried iSCSI however noticed that despite the nodes logging into the portal and mounting the target, kubelet would complain about not getting the path to the target.
Searching on Google seemed to result in other people experiencing the same behavior but no detailed solutions. I will need to revisit at a later time. However, an alternative would be to use rook which deploys a ceph cluster within your Kubernetes cluster.
My deployment of rook is only ten hours old at the time of writing this article but I hope to write a detailed article of my setup in days to come.
Decisions on the network plugins to use is often a huge decision but it was not in my case as my cluster is only for my personal use. Beyond the support for network policies and ease of deployment not much thought went into this piece. You could however, think about encryption, ease of deployment, performance and resource usage to mention a few.