Why can't we simply use the podSelector labels of kube-apiserver to keep the network policy generic?

As these IPs vary from cluster to cluster, hard to maintain or version control this network policy!