from Hacker News

Volume Monitoring in Kubernetes with Prometheus

by qubit23 on 10/17/17, 12:55 AM with 7 comments

  • by lobster_johnson on 10/17/17, 2:28 AM

    Wouldn't it be better to run node_exporter (with only the fs metrics enabled) in a sidecar container in the pod that mounts the volume? That way, you can get the metrics scraped with the right pod labels. If you do it by node, you will only get the node name as a label.

  • by peachia on 10/17/17, 3:45 AM

    kubelet v1.8 collects volume stats, the proposal linked in the summary was forked, and the fork accepted & implemented: https://github.com/kubernetes/community/pull/930

  • by discordianfish on 10/17/17, 8:31 AM

    Hi, node exporter maintainer here.

    Yes, running the node exporter in a container is still tricky which is why we recommend running it straight on the host. I like the nsenter idea, wondering if it could be integrated without shelling out but I think there were some inherit problems with Goroutines and nsenter preventing that. If your interested in exploring this, feel free to open an github issues. Even better though would be support of unisolated workloads in kubernetes, similar to rkt stage1.

  • by bogomipz on 10/17/17, 3:18 PM

    >" In this post, we discussed monitoring volumes on Kubernetes using Prometheus, node-exporter, and nsenter. One disadvantage of this approach is that node-exporter must be run as a privileged container. In the future, it may be better to have Kubelet collect the volume stats so that we can get the stats directly from Kubernetes instead of running node-exporter as a privileged container."

    I thought that node-exporter was meant to run in a container or on the host itself. Are there any potential downside to the single node-exporter in a side car approach?

  • by alexnewman on 10/17/17, 1:20 AM

    Now I just need a flexdriver which receives the warning and activates the code to create a new ebs volume and adds it to whatever lm k8s volume. EBS gp2 is uselessly slow without just combining them into one super danger mount. As with everything else in amazon, never use one of x. Whenever I use S3 and EBS i put an abstraction between me and them because amazon.