$HOST_INTERFACE might not exist at runtime

[leprechaun]

So,

I'm running a cluster using kops, which means I have to specify --host-interface cbr0. kube2iam worked fine when I first tested it, on an existing cluster. However, when including kube2iam as a DaemonSet I on a new cluster, it crashes.

What's going on, and I'm only guessing here, the instances won't bother creating the cbr0 bridge until that node runs a Pod w/ hostNetworking: false, at which point kube2iam can successfully run. Anytime before that, it CrashLoops with a message about the interface not existing (however it happens to format this error message, https://github.com/jtblin/kube2iam/blob/master/iptables/iptables.go#L38)

I'm willing to send in a patch, but haven't found good way without downsides. Thinking of something like this:

--wait-for-interface, which would let us start the server, and asynchronously add the iptables forwarding, in one of three ways:

Loop / Sleep while the interface doesn't exist

Ugly, but works. Uses extra resources while polling. How aggressively we poll determines how reliable it can be made (eg, how small the time window between pod creation and pods requesting creds could be)

Netlink / RTMGRP_LINK

http://man7.org/linux/man-pages/man7/netlink.7.html. We can get notifications when network interfaces are created/deleted / ip addresses changed etc. However, I'm unsure about how well go supports it atm.

• syscall: error in definition of constant RTNLGRP_IPV4_IFADDR golang/go#15080
• https://godoc.org/golang.org/x/sys/unix

This might be the best option.

TTL mangling hack

If neither of the above, it would also be possible to mangle all the packets destined to the meta-data service, unless their TTL had been set to a pre-selected number (used as sign that we don't need to proxy again). This would definitely work, but at the cost of a malicious pod being able to circumvent the proxy at obtain the hosts own IAM creds.

Thoughts?

[mgoodness]

This appears to be new behavior with K8s v1.5.1.

[jkemp101]

This is happening for me with v1.4.8 also. I think I'm going to run a tiny daemonset that will hopefully ensure cbr0 is created by the time kube2iam restarts once. Otherwise I'm afraid kube2iam restarts will be backed off and not ready to serve requests if a pod gets scheduled that needs AWS access.

[countergram]

I am also having this issue. Is there any workaround other than a hostNetworking: false daemonset? It seems to work, but I'm worried about race conditions on restarts/scaling.

[benbooth493]

Is there any solution to this?

[mgoodness]

I wonder if, after #38, a work-around is to set --host-interface=cbr+. That should make kube2iam ignore the missing interface instead of erroring. Not ideal, as kube2iam really shouldn't create the iptables rule until/unless the interface exists. But until there's something better...

I'll try this on a test cluster and report back.

[jbraswell]

@mgoodness Were you able to run a test with --host-interface=cbr+?

It would be nice to get a patch in for this. Would just adding a flag to disable the interface check be sufficient?

[jtblin]

Looking at the code passing --host-interface=cbr+ would work. Not ideal, might be better than adding yet another flag.

// CheckInterfaceExists - validates the interface passed exists for the given system, ignores wildcard networks.
func CheckInterfaceExists(hostInterface string) error {
	if strings.Contains(hostInterface, "+") {
		// wildcard networks ignored
		return nil
	}

	_, err := net.InterfaceByName(hostInterface)
	return err
}

[ahawkins]

We're hitting this issue as well. Our solution is to wait to see if things start successfully; if not wait and kill pods until they start. It works but certainly not ideal if you are creating new nodes frequently.

Edit: I'm using flannel so --host-interface=cni0. Here's the logs for the failing kube2iam pod:

workstation $ kubectl logs kube2iam-kube2iam-bm9pj -n kube-system
time="2017-07-19T10:25:19Z" level=info msg="base ARN autodetected, arn:aws:iam::527591747969:role/"
time="2017-07-19T10:25:19Z" level=fatal msg="route ip+net: no such network interface"

[sethsamuel]

Hitting this same problem with kube-aws and --host-interface=cni0. Using a dummy sleep 10000d daemon to kick the interface into existence is not ideal but seems to work.

[TattiQ]

Hi guys, hitting the same issue with cni0 , "route ip+net: no such network interface" . dummy daemonset does not seem to improve the situation much. Some of the pods still start up earlier than kube2iam causing access denied issues. @mgoodness Were you able to run a test with --host-interface=cbr+? If yes, what's the result?

[bnogas]

@TattiQ I can confirm that running with option --host-interface=cni+ (--hostinterface=<type>+) works as a temporary workaround.

[TattiQ]

@bnogas I can now confirm that too! That's the only thing that worked for me , just setting a + sign there to get rid of the interface check.

[Arnold1]

hi, how to fix it? I currently use:
helm install stable/kube2iam --namespace kube-system --name kube2iam --set=extraArgs.base-role-arn=arn:aws:iam::13123123123:role/,extraArgs.default-role=kube2iam-default,host.iptables=true,host.interface=cbr0,rbac.create=true

@jtblin @jbraswell what is the suggested way to fix it?