Upanshu,

The error message you're getting is what's expected if you still have SELinux enabled on that system.  I don't think you need to worry about lctl or the LNET settings right now - The mkfs.lustre is failing because it can't make the temporary directory.

I'd make sure selinux is set to disabled in /etc/selinux/config.
The line which starts 'selinux=' should read 'disabled'.

You can check whether or not that setting has taken effect with:
getenforce

I'd suggest making sure of that config file change and restarting the system, then try your format command again.

- Patrick Farrell