Upanshu,
The error message you're getting is what's expected if you still have SELinux enabled on that system. I don't think you need to worry about lctl or the LNET settings right now - The mkfs.lustre is failing because it can't make the temporary directory.
I'd make sure selinux is set to disabled in /etc/selinux/config.
The line which starts 'selinux=' should read 'disabled'.
You can check whether or not that setting has taken effect with:
getenforce
I'd suggest making sure of that config file change and restarting the system, then try your format command again.
- Patrick Farrell