Difference between revisions of "Troubleshoot NFS"

From FOG Project
Jump to: navigation, search
(Mounting the remote directories)
 
(52 intermediate revisions by the same user not shown)
Line 1: Line 1:
Article under construction. Last modified on May 25th, 2015
 
 
Some sections are inaccurate/incomplete/nonworking/badly organized. Please check back in a few days.
 
  
  
 +
== NFS's roles in FOG ==
  
== NFS's roles in FOG ==
+
NFS is used to transfer images to and from hosts in FOG, and is used on both the host and server. The server's setting file controls what files & directories are exported, and their options. NFS allows writing to the /images/dev directory and allows reading from the /images directory. During imaging, the host mounts either /images/dev (for capturing an image) or /images (for deploying an image).
  
NFS is used to transfer images to and from clients in FOG, and is used on both the client and server. The server's setting file controls what files & directories are exported, and their options. NFS allows writing to the /images/dev directory and allows reading from the /images directory. During imaging, the client mounts either /images/dev (for uploading or capturing an image) or /images (for downloading or deploying an image).
+
During capture, FOS (fog operating system) captures images to/images/dev/<MAC Address Of Client>
  
During upload/capture, NFS uploads images into a folder in /images/dev/<MAC Address Of Client>
+
During download/deployment, NFS downloads images from/images/<Image Path>
During download/deployment, NFS downloads images from /images/<Image Path>
 
  
Please note that FTP is used to move images from /images/dev to /images.
+
Please note that FTP is used to move images from /images/dev to /images. Also note that an imported directory can not be re-exported. For example, if you have a remote share mounted to your FOG server, you cannot export this directory via /etc/exports for imaging purposes. If you'd like your storage to be remote, an additional storage node or an FTP & NFS capable NAS device must be used.
  
 
== Testing NFS ==
 
== Testing NFS ==
  
  
=== Using a FOG debug deployment for testing (easy/quick way) ===
+
=== Using a FOG debug deployment for testing (easy & quick way) ===
  
 
The first thing we must do is create a test file that we use to test with. On your FOG server:
 
The first thing we must do is create a test file that we use to test with. On your FOG server:
Line 29: Line 26:
  
 
At the client, if it did not WOL, turn it on.
 
At the client, if it did not WOL, turn it on.
After the client shows options on the screen, you can press [enter] to be given a command prompt.
+
After the client shows options on the screen, you can press [enter] twice to be given a command prompt.
  
 
You first need to create two directories to mount to:
 
You first need to create two directories to mount to:
Line 38: Line 35:
 
Next, we will mount to FOG's remote image directories like this:
 
Next, we will mount to FOG's remote image directories like this:
  
<pre>mount -o nolock,proto=tcp,rsize=32768,intr,noatime $storage /images
+
<pre>mount -o nolock,proto=tcp,rsize=32768,intr,noatime x.x.x.x:/images /images
mount -o nolock,proto=tcp,rsize=32768,intr,noatime 10.0.0.3:/images/dev/ /images/dev</pre>
+
mount -o nolock,proto=tcp,rsize=32768,intr,noatime x.x.x.x:/images/dev/ /images/dev</pre>
  
  
Next, we will execute a command that will test download and upload at the same time. We will attempt to read /images/test.txt and write that file to /images/dev/test.txt
+
Next, we will execute a command that will test the NFS aspects of deploy and capture at the same time. We will attempt to read /images/test.txt and write that file to /images/dev/test.txt
  
 
<pre>cp /images/test.txt /images/dev/test.txt</pre>
 
<pre>cp /images/test.txt /images/dev/test.txt</pre>
Line 49: Line 46:
 
<pre>cat /images/dev/test.txt</pre>
 
<pre>cat /images/dev/test.txt</pre>
  
 +
=== Using a separate Linux machine for testing (hard & long way) ===
  
 +
==== below r3472 (1.2.0 is below r3472, if you're above this, proceed to 2.2.2)====
  
 +
inside /etc/exports on the FOG server,
 +
Set the fsid for /images to 0 and /images/dev to 1
  
 +
<pre>sudo vi /etc/exports</pre>
  
=== Using a separate Linux machine for testing (hard/long way) ===
+
instructions on using Vi: [[Vi]]
  
 +
Save that, exit that, then run this:
 +
<pre>exportfs -a</pre>
  
==== Mounting the remote directories ====
+
'''Restart NFS and RPC''' (see services below)
  
Linux must have an existing directory to mount remote directories to. Below, we create two directories and mount to each of them. On a separate Linux machine (not your FOG server), you can edit the "mount" command to mount a remote directory. After the mounts have been configured, we can test uploading a file and downloading a file.
+
After restarting NFS and RPC, '''proceed to the steps just below''':
 +
 
 +
==== r3473 and above ====
 +
 
 +
 
 +
The first thing we must do is create a test file that we use to test with. On your FOG server:
 +
<pre>echo 'This is the text I use to test with.' > /images/test.txt</pre>
  
 
At the CLI of the '''separate Linux machine''':
 
At the CLI of the '''separate Linux machine''':
  
 
Create local directories
 
Create local directories
 +
 
<pre>mkdir /images
 
<pre>mkdir /images
 
mkdir /images/dev</pre>
 
mkdir /images/dev</pre>
  
 +
You can configure the mounts for the /images and the /images/dev directories of your FOG server on your separate Linux machine by executing these two commands:
  
 +
<pre>mount x.x.x.x:/images /images
 +
mount x.x.x.x:/images/dev /images/dev</pre>
  
You can configure the mounts for the /images and the /images/dev directories of your FOG server on your '''separate Linux machine''' by executing these two commands:
 
 
<pre>mount 10.0.0.3:/images/dev /FOGdev
 
mount 10.0.0.3:/images /FOGimages</pre>
 
  
Your /images and /images/dev FOG directories are now mounted.
+
Next, we will execute a command that will test the NFS aspects of deploy and capture at the same time. We will attempt to read /images/test.txt and write that file to /images/dev/test.txt
  
==== Testing Uploading ====
+
<pre>cp /images/test.txt /images/dev/test.txt</pre>
  
 +
If you recieved no errors, you're probably good to go. You can confirm all went well by looking at the contents of the moved file:
 +
<pre>cat /images/dev/test.txt</pre>
  
You may test uploading to the /images/dev using the '''seperate Linux machine''' with this:
+
== NFS & RPC / Portmap Service ==
<pre>echo 'This is the text that I will upload' > /uploadtest.txt
 
cp /uploadtest.txt /FOGdev/uploadtest.txt</pre>
 
  
You may confirm that the file transfered by viewing it on your '''FOG server''' using:
 
<pre>cat /images/dev/uploadtest.txt</pre>
 
  
Remove the test file from your '''FOG Server''' like this:
 
<pre>rm /images/dev/uploadtest.txt</pre>
 
  
==== Testing Downloading ====
+
=== Fedora 20/21/22/23/24 & CentOS 7 ===
You may test downloading from the /images directory like this:
 
  
 
+
NFS Status:
On your '''FOG SERVER''', execute this:
 
<pre>echo 'This is the text that I will download' > /images/downloadtest.txt</pre>
 
 
 
On your '''SEPERATE Linux machine''', execute this:
 
<pre>cp /FOGimages/downlaodtest.txt /downloadtest.txt</pre>
 
 
 
You may view the file on your '''separate Linux machine''' like this:
 
<pre>cat /FOGimages/downloadtest.txt</pre>
 
 
 
Remove the test file from your '''FOG Server''' like this:
 
<pre>rm /images/downloadtest.txt</pre>
 
 
 
== NFS Service ==
 
 
 
 
 
 
 
=== Fedora 20/21 ===
 
 
 
*Check the status of NFS with
 
 
<pre>systemctl status nfs-server</pre> (should be on and green, no errors, and enabled)
 
<pre>systemctl status nfs-server</pre> (should be on and green, no errors, and enabled)
 
 
*stop, start, restart NFS service
 
<pre>systemctl stop nfs-server</pre>
 
<pre>systemctl start nfs-server</pre>
 
  
 
The restart command is most useful, if any errors are encountered during manual start/restart, they are displayed.
 
The restart command is most useful, if any errors are encountered during manual start/restart, they are displayed.
Line 123: Line 104:
 
Enable NFS on boot:
 
Enable NFS on boot:
 
<pre>systemctl enable nfs-server</pre>
 
<pre>systemctl enable nfs-server</pre>
 +
 +
 +
RPC Status:
 +
<pre>systemctl status rpcbind</pre>
 +
 +
Restart RPC:
 +
<pre>systemctl restart rpcbind</pre>
 +
 +
Enable RPC on boot:
 +
<pre>systemctl enable rpcbind</pre>
  
 
=== Ubuntu ===
 
=== Ubuntu ===
Line 132: Line 123:
 
<pre>sudo service nfs-kernel-server restart</pre>
 
<pre>sudo service nfs-kernel-server restart</pre>
  
 +
Enable on boot:
 +
<pre>update-rc.d nfs-kernel-server defaults</pre>
 +
 +
RPC status:
 +
<pre>sudo service rpcbind status</pre>
  
Enable on boot ???
+
Restart RPC:
<pre>Code to enable on boot here ??? </pre>
+
<pre>sudo service rpcbind restart</pre>
  
== NFS Settings File ==
+
Enable RPC on boot:
 +
<pre>update-rc.d rpcbind defaults</pre>
  
 +
== FOG's web console NFS settings ==
  
 +
Applies to FOG 1.2.0 and up.
  
=== Fedora 20/21 ===
+
Inside of here:
  
The primary NFS settings file for Fedora 20/21 is located here:
+
Storage Management -> [YourStorageNode] -> Image Path
<pre>/etc/exports</pre>
 
  
 +
This is the actual path of the images directory on the storage node. In a basic installation, the storage node is hosted on the FOG server itself. The image path should be reflected inside of the /etc/exports file (See NFS Settings below).
  
On a standard FOG install where everything is self-contained on one system, it should look like this:
+
Most commonly, the image path is simply:
<pre>sample config file here</pre>
+
<pre>/images</pre>
  
=== Ubuntu ===
+
== NFS Settings File ==
  
The primary NFS settings file for Ubuntu is located here:
+
The primary NFS settings file is located here:
 
<pre>/etc/exports</pre>
 
<pre>/etc/exports</pre>
 +
 +
To view the contents of the file:
 +
<pre>cat /etc/exports</pre>
  
  
 
On a standard FOG install where everything is self-contained on one system, it should look like this:
 
On a standard FOG install where everything is self-contained on one system, it should look like this:
<pre>sample config file here</pre>
+
<pre>/images *(ro,sync,no_wdelay,no_subtree_check,insecure_locks,no_root_squash,insecure,fsid=0)
 +
/images/dev *(rw,async,no_wdelay,no_subtree_check,no_root_squash,insecure,fsid=1)
 +
</pre>
 +
 
 +
To edit the file:
 +
<pre>sudo vi /etc/exports</pre>
 +
 
 +
Instructions on using Vi: [[Vi]]
  
 
== Creating & verifying .mntcheck files ==
 
== Creating & verifying .mntcheck files ==
  
.mntcheck is a hidden and empty file that a FOG client uses during image upload/capture and image download/deployment to verify an NFS share is mounted correctly.
+
.mntcheck is a hidden and empty file that a FOG client uses during image capture and image download/deployment to verify an NFS share is mounted correctly.
  
  
Line 172: Line 181:
  
 
This should return two results. One for /images and one for /images/dev
 
This should return two results. One for /images and one for /images/dev
 +
 +
Permissions should also be set appropriately for these files. See the permissions section.
  
 
== Disable & Verify Firewall ==
 
== Disable & Verify Firewall ==
Line 180: Line 191:
  
 
== Permissions ==
 
== Permissions ==
 +
 +
For the purposes of this article (troubleshooting), the /images directory should have 777 permissions set recursively.
 +
 +
You can do that like this:
 +
<pre>chmod -R 777 /images</pre>
 +
 +
== Common problems and fixes ==
  
  
Permissions go here ???
+
=== My problem isn't in the WiKi! ===
  
 +
{{:My problem isn't in the WiKi!}}
  
  
  
  
== Common problems and fixes ==
+
=== Image Capture: Error Checking Mount ===
 +
 
 +
 
 +
The Client mounts the NFS share successfully but throws an error while checking the mounted Share.
  
 +
<pre>
 +
* Preparing to send image file to server
 +
* Mounting File System......................................mount:ser.ver.ipa.ddr:/data/images/dev/ failed, reason given by server: Permission denied
 +
Done
  
=== My problem isn't in the WiKi! ===
+
* Checking Mounted File System ........................
 +
###########################################################################################
 +
#                                                                                        #
 +
# An error has been detected                                                              #
 +
#                                                                                        #
 +
###########################################################################################
 +
 +
Fatal Error: Failed to mount NFS Volume.
 +
 +
###########################################################################################
 +
#                                                                                        #
 +
# Computer will reboot in 1 minute.                                                      #
 +
#                                                                                        #
 +
###########################################################################################
  
{{:My problem isn't in the WiKi!}}
+
</pre>
  
 +
The error in FOG 1.3.0 is "Could not mount images folder (/bin./fog.download)" as pictured below.
  
 +
[[File:Could not mount images folder.png]]
  
 +
In the folder on your server, check for the '''.mntcheck files'''. If these are not there then perform the following commands [Linux Systems Only]
  
=== Image Upload: Error Checking Mount ===
+
'''touch /images/dev/.mntcheck '''
  
{{:Image Upload: Error Checking Mount}}
+
'''touch /images/.mntcheck '''
  
----
 
  
== Gibberish / notes (temporary work space till article completion) ==
+
If still receiving the same error message after perfroming the above commands, you may need to edit /etc/exports to include your new mount point, i.e. /data/images and /data/images/dev with corresponding permissions.
  
 +
See following examples of /etc/exports:
  
*testing
+
Example 1
  
*mounting
+
<pre>/images *(ro,sync,no_wdelay,no_subtree_check,insecure_locks,no_root_squash,insecure,fsid=0)
 +
/images/dev *(rw,async,no_wdelay,no_subtree_check,no_root_squash,insecure,fsid=1)
 +
</pre>
  
*/etc/exports
+
Example 2
  
*permissions
+
<pre>/data/images *(ro,sync,no_wdelay,no_subtree_check,insecure_locks,no_root_squash,insecure,fsid=0)
 +
/data/images/dev *(rw,async,no_wdelay,no_subtree_check,no_root_squash,insecure,fsid=1)
 +
</pre>
  
*writing/uploading
 
  
*reading/downloading
+
----

Latest revision as of 06:37, 7 January 2017


NFS's roles in FOG

NFS is used to transfer images to and from hosts in FOG, and is used on both the host and server. The server's setting file controls what files & directories are exported, and their options. NFS allows writing to the /images/dev directory and allows reading from the /images directory. During imaging, the host mounts either /images/dev (for capturing an image) or /images (for deploying an image).

During capture, FOS (fog operating system) captures images to: /images/dev/<MAC Address Of Client>

During download/deployment, NFS downloads images from: /images/<Image Path>

Please note that FTP is used to move images from /images/dev to /images. Also note that an imported directory can not be re-exported. For example, if you have a remote share mounted to your FOG server, you cannot export this directory via /etc/exports for imaging purposes. If you'd like your storage to be remote, an additional storage node or an FTP & NFS capable NAS device must be used.

Testing NFS

Using a FOG debug deployment for testing (easy & quick way)

The first thing we must do is create a test file that we use to test with. On your FOG server:

echo 'This is the text I use to test with.' > /images/test.txt


Select the problematic client from your Hosts list in the web UI, Choose "Basic Tasks", then pick download. Create an immediate debug task. See picture below:

Debug Download Task.png


At the client, if it did not WOL, turn it on. After the client shows options on the screen, you can press [enter] twice to be given a command prompt.

You first need to create two directories to mount to:

mkdir /images
mkdir /images/dev


Next, we will mount to FOG's remote image directories like this:

mount -o nolock,proto=tcp,rsize=32768,intr,noatime x.x.x.x:/images /images
mount -o nolock,proto=tcp,rsize=32768,intr,noatime x.x.x.x:/images/dev/ /images/dev


Next, we will execute a command that will test the NFS aspects of deploy and capture at the same time. We will attempt to read /images/test.txt and write that file to /images/dev/test.txt

cp /images/test.txt /images/dev/test.txt

If you recieved no errors, you're probably good to go. You can confirm all went well by looking at the contents of the moved file:

cat /images/dev/test.txt

Using a separate Linux machine for testing (hard & long way)

below r3472 (1.2.0 is below r3472, if you're above this, proceed to 2.2.2)

inside /etc/exports on the FOG server, Set the fsid for /images to 0 and /images/dev to 1

sudo vi /etc/exports

instructions on using Vi: Vi

Save that, exit that, then run this:

exportfs -a

Restart NFS and RPC (see services below)

After restarting NFS and RPC, proceed to the steps just below:

r3473 and above

The first thing we must do is create a test file that we use to test with. On your FOG server:

echo 'This is the text I use to test with.' > /images/test.txt

At the CLI of the separate Linux machine:

Create local directories

mkdir /images
mkdir /images/dev

You can configure the mounts for the /images and the /images/dev directories of your FOG server on your separate Linux machine by executing these two commands:

mount x.x.x.x:/images /images
mount x.x.x.x:/images/dev /images/dev


Next, we will execute a command that will test the NFS aspects of deploy and capture at the same time. We will attempt to read /images/test.txt and write that file to /images/dev/test.txt

cp /images/test.txt /images/dev/test.txt

If you recieved no errors, you're probably good to go. You can confirm all went well by looking at the contents of the moved file:

cat /images/dev/test.txt

NFS & RPC / Portmap Service

Fedora 20/21/22/23/24 & CentOS 7

NFS Status:

systemctl status nfs-server
(should be on and green, no errors, and enabled)

The restart command is most useful, if any errors are encountered during manual start/restart, they are displayed.

systemctl restart nfs-server

Enable NFS on boot:

systemctl enable nfs-server


RPC Status:

systemctl status rpcbind

Restart RPC:

systemctl restart rpcbind

Enable RPC on boot:

systemctl enable rpcbind

Ubuntu

NFS status:

sudo service nfs-kernel-server status

The restart command is most useful, if any errors are encountered during manual start/restart, they are displayed.

sudo service nfs-kernel-server restart

Enable on boot:

update-rc.d nfs-kernel-server defaults

RPC status:

sudo service rpcbind status

Restart RPC:

sudo service rpcbind restart

Enable RPC on boot:

update-rc.d rpcbind defaults

FOG's web console NFS settings

Applies to FOG 1.2.0 and up.

Inside of here:

Storage Management -> [YourStorageNode] -> Image Path

This is the actual path of the images directory on the storage node. In a basic installation, the storage node is hosted on the FOG server itself. The image path should be reflected inside of the /etc/exports file (See NFS Settings below).

Most commonly, the image path is simply:

/images

NFS Settings File

The primary NFS settings file is located here:

/etc/exports

To view the contents of the file:

cat /etc/exports


On a standard FOG install where everything is self-contained on one system, it should look like this:

/images *(ro,sync,no_wdelay,no_subtree_check,insecure_locks,no_root_squash,insecure,fsid=0)
/images/dev *(rw,async,no_wdelay,no_subtree_check,no_root_squash,insecure,fsid=1)

To edit the file:

sudo vi /etc/exports

Instructions on using Vi: Vi

Creating & verifying .mntcheck files

.mntcheck is a hidden and empty file that a FOG client uses during image capture and image download/deployment to verify an NFS share is mounted correctly.


To create these files, on the FOG server:

touch /images/.mntcheck
touch /images/dev/.mntcheck

Verify these files with:

ls -laR /images | grep .mntcheck

This should return two results. One for /images and one for /images/dev

Permissions should also be set appropriately for these files. See the permissions section.

Disable & Verify Firewall

Fedora 20/21/22/23

Disable/stop Firewall

systemctl disable firewalld.service
systemctl stop firewalld.service

Can be undone with "start" and "enable".

Check Firewall in Fedora 20/21/22/23

systemctl status firewalld.service

Fedora 16

Add /bin/bash to /etc/shells as the vsftpd yum install does not do it correctly causing tftp timeout message


Debian/Ubuntu

sudo iptables -L

If disabled, the output should look like this:

Chain INPUT (policy ACCEPT)
target prot opt source destination 

Chain FORWARD (policy ACCEPT)
target prot opt source destination 

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Disable Ubuntu Firewall

sudo ufw disable

Disable Debian Firewall

iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT

Other debian settings:

/etc/hosts.deny

This setting in the above file will deny traffic from any source except locally:

ALL:ALL EXCEPT 127.0.0.1:DENY

Comment out this line like so:

#ALL:ALL EXCEPT 127.0.0.1:DENY

Windows 7

Start -> Control Panel -> View by "Small icons" -> Windows Firewall -> Turn Windows Firewall On or Off -> Turn off all three.

Configuring firewall on Linux

To set the firewall for Linux to only allow what is necessary, please see the FOG security article.


Permissions

For the purposes of this article (troubleshooting), the /images directory should have 777 permissions set recursively.

You can do that like this:

chmod -R 777 /images

Common problems and fixes

My problem isn't in the WiKi!

If you have a problem with FOG, or have a solution to a problem with FOG, please visit the forums for help or to report your solution. We try to keep the WiKi updated with things found in the forum. You can visit the forum here: FOG Forums



Image Capture: Error Checking Mount

The Client mounts the NFS share successfully but throws an error while checking the mounted Share.

* Preparing to send image file to server 
* Mounting File System......................................mount:ser.ver.ipa.ddr:/data/images/dev/ failed, reason given by server: Permission denied 
Done 

* Checking Mounted File System ........................ 
########################################################################################### 
#                                                                                         # 
# An error has been detected                                                              # 
#                                                                                         # 
########################################################################################### 
 
Fatal Error: Failed to mount NFS Volume. 
 
########################################################################################### 
#                                                                                         # 
# Computer will reboot in 1 minute.                                                       # 
#                                                                                         # 
########################################################################################### 

The error in FOG 1.3.0 is "Could not mount images folder (/bin./fog.download)" as pictured below.

Could not mount images folder.png

In the folder on your server, check for the .mntcheck files. If these are not there then perform the following commands [Linux Systems Only]

touch /images/dev/.mntcheck

touch /images/.mntcheck


If still receiving the same error message after perfroming the above commands, you may need to edit /etc/exports to include your new mount point, i.e. /data/images and /data/images/dev with corresponding permissions.

See following examples of /etc/exports:

Example 1

/images *(ro,sync,no_wdelay,no_subtree_check,insecure_locks,no_root_squash,insecure,fsid=0)
/images/dev *(rw,async,no_wdelay,no_subtree_check,no_root_squash,insecure,fsid=1)

Example 2

/data/images *(ro,sync,no_wdelay,no_subtree_check,insecure_locks,no_root_squash,insecure,fsid=0)
/data/images/dev *(rw,async,no_wdelay,no_subtree_check,no_root_squash,insecure,fsid=1)