HTTPS

Secure connections like HTTPS have become state of the art all over the web over the years. While FOG did use secure encryption (not HTTPS but a custom secure channel) for the fog-client communication since 2016 already the FOG web UI was still using plain HTTP. Using HTTPS is not as easy as generating a certificate and setting Apache to use it because PXE boot also relies on HTTP(S) communication with the FOG server.

Installation

We try to make setting up a fully HTTPS-enabled FOG server more convenient and encourage people to use it but still don't consider it wise to make it the default yet because it's a complex topic (FOG server, PXE boot, fog-client). Up until now you had to use the command line switch "--force-https" to enable HTTPS but with FOG 1.5.8 the installer will ask you if it should be enabled or not. Both ways you will end up with FOG run in HTTPS mode.

The installer will generate a different Apache configuration to enable HTTPS and redirect all requests from HTTP to HTTPS (minor exceptions exist). While this might sound simple there is really a lot more to it in the FOG world. Read on to learn about this in detail.

FOG web UI

The Web UI will be accessible through the new URL https://fogserver/fog/ but will also redirect requests going to the old URL to HTTPS.

All fine but now I get warnings in my web browser saying this connection is not secure. Yes, this is because we can't offer certificates signed by an official certificate authority (which your web browser would trust). We use self generated CA and certificates which are not known to your browser. You can either ignore the warning or grab /opt/fog/snapins/ssl/CA/.fogCA.pem from your FOG server (e.g. using WinSCP or scp) and import that to your (browser) certificate store.

• Firefox: Preferences -> Privacy & Security -> Certificates -> View Certificates -> Your Certificates -> Import...
• Chrome: Settings -> Show advanced settings -> HTTPS/SSL -> Manage certificates -> Your Certificates -> Import...
• Opera: Browser settings -> Advanced -> Privacy & security -> Manage certificates -> Your Certificates -> Import
• IE/Edge: cmd: certutil -addstore -f -user "Root" path\to\.fogCA.pem

PXE boot

When enabling HTTPS the installer compiles custom iPXE binaries for you including your personal FOG server CA certificate to be able to communicate with your secure FOG webserver. Manual adjustments should not be needed for this to work but it's quite likely this is causing trouble for some of you. If you see the error message https://x.x.x.x/fog/service/ipxe/boot.php... Permission denied ... on PXE booting you will be dropped to the iPXE command shell. Running the command certstat will show you the certificates known to iPXE at this stage:

iPXE> certstat

FOG Server CA: ... [PERMANENT]

x.x.x.x: ...

The output might differ from what you see. In this example we see that the FOG Server CA cert is embedded into the binary (permanent) and the following line shows the certificate iPXE received when contacting the webserver but it's unable to validate this cert. If it would be able to check the certificates both lines would be marked as [VALIDATED]. So in this case the CA cert compiled into the binary doesn't match the one which the web server certificate was signed with. More often you might just see no line starting with FOG Server CA. The binary was compiled with no embedded CA cert and iPXE is not to verify the cert received from the webserver.

Either way you need to check your CA and certificate files on your FOG server and take a look at the installer log files in fogproject/bin/error_logs/ to see why it didn't succeed compiling the right certificate into the iPXE binaries.

Wrong system time can cause an issue as iPXE also checks if the embedded root CA certificate is valid based on the time. iPXE receives the current time from the BIOS / UEFI firmware and fails with ... Permission denied ... on the HTTPS connection if it can't validate the root CA cert due to it not being valid with the wrong time set on the machine.

If you can't find what's causing this you might consider re-running the FOG installer using command line options to re-generate the SSL keys and certs. But be aware this will break communication with all your fog-clients talking to this FOG server! We do NOT recommend using this unless you really know what you are doing. Enough warning, here you go: ./installfog.sh --recreate-ca

fog-client

When the new fog-client came to life a few years back it was intended to enable secure communication between client and FOG server without forcing the webserver to HTTPS because the implications with PXE booting seemed too complex to force all users straight away. Therefore an encrypted communication channel was implemented that can be delivered over simple HTTP protocol without changing the webserver configuration.

Now if you enable HTTPS on your FOG server you will need to update your fog-client settings as well. Edit C:\Program Files (x86)\FOG\settings.json and set HTTPS to 1. Save and restart the client.

Custom CA and certificates

In many environments certificates from an internal CA are used. While you can switch over to use your custom cert with FOG you need to be aware of possible culprits. Using a custom CA was given enough thought when HTTPS support was added to FOG and so we need to kind of build around what we currently have - FOG 1.5.8 as of writing this in Feb 2020.

As described above the fog-client software uses internal encryption which is not the same as HTTPS. While switching to HTTPS with a custom CA does work with the fog-client by adjusting the settings.json you need to choose one of two ways for making the internal encryption work as well:

• Re-compile fog-client software to trust your custom CA:

  Replace the strings FOG Server CA and CN=FOG Server CA in RSA.ca and Communication.cs to match your custom CA common name. Then re-compile your own custom fog-client installer binaries. I won't go into any more details for now as I don't think many people will choose this path, it has other drawbacks (talking about auto upgrading) and we'll probably re-think and change this some time in the future anyway.

• Use your custom CA for Apache configuration only but stick to FOG CA for fog-client internal encryption. While it might seem complicated at first I still think this is currently the easiest way to make it work:
  • Install FOG as usual but enable HTTPS (command line parameter or answer yes to the installer question on HTTPS). This will generate the FOG Server CA cert needed for the fog-client internal encryption.
  • Grab your custom CA certificate (1), web server certificate (2) & key (3) file and put those in the following paths (overwriting the original files generated by FOG installer):
    1. CA certificate: /var/www/html/fog/management/other/ca.cert.pem (use PEM format and do not touch the ca.cert.der file in that same folder as this is used by fog-client's internal encryption)
    2. web server certificate: /var/www/html/fog/management/other/ssl/srvpublic.crt (as well PEM format)
    3. web server key file: /opt/fog/snapins/ssl/.srvprivate.key
  • Make sure you have your custom CA certificate imported into the Windows cert store on your client machines and adjusted settings.json to use HTTPS as well.
  • Restart Apache webserver
  • Re-build iPXE binaries using your custom CA and test PXE booting:

sudo -i
cd path/to/fogproject-source/utils/FOGiPXE/
./buildipxe.sh /var/www/html/fog/management/other/ca.cert.pem
cd ../../packages/tftp/
find -type f -exec cp -Rfv {} /tftpboot/{} \;

Notice: Be aware that whenever you re-run the FOG installer (update or for whatever other reason) it will overwrite your custom /var/www/html/fog/management/other/ca.cert.pem and /var/www/html/fog/management/other/ssl/srvpublic.crt file. So you will need to put those two files back in place and restart Apache. Re-compile of iPXE binaries should not be needed. Don't see this as bad intention as it's simply how the installer was created to make sure things were "correct" on every run. We will consider changing this behavior as more people will use custom CAs.

Storage node setups

TBD

Known issues

1. We have seen issues with PXE booting when certificates from a certain vendor were used. Find details here:

   https://forums.fogproject.org/topic/12768/not-able-to-tftp-boot-invalid-argument-error

   http://forum.ipxe.org/showthread.php?tid=16998

2. When changing or re-creating (be careful!) the CA you need to make sure the rootcert part of iPXE is being rebuild.

   Either upgrade to the latest (dev) version or update your build script manually according to the fix you find on github.