Diff to HTML by rtfpessoa

Files changed (1) hide show

{1272 → latest}/elsarticle-template.tex +55 -74

{1272 → latest}/elsarticle-template.tex RENAMED Viewed

@@ -66,7 +66,8 @@
 \begin{frontmatter}
-\title{Systems Hardening through the use of Secure Enclaves}
+%\title{Systems Hardening through the use of Secure Enclaves}
+\title{Hardening of cryptographic operations through the use of Secure Enclaves}
 \author[cracs]{André Brandão}
@@ -75,7 +76,7 @@
 \ead{[email protected]}
 \author[cracs]{Rolando Martins}
 \ead{[email protected]}
-\address[cracs]{CRACS, C3P \& Faculty of Sciences, University of Porto, Portugal}
+\address[cracs]{CRACS/INESC-TEC, C3P \& Faculty of Sciences, University of Porto, Portugal}
 \begin{abstract}
@@ -111,6 +112,8 @@
 Given that porting an application to a new environment is usually hard, not much work has been done to attempt running day-to-day server software on Intel SGX. Focus has mainly been on tools that separate code automatically~\cite{lind2017glamdring} or running unmodified applications inside an enclave~\cite{scone, haven, graphenesgx, sgxlkl}. The main issue associated with these approaches is that they do not consider each application's specificity leading to incompatibilities or/and significant performance loss.
+In this work we aim to mimic an hardware security module through an Intel SGX enclave, giving it similar guarantees as an Hardware Security Modules without the inconvenience of requiring the deployment of additional hardware on the host.
 \subsection{Contributions} \label{Objectives}
@@ -146,44 +149,7 @@
 With the evolution of the architecture and the introduction of hypervisors, deeper levels of privilege were introduced. One example of this is the hypervisor mode, also known as ring -1, capable of preempting and isolating kernel code. Ring -2 refers to the system management mode (SMM), which can seize the hypervisor code and has nearly unrestricted access to the system~\cite{domas2015memory}. It is also in charge of controlling power management, system hardware and run proprietary code from Intel and the motherboard manufacturer.
-Trusted execution environments may break the hierarchical layers of privilege by only allowing software to run in a less privileged code and deny access from higher privileged codes.
-\subsection{Virtualization}
-%Description
-Virtualization is the concept of creating a virtual version of something. This is usually achieved by an abstraction layer that allows the creation of virtual "hardware". In this subsection, we will approach the methods utilized to virtualize the hardware the operating system runs off.
-\begin{figure}[ht]
- \centering
- \includegraphics[width=0.49\textwidth]{Figures/VirtualizationTypes(3).png}
- \caption{Common virtualization types}
- \label{fig:VirtualizationTypes}
-\end{figure}
-\subsubsection*{Virtualization via Virtual Machines}
-To be able to create virtual machines, the bare machine must utilize a Virtual Machine Monitor (VMM), also commonly known as a hypervisor. The VMM has full control of the machine resources and aims to create an efficient, isolated virtual machine (a and b in Figure \ref{fig:VirtualizationTypes}) duplicate of the real machine~\cite{popek1974formal}, and this is accomplished by creating a Basic Machine Interface~\cite{buzen1973evolution} for each virtual machine. Buzen and Galglirdi~\cite{buzen1973evolution} define the primary machine interface as the set of all software visible objects and instructions supported by the system's hardware and firmware.
-Nowadays, a hypervisor can be either of type one or type two. Type one is also known as bare-metal hypervisors (b) in Figure \ref{fig:VirtualizationTypes}), meaning there is no other operating system running parallel or beneath the hypervisor, allowing direct access to the hardware without any overhead. As the hypervisor itself is the "Operating System" installed on the physical machine, these types of hypervisors generally either have no interface for the user or a very limited one, requiring another machine to configure it and deploy virtual machines.
-On the other hand, type two hypervisors run above a common operating system such as Windows or Linux and exist mainly for convenience for the end-user to run a virtual machine without requiring a dedicated machine for it. The operating system handled memory allocation and I/O, providing an increased overhead compared to type 1 hypervisors. Type 2 hypervisors may be less flexible on the resources and capabilities they provide, depending on the underlying operating system.
-The only way for malicious or compromised virtual machines to affect others is to attack first the hypervisor or the processor to break the isolation barrier provided. The latter can often be attacked by leveraging, for example, side-channel vulnerabilities to extract information from other virtual machines or even the hypervisor.
-Virtualization via virtual machines has become quite popular as it allows for the same software to run on a panoply of different hardware without requiring changes to the software. They are allowing simultaneously, if the processor supports, to isolate virtual machines at the hardware level completely, e.g. AMD SEV-ES.
-\subsubsection*{Virtualization via Containers}
-Recently there has been an uprise in the interest of lightweight virtualization through containerization (d) in Figure \ref{fig:VirtualizationTypes}). The kernel is responsible for isolating the different containers and protecting itself from it.
-The containers share the kernel with the host, which restricts the operating system that can run under the virtualization to one that shares the same kernel. It also means a bug in the kernel may compromise all the containers associated with it and the host operating system.
-Active isolation, on Linux, is performed via \textit{cgroups}~\cite{zhuang2017taming} and \textit{namespaces}. Cgroups allow for the kernel to limit and measure the resources utilized by a group of processes, while namespaces allow the kernel to limit the processes' visibility of the rest of the system. Because only one kernel is used, the performance and size overhead is significantly reduced compared to a virtual machine.
-\subsubsection*{Virtualization via Library OS}
-A Library OS is a library loaded into the user space of a process providing as many kernel-like features as possible from userspace, only transitioning to kernel space when required, i.e., interfacing with the hardware.
-Compared to containers, the great advantage of this approach is that it dramatically reduces the number of system-calls that kernel must defend itself from the application. It may also be possible to create a host OS-dependent abstraction layer between the library OS and the actual kernel, allowing the same executable to run in completely different environments (e.g., Linux and Windows) or even microkernels that provide minimal functionality.
-As the virtualization provided by a library OS is made purely through userspace, the security boundaries it provides are essentially none, as an attacker can read the whole memory space in the process. Nothing stops them from issuing syscall instructions to interact directly with the kernel without going through the library OS.
+Trusted execution environments may break the hierarchical layers of privilege by only allowing software to run in a less privileged code and denying access to higher privileged code.
 \subsection{Trusted Execution Environment}
 Trusted Execution Environment (TEE) is often referred to as a secure, integrity-protected programmable environment with memory and sometimes storage capabilities~\cite{sabt2015trusted}.
@@ -254,10 +220,8 @@
 \subsection{Running legacy application on Intel SGX}
-Since porting entire applications is typically an arduous task, focus has primarily been on automatically porting applications~\cite{lind2017glamdring} or running unmodified applications inside Intel SGX~\cite{haven, scone,sgxlkl,graphenesgx}. This Section approaches some solutions that aim to run unmodified applications inside the enclave and identify the key differences in each solution.
-\subsubsection{Scone}
-As the enclave in Intel SGX excludes the operating system from its trust computing base (TCB), thus eliminating the \textit{syscall} instruction, it means the number of applications that can run natively, without any changes, on Intel SGX is somewhat limited. This Section will look at a few solutions that aim to run unmodified applications on Intel SGX.
+Since porting entire applications is typically an arduous task, focus has primarily been on automatically porting applications~\cite{lind2017glamdring} or running unmodified applications inside Intel SGX~\cite{haven, scone,sgxlkl,graphenesgx}.
+As the enclave in Intel SGX excludes the operating system from its trust computing base (TCB), thus eliminating the \textit{syscall} instruction, it means the number of applications that can run natively, without any changes, on Intel SGX is somewhat limited. This Section approaches some solutions that aim to run unmodified applications inside the enclave and identify the key differences in each solution.
 \begin{figure}[ht]
  \centering
@@ -270,7 +234,7 @@
 Figure \ref{fig:securecontainers}a shows a design that places a Library OS and a shielding mechanism inside the enclave. The library OS allows the enclave to drastically reduce the number of system calls made to the kernel, thus decreasing the performance penalty associated with leaving and entering the enclave. The shielding layer protects a security-sensitive set of system calls, by, for example, encrypting and decrypting I/O operations. The disadvantage of this type of design is that by integrating the library OS inside the enclave, we are significantly increased the size of TCB.
 %isto não devia aparecer noutro lado? \ref{fig:securecontainers}b?
-Figure \ref{fig:securecontainers}b shows the extreme opposite of the previous design. The application and its libraries are loaded onto the enclave with a shim C library. The shim library intercepts the C library calls and redirects them to the C library that is loaded outside the trusted environment. çç
+Figure \ref{fig:securecontainers}b shows the extreme opposite of the previous design. The application and its libraries are loaded onto the enclave with a shim C library. The shim library intercepts the C library calls and redirects them to the C library that is loaded outside the trusted environment.
 To our knowledge, no current implementation follows this extreme approach as they typically implement some shielding layer. This solution would also imply a significant increase in the number of transitions to and from the enclave, decreasing the application's overall performance.
@@ -322,7 +286,7 @@
 Graphene-SGX~\cite{graphenesgx} handles Intel SGX as just one more environment on which the applications may run. To run unmodified applications under Intel SGX, a port of the PAL was made to Intel SGX. This allows, similarly to Haven, a mutual distrust from the host and the guest. It is one of the few, if not the only solution, that allows the creation of forks of existing processes. When a process is forked, a new clean process is created. Then the two enclaves, via an inter-enclave remote process communication stream, exchange an encryption key, validates the CPU-generated attestation of each other, and migrates the parent process snapshot. The current solution leaves for future work the protection of the network and file system.
-To load an unmodified program to Graphene-SGX, it first creates hashes of the program, its dependencies, saves it to a file, and signs it, essentially creating a whitelist of permitted files. Later, when the application executes inside the enclave, checks the executable and files against the created whitelist.
+To load an unmodified program to Graphene-SGX, it first creates hashes of the program, its dependencies, saves it to a file, and signs it, essentially creating a whitelist of permitted files. Later, when the application executes inside the enclave, checks the executable and the utilized files against the created whitelist.
 \subsection{SGX Native Applications}
 In this Section, we overview some solutions that utilize Intel SGX in order to widen its threat model to have stronger guarantees that the desired data is kept confidential from even the operating system itself.
@@ -345,18 +309,18 @@
 To guarantee that the user is in fact, speaking with a genuine Intel SGX enclave, remote attestation is performed through the same efficient remote attestation protocol as proposed by the SafeKeeper.
 \subsubsection{tpmsgx}
-The utilization of Intel SGX on a cloud-like environment is especially difficuly because data is encrypted with a per enclave per device key and because the physical memory available to the program is at most 128 MB, with around 40 MB being already used for the management of SGX itself.
+The utilization of Intel SGX on a cloud-like environment is especially difficult because data is encrypted with a per enclave per device key and because the physical memory available to the program is at most 128 MB, with around 40 MB being already used for the management of SGX itself.
 Dave Tian et al.~\cite{tian2019practical} propose a system so that applications can leverage Intel SGX in a cloud-like environment for multiple users through the use of an emulated TPM and LXC containers.
-The solution acts like a "TPM as a service" for applications to use without resorting to solutions like Haven~\cite{haven} that requiring moving a big code base to the enclave. Unlike traditional TPMs, it does not allow the TPM use before the operating system is initialized, meaning it cannot be used for secure boot. This service allows for multiple a
+The solution acts like a "TPM as a service" for applications to use without resorting to solutions like Haven~\cite{haven} that requiring moving a big code base to the enclave. Unlike traditional TPMs, it does not allow the TPM use before the operating system is initialized, meaning it cannot be used for secure boot. This service allows for multiple applications to utilize a single enclave as TPM, rather than each application initializing its own enclave.
 The implementation specifies that, if remote attestation is enabled, communication between the server and the client is encrypted by an AES128 shared secret that is established between them. The publication does not specify if or how the client authenticates communicating with a genuine Intel SGX enclave.
 \subsubsection{SGX-Kernel}
 Although Intel SGX is limited to running code in user mode, Lars Richter et al.~\cite{richter2016isolating} propose a solution that allows the kernel to delegate some of the work to an enclave, in order to isolate kernel with Intel SGX.
-The system is compromised by two components, a kernel module, and a secure enclave running in user mode. The kernel module acts as proxy for the secure enclave and their communicated is made through a Netlink interface, allowing the kernel to delegate work to be done in the secure enclave.
+The system is compromised by two components, a kernel module, and a secure enclave running in user mode. The kernel module acts as proxy for the secure enclave and their communication is made through a Netlink interface, allowing the kernel to delegate work to be done in the secure enclave.
 A proof of concept is demonstrated by creating a file system managed by the enclave, which is responsible not only for its storage but encryption as well, guaranteeing, that the encryption key is never exposed to other applications or other kernel mode code.
@@ -408,11 +372,11 @@
 \subsubsection{Mitigating Memory Corruption Vulnerabilities in SGX Enclaves}
 \label{mitigate_memory_subsection}
-Although there are legitimate cases to receive and return pointers without safety checks, this is highly discouraged. If incorrectly used, it could very easily allow memory corruption within the enclave from untrusted code. Leading to arbitrary read and writes in the enclave and maybe even code execution in certain cases~\cite{cloosters2020teerex}.
+Although there are legitimate cases to receive and return pointers without safety checks, this is highly discouraged. If incorrectly used, it could very easily allow memory corruption within the enclave from untrusted code, leading to arbitrary reads, writes and maybe even code execution in certain cases~\cite{cloosters2020teerex}.
 Any function within the enclave that accepts a pointer without safety checks and is exposed to untrusted code could be a potential read/write primitive for an attacker.
-Appendix \ref{read_primitive} shows a very simple function in WolfSSL~\cite{wolfssl}. It returns the variable \textit{rfd} in the pointer to a structure of type \textit{WOLFSSL}. At a lower level, this means that the processor will add the offset of \textit{rfd} in the structure \textit{WOLFSSL} to the pointer provided in \textit{ssl}, read the contents of the resulting address, and return it. While it may seem harmless to expose this function to untrusted code, this gives an attacker a read primitive to an Intel SGX enclave. This is because nothing guarantees that the pointers passed to the function are of the type \textit{WOLFSSL}. This function will return an integer at the offset of variable the \textit{rfd} in the \textit{WOLFSSL} structure pointed by the parameter \textit{ssl}. Since the parameter \textit{ssl} can point to anywhere in memory, an attacker can read any memory inside the enclave. Similarly, the function in appendix \ref{write_primitive} shows a write primitive to an Intel SGX enclave if exposed to untrusted code.
+\ref{read_primitive} shows a very simple function in WolfSSL~\cite{wolfssl}. It returns the variable \textit{rfd} in the pointer to a structure of type \textit{WOLFSSL}. At a lower level, this means that the processor will add the offset of \textit{rfd} in the structure \textit{WOLFSSL} to the pointer provided in \textit{ssl}, read the contents of the resulting address, and return it. While it may seem harmless to expose this function to untrusted code, this gives an attacker a read primitive to an Intel SGX enclave. This is because nothing guarantees that the pointers passed to the function are of the type \textit{WOLFSSL}. This function will return an integer at the offset of variable the \textit{rfd} in the \textit{WOLFSSL} structure pointed by the parameter \textit{ssl}. Since the parameter \textit{ssl} can point to anywhere in memory, an attacker can read any memory inside the enclave. Similarly, the function in \ref{write_primitive} shows a write primitive to an Intel SGX enclave if exposed to untrusted code.
 To mitigate this attack, the enclave must perform safety checks on the passing points. In our solution, we chose to create a hash map for each type of structure used and exported to untrusted code. This allowed for constant-time access to the saved pointers regardless of the number of existing pointers.
@@ -453,7 +417,7 @@
 If the cryptography library is inside an Intel SGX enclave, the application may still register the callback to functions outside the enclave, but when the enclave tries to execute that function, a segmentation fault will occur. This is because enclaves are not allowed to execute instructions outside its memory range without first issuing a special \textit{ocall} instruction to leave the enclave.
-To overcome the issue mentioned above, we resort once again to hash maps, which will hold a pointer to an array containing all possible callbacks in the \textit{BIO\_METHOD} and a static callback for each callback which will resolve the correct pointer and forward it to untrusted code to execute. Appendix~\ref{callback_handling} shows a simplified concept of this in practice, when \textit{BIO\_meth\_set\_read} is called instead of forwarding the call to the enclave, it is saved in its corresponding slot in the created array, the registered callback to WolfSSL will instead be a function within the enclave. When called, the callback will obtain the real pointer saved in the array and tell the untrusted code to execute it. The functions \textit{GetBioCallbackArray} and \textit{CreateBioCallbackArray} simply get and create the necessary array in the hash map.
+To overcome the issue mentioned above, we resort once again to hash maps, which will hold a pointer to an array containing all possible callbacks in the \textit{BIO\_METHOD} and a static callback for each callback which will resolve the correct pointer and forward it to untrusted code to execute. \ref{callback_handling} shows a simplified concept of this in practice, when \textit{BIO\_meth\_set\_read} is called instead of forwarding the call to the enclave, it is saved in its corresponding slot in the created array, the registered callback to WolfSSL will instead be a function within the enclave. When called, the callback will obtain the real pointer saved in the array and tell the untrusted code to execute it. The functions \textit{GetBioCallbackArray} and \textit{CreateBioCallbackArray} simply get and create the necessary array in the hash map.
 \subsubsection*{Additional Getters/Setters}
 Even though OpenSSL 1.1 removed many of its structures from the public header files so that they become opaque and force the usage of accessor functions~\cite{openssl11changes}, not everything got the same treatment. Structures such as the \textit{GENERAL\_NAME} are not opaque and still rely on the application to directly access the data in the pointer returned by OpenSSL.
@@ -462,7 +426,7 @@
 Secondly, Intel SGX enclaves when running in release mode do not allow access from untrusted code. This means, similarly to the first issue, trying to access the data it points to will cause an access violation. Because of the latter issue, we believe that TaLoS~\cite{aublin2017talos}, even though it claims support for Apache's HTTP server, it does not handle these cases within the LibreSSL library. This means it will only work in pre-production mode, allowing access to the enclave's memory from any code. Attempting to use the library in production, i.e., enclave compiled in release mode, would cause segmentation faults when accessing data that resides in the enclave.
 \subsubsection*{Sealed Key Detection}
-As we wanted to retain as many features as of the original module, we decided to load normal keys and sealed keys by the enclave. We modified the module first to load a sealed key, and if it failed, try to load the key normally.
+As we wanted to retain as many features as possible from the original module, we decided to load normal keys and sealed keys by the enclave. We modified the module first to load a sealed key, and if it failed, try to load the key normally.
 For convenience and help users identify what kind of key they have in storage, we append the suffix '.sealed' to the end of the file's name.
 \subsubsection{Limitations}
@@ -493,12 +457,11 @@
 \subsection{OpenSSL Engine Integration} \label{custom_engine}
 %This Section will describe our implementation of a key store that uses Intel SGX to keep its contents secrets and its integration with an OpenSSL engine.
-As we will see in Section \ref{chap_results}, the previous solution comes with some big caveats and a high-performance penalty. Thus, we started looking for other ways to protect the private key used by untrusted software. This Section describes our implementation of a key store that uses Intel SGX to keep its contents secrets and its integration with an OpenSSL engine. This implementation differs from the previous solution. It allows for any application that uses OpenSSL as its cryptography library to leverage the keys protected by an enclave, guaranteeing its integrity and confidentiality, working very similarly to a hardware security module a pkcs\#11 device. Unlike the previous solution, it does not protect anything besides the private key utilized, meaning the established symmetric keys by applications in their TLS connections are vulnerable to the attacker.
+As we will see in Section \ref{chap_results}, the previous solution comes with some big caveats and a high-performance penalty. Thus, we started looking for other ways to protect the private key used by untrusted software. This Section describes our implementation of a key store that uses Intel SGX to keep its contents secrets and its integration with an OpenSSL engine. This solution it allows for any application that uses OpenSSL as its cryptography library to use the keys protected by an enclave, guaranteeing its integrity and confidentiality, this approach works very similarly to a hardware security module, a pkcs\#11 device, an Android's phone keystore system\cite{android_keystore_system} or web-services like HashiCorps' vault\cite{hashicorp}. Unlike the previous implementation it does not protect anything besides the private key utilized, meaning that if an application utilizes the private key to establish established symmetric keys in their TLS connections these would be accessible to an attacker with read access to the address-space of the application holding these keys.
 The solution described in this Subsection is open source and available in a public repository on GitHub~\cite{sgx_keystore_openssl_engine}.
-In this implementation the enclave works in a very similar manner to a pkcs\#11 device. It exposes a limited number of functions so that untrusted code can load private keys and perform operations with it (i.e. Encryption and Decryption). Table \ref{table:enclave_functions} shows the functions exported by the enclave and their functionality. As the \textit{RSA} object in OpenSSL needs to contain at least the modulus and the public key exponent, a function exists to export values out of the enclave.
+As in this implementation the enclave works in a very similar manner to a pkcs\#11 device. It exposes a limited number of functions limiting the attack surface. The exposed functions only need to allow untrusted code to load private keys and perform operations with it (i.e. Encryption and Decryption). Table \ref{table:enclave_functions} shows the functions exported by the enclave and their functionality. As the \textit{RSA} object in OpenSSL needs to contain at least the modulus and the public key exponent, a function exists to export values out of the enclave.
 \renewcommand{\arraystretch}{1.2}
 \begin{table*}[t]
@@ -529,7 +492,7 @@
 \renewcommand{\arraystretch}{1}
-To make it possible for applications that use OpenSSL to use our solution easily, we implemented an OpenSSL Engine, which at a very high level essentially instructs OpenSSL on how to load and utilize keys from a custom solution. Our engine implementation was based on the Android Open Source Project~\cite{aosp_system_security} due to its simplicity. As the source code targets a version of OpenSSL before 1.1, with the OpenSSL Engine's help for pkcs\#11 devices, libp11~\cite{github_libp11}, we updated its source for a more recent version of OpenSSL.
+To make it possible for applications that use OpenSSL to use our solution easily, we implemented an OpenSSL Engine, which at a very high level essentially instructs OpenSSL on how to load and utilize keys from a custom solution. Our engine implementation was based on the Android Open Source Project~\cite{aosp_system_security} due to its simplicity. As the source code targets a version of OpenSSL before 1.1, with the OpenSSL Engine's help for pkcs\#11 devices, libp11~\cite{github_libp11}, we updated its source to OpenSSL 1.1.1g.
 The Engine registers a callback on OpenSSL for the event that loads a key. When this callback is called, it attempts to load the key. If the key is successfully loaded from the key store, custom methods are set for the encryption and decryption of data when using the key. These methods, when called, will forward the request to the enclave.
@@ -549,29 +512,29 @@
 \subsubsection{Cryptography Library}
-Unlike the implementation in Section \ref{http_implementation} the functionality required from the cryptography library is much smaller. The only requirements being the ability to load an RSA private key and perform operation with it. Fortunately, even though the functionality of Intel® SGX SSL~\cite{intel-sgx-ssl} is rather limited, it provides support for the required features. We chose this library over WolfSSL~\cite{wolfssl} as it is a fork from OpenSSL. It is giving us a simple one to one match of the functions utilized by other OpenSSL Engines.
+Unlike the implementation in Section \ref{http_implementation} the functionality required from the cryptography library is much smaller. The only requirements needed by the library is the ability to load an RSA private key and perform operation with it. Fortunately, even though the functionality of Intel® SGX SSL~\cite{intel-sgx-ssl} is rather limited, it provides support for the required features. We chose this library over WolfSSL~\cite{wolfssl} as it is a fork from OpenSSL. It is giving us a simple one to one match of the functions utilized by other OpenSSL Engines.
 \subsubsection{Configuring OpenSSL}
-To make OpenSSL aware of the Engine and make it so that other applications can use it, a few modifications need to be made to install OpenSSL. On an installation of OpenSSL through Ubuntu's package manager, appendix \ref{openssl_engine_config} needs to be added to the configuration file \textit{/etc/ssl/openssl.cnf}. In addition to configuring OpenSSL, the Engine needs to be copied to the configuration file's path. This should make it possible for any application to use the Engine, even from command like as shown in appendix \ref{openssl_engine_cmd}.
+To make OpenSSL aware of the Engine and make it so that other applications can use it, a few modifications need to be made to install OpenSSL. On an installation of OpenSSL through Ubuntu's package manager, \ref{openssl_engine_config} needs to be added to the configuration file \textit{/etc/ssl/openssl.cnf}. In addition to configuring OpenSSL, the Engine needs to be copied to the configuration file's path. This should make it possible for any application to use the Engine, even from command like as shown in \ref{openssl_engine_cmd}.
 \subsubsection{Required Modifications to Applications}
-Applications that utilize OpenSSL may easily be adapted to use any engine available. By calling the OpenSSL function \textit{ENGINE\_by\_id} with the id of the Engine, returning a reference to the Engine is obtained. Subsequently, the application must call \textit{ENGINE\_init} to initialize it. To load a private key from the Engine, OpenSSL provides a function \textit{ENGINE\_load\_private\_key}, which attempts to load a key from the Engine and returns a reference to a \textit{EVP\_PKEY} object which can be used as if it were a key loaded through the conventional OpenSSL API.
+Applications that utilize OpenSSL may easily be adapted to use any engine available. By calling the OpenSSL function \textit{ENGINE\_by\_id} with the id of the Engine, a reference to the Engine is obtained. Subsequently, the application must call \textit{ENGINE\_init} to initialize it. To load a private key from the Engine, OpenSSL provides a function \textit{ENGINE\_load\_private\_key}, which attempts to load a key from the Engine and returns a reference to a \textit{EVP\_PKEY} object which can be used as if it were a key loaded through the conventional OpenSSL API.
 \subsubsection*{Adding support to Apache's HTTP server}
-When Apache's HTTP server attempts to load a key or a certificate, it checks if the provided Uniform Resource Identifier (URI) contains a colon and if its prefix is a supported OpenSSL Engine. If it is a supported engine, the server attempts to load and initialize the Engine with the same id as the prefix in the URI, in our case, sgxkeystore. Eventually, requesting it to load the key in the specified URI. For example, the URI \textit{sgxkeystore:key.pem.sealed} would result in the initialization of the engine \textit{sgxkeystore}, loading the specified URI.
-As the original code already supports pkcs\#11 devices through libp11's OpenSSL engine~\cite{github_libp11}, adding support for another engine is a simple process. The only changes required are to detect the prefix in the URI, as shown by the patch in appendix~\ref{httpd_engine_support_patch}.
+When Apache's HTTP server attempts to load a key or a certificate, it checks if the provided Uniform Resource Identifier (URI) contains a colon and if its prefix is a supported OpenSSL Engine. If it is a supported engine, the server attempts to load and initialize the Engine with the same id as the prefix in the URI, in our case, sgxkeystore. For example, the URI \textit{sgxkeystore:key.pem.sealed} would result in the initialization of the engine \textit{sgxkeystore}, loading the key "key.pem.sealed".
+As the original code already supports pkcs\#11 devices through libp11's OpenSSL engine~\cite{github_libp11}, adding support for another engine is a simple process. The only changes required are to detect the prefix in the URI, as shown by the patch in ~\ref{httpd_engine_support_patch}.
 \subsubsection{Security Analysis}
-This Subsectionpresents a security analysis of creating a key store with SGX and making it usable to external applications.
+This Subsection presents a security analysis of creating a key store with SGX and making it usable to external applications.
 \subsubsection*{Access to Encryption Keys}
 Unlike the solution presented in Section~\ref{http_implementation} this implementation does not protect the symmetric keys during a TLS connection. It works very similarly to a hardware security module (HSM) or, more precisely softHSM~\cite{softhsm}, which emulates an HSM in software. As our solution only implemented support for RSA keys, it only guarantees the secrecy of these keys. Meaning if the key store is utilized for TLS, the agreed symmetric key would be exposed to untrusted code.
 \subsubsection*{Key Usage}
-Like the first solution presented in \ref{http_implementation}, the current implementation allows for any code being executed to make requests to the sgx key store to sign and decrypt data with the private key. The current solution utilizes UNIX domain sockets to communicate with other applications, and this means that it is easily adaptable to network sockets so that future iterations could leverage a solution as presented in~\cite{keysintheclouds}. Allowing only authenticated access to the keys and logging key usage.
+Like the first solution presented in \ref{http_implementation}, the current implementation allows for any code being executed to make requests to the sgx key store to sign and decrypt data with the private key. The current solution utilizes UNIX domain sockets to communicate with other applications, this means that it is easily adaptable to network sockets so that future iterations could leverage a solution as presented in~\cite{keysintheclouds}, allowing only authenticated access to the keys and logging key usage.
 \section{Results}\label{chap_results}
@@ -699,23 +662,39 @@
+\section{Threat Model} \label{security_analysis}
+In this section, we provide a threat analysis of the proposed system. We identify potential threats of each implementation and define how an attacker may attempt to exploit the system and the limitations we introduce to block that thread.\\
+\noindent\textbf{Key Negotiation}\\
+The impersonation attack on the HTTPS protocol is normally due to the unauthorized access to the private key on the server~\cite{callegati2009man} which can affect either current and previous communications. In order to mitigate this, we implemented OpenSSL inside SGX, giving an extra layer of security to the private and symmetric key management.\\
+\noindent\textbf{Physical Access to the Machine}\\
+If an attacker is able to compromise both machines, it will have a similar security vulnerability as in a reverse proxy. However, contrary to the reverse proxy approach, the attacker will not know the symmetric and private keys. In the option with the symmetric key exposed, the attacker will be capable of accessing the information of a specific connection, but it will be limited to the session that passes through the appliance. This is a normal assumption if we assume that reverse proxy tends to use no encryption in communication with internal machines.
+In this new scenario, the attacker will have clear limitations to access the private key, meaning that even if the system is compromised, the private key's security is never compromised.\\
+\noindent\textbf{DoS scenario}\\
+Denial-of-service attack limits the machine or network resource unavailable to its real users by temporarily disrupting services of a network component connected to the Internet. In this scenario, the option with the symmetric key exposed focuses on mitigating the delay associated with manipulating the private key. As we can see in Section \ref{chap_results}, the option with the symmetric key exposed has a very close performance comparison, meaning that your solutions will not be the bottleneck in case of a DoS attack to the apache server.
 \section{Conclusion and Future Work} \label{chap_conclusion}
 In this Section, we overview our work and results accomplished throughout this work. We start by making a brief overview of the research and development made throughout our work, commenting on the goals achieved and complexity of each project. We then talk about the results obtained, their effectiveness and viability. Lastly, we hint on possible future directions of this work, as an ongoing effort to minimize the impact on performance by secure enclaves.
-%so that the utilization of Intel SGX secure enclaves impacts significantly less the applications used during the development of this work.
 \subsection{Research and Development}
 Despite our efforts, we struggled to find applications that utilized Intel SGX. Instead, we found many solutions that either automatically porting applications~\cite{lind2017glamdring} or ran unmodified applications inside Intel SGX~\cite{haven,scone,sgxlkl,graphenesgx}. As seen by the results obtained in Chapter \ref{chap_results}, some of these solutions introduce limitations that only allow certain types of applications to run inside SGX.
-%Given this state-of-affairs, we decided to port some applications to utilize Intel SGX through the SGX SDK and compare how these would perform relatively to their original counterparts running without Intel SGX and running with the previously mentioned solutions.
-%%TODO: verifica
 Given this state-of-affairs, we decided to port some applications to utilize Intel SGX through the SGX SDK and compare how these would perform relatively to their original counterparts running without any form of support from Intel SGX.
 The Apache's web server can secure connections via TLS through a separate module called \textit{mod\_ssl}. Porting this module was harder than expected, taking a considerable portion of the time spent on this work. We encountered several issues, ranging from finding a compatible library with it and Intel SGX, to modifying the cryptography library to use it from untrusted code and through its OpenSSL compatibility layer.
-%%TODO: verfica
 Developing the Keystore for Intel SGX took significantly less effort than porting \textit{mod\_ssl}, with the harder task being modification of the OpenSSL Engine with proper documentation available.
 Additionally, in Section~\ref{http_implementation}, we also identified some issues with Intel's SGX cryptography library, TaLoS, and explained why it would not be suitable for our purposes.
@@ -724,26 +703,25 @@
 We successfully integrated the \textit{mod\_ssl} with WolfSSL and Intel SGX so that the termination of the TLS connection is made within a secure enclave to protect both the private key and the generated symmetric key during the handshake. However, this protection comes with a great performance penalty compared to both the original code of the module and Graphene-SGX, decreasing the performance as much as 90\% and 70\%, respectively.
-%%TODO: checka o EN para veres a dif.
 Our second approach was aimed to guarantee secrecy of the private keys and involved implementing a Keystore in Intel SGX and integrating with an OpenSSL engine so that applications could use it transparently. Applying this solution to the Apache webserver resulted in no measurable overhead when compared to the unmodified version while performing significantly better than Graphene-SGX. This solution's caveat is that it does not protect the symmetric key agreed upon during the TLS handshake. Table \ref{table:rsa_speed} shows that there is, in fact, some overhead in our solution, but it was not significant enough to affect the web server.
 We were limited to a single Intel SGX capable machine to test our solutions and the others. Additionally, we could not compile a Linux kernel with KVM supporting Intel SGX for guest virtual machines to analyze how Intel SGX would perform in a cloud-like scenario, limiting us to only being able to utilize docker.
-%%TODO: isto está documentado em algum sitio. Se nao, nos reportamos isto?
-%%TODO: acho que nao deviamos colocar isto nestes termos. A segunda aproximação é mais rapida mas não protege a negociação das chaves simetricas, enquanto que a primeira protege mas é mais lenta.Não devemos dar recomendações.
 Given our second approach versatility and performance, it is harder to recommend the first implementation, which has an inferior performance and may make it harder to port future updates made to the original \textit{mod\_ssl}. While the first solution does protect the TLS connection inside the secure enclave, an attacker still has access to the web server's code and memory, meaning the encrypted and decrypted data can be read on the read and write callbacks of the web server, which means the protection is not that much greater when compared to the implemented OpenSSL Engine.
 \subsection{Future Work}
 Although we achieved our goals and utilized Intel SGX to increase key secrecy in real-world scenarios, our solution is not without limitations, pointed throughout this paper. In the future, we would like to address these limitations.
 \subsubsection*{Asynchronous Calls}
-All our solutions, for simplicity, implemented synchronous function calls to and from the enclave. We acknowledge that this is not ideal and brings a significant overhead to certain work scenarios. TaLoS~\cite{aublin2017talos} has shown that the implementation of asynchronous has improved performance on their workload by as much as 117\%. In Section \ref{ioapp_section}, we can see that other solutions like Scone~\cite{scone} and~\cite{graphenesgx}, which implemented asynchronous function calls perform significantly better than our solution when a high amount of enclave transitions are made. To mitigate some of our solutions' performance penalty by implementing asynchronous function calls.
+All our solutions, for simplicity, implemented synchronous function calls to and from the enclave. We acknowledge that this is not ideal and brings a significant overhead to certain work scenarios. TaLoS~\cite{aublin2017talos} has shown that the implementation of asynchronous has improved performance on their workload by as much as 117\%. In Section \ref{ioapp_section}, we can see that other solutions like Scone~\cite{scone} and~\cite{graphenesgx}, which implemented asynchronous function calls perform significantly better than our solution when a high amount of enclave transitions are made. To mitigate some of our solutions' performance penalty future work could implement asynchronous function calls.
 \subsubsection*{Public-key Cryptography Support}
 Due to time constraints, the solution presented in Section \ref{custom_engine} only implemented support for RSA public-key cryptography. Consequently, applications utilizing our engine are limited to RSA public-key cryptography. Future work of this solution should be able to support other public-key cryptography algorithms such as Elliptic-curve cryptography.
 \subsection*{Acknowledgment}
-This work of Jo\~ao S. Resende was supported by Funda\c{c}\~ao para a Ci\^encia e Tecnologia (FCT), Portugal (PD/BD/128149/2016). This work has been supported by the EU H2020-SU-ICT-03-2018 Project No. 830929 CyberSec4Europe (cybersec4europe.eu).
+This work of Jo\~ao S. Resende was supported by Funda\c{c}\~ao para a Ci\^encia e Tecnologia (FCT), Portugal (PD/BD/128149/2016). This work has been supported by the EU H2020-SU-ICT-03-2018 Project No. 830929 CyberSec4Europe (cybersec4europe.eu) and project POCI-01-0247-FEDER-039598 (COP - Container Policing Hardening). Rolando Martins was partially funded by project POCI-01-0247-FEDER-041435 (SafeCities), financed by Fundo Europeu de Desenvolvimento Regional (FEDER), through COMPETE 2020 and Portugal 2020.
 \appendix
 \section{Read and Write Primitives within Intel SGX} % Main appendix title
@@ -828,7 +806,10 @@
  &BIO_meth_set_read_callback_handler);
 }
 \end{verbatim}
-The code snippet above shows how we handle the callbacks from trusted to untrusted code. As Intel SGX can not jump between trust and untrusted without first issuing a special instruction, steps must be taken to accommodate for this requirement. When the application wishes to register a callback the function \textit{sgx\_BIO\_meth\_set\_read} is called, this function will store the callback in a callback array for the specified \textit{BIO\_METHOD} object and instead register a special stub that resided in trusted code with the cryptography library. When the stub function is called we resolve the actual function to be called within untrusted code through the aforementioned array, and forward the function pointer to a proxy function in untrusted code which will ultimately call the registered function.
+The code snippet above shows how we handle the callbacks from trusted to untrusted code. As Intel SGX can not jump between trust and untrusted without first issuing a special instruction, steps must be taken to accommodate for this requirement. The SGX SDK partially abstracts this by creating proxy functions which replaces the return value that indicates if the call was successful\cite{sgxlinuxsdk} and moving the real return value to the first parameter as a pointer. When the application wishes to register a callback the function \textit{sgx\_BIO\_meth\_set\_read} is called, this function will store the callback in a callback array for the specified \textit{BIO\_METHOD} object and instead register a special stub that resided in trusted code with the cryptography library. When the stub function is called, we resolve the actual function to be called within untrusted code through the aforementioned array and forward the function pointer to a proxy function in untrusted code, ultimately calling the registered function.
 \subsection{OpenSSL Engine Configuration}
 \label{openssl_engine_config}