``eks`` - Amazon EKS ==================== The ``eks`` runtime integrates with EKS, AWS's managed Kubernetes offering. .. contents:: Settings -------- .. autopydantic_model:: hydroplane.runtimes.eks.Settings Example Configuration --------------------- Here's an example of a configuration file that uses the ``eks`` runtime and the ``local`` secret store. This configuration sets up Hydroplane to talk to an EKS cluster named ``my-cool-cluster`` in the ``us-west-2`` AWS region: .. code-block:: yaml --- secret_store: secret_store_type: local store_location: ~/.hydro_secrets runtime: runtime_type: eks cluster_name: my-cool-cluster region: us-west-2 credentials: access_key: access_key_id: secret_name: aws-credentials key: AccessKeyId secret_access_key: secret_name: aws-credentials key: SecretAccessKey Quickstart ---------- Step 1: Create an EKS cluster and IAM user ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In order to use the ``eks`` runtime, you first have to have an EKS cluster. We'll use `hydro-project/eks-setup `_ to do that. Follow the instructions in the "Quickstart" section of that repo's README and then come back here. When you're done with those instructions, you should have two things: * An EKS cluster (we'll assume that the cluster's name is ``hydro-eks`` and that it was created in ``us-west-2`` in subsequent steps for brevity) * The access key and secret key for a user in the cluster's ``cluster-admins`` group (if you added a user using ``eks-setup user add``, that user is in the ``cluster-admins`` group) Step 2: Configure Hydroplane ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Create a file named ``eks.yml`` with the following contents: .. code-block:: yaml --- secret_store: secret_store_type: local store_location: ~/.hydro_secrets runtime: runtime_type: eks cluster_name: hydro-eks region: us-west-2 credentials: access_key: access_key_id: secret_name: aws-credentials key: AccessKeyId secret_access_key: secret_name: aws-credentials key: SecretAccessKey If you haven't already, initialize your local secret store: .. code-block:: bash bin/local-secret-store init Make a note of the password you used when configuring the store; you'll need it when Hydroplane starts. Now, create a file called ``creds.json`` that looks like this: .. code-block:: json {"AccessKeyId":"","SecretAccessKey":""} Add that file to the secret store: .. code-block:: bash bin/local-secret-store add -f creds.json aws-credentials and delete the plaintext original afterwards: .. code-block:: bash rm creds.json Step 3: Run Hydroplane ^^^^^^^^^^^^^^^^^^^^^^ Now you've (finally) got an EKS cluster and the ability to authenticate to it! Let's launch Hydroplane: .. code-block:: bash bin/hydroplane -c eks.yml You'll be prompted for the password to your local secret store. Once you enter it, the server should bind and start accepting requests. In a separate terminal, from the root of the ``hydroplane`` repo, let's make sure we can list processes: .. code-block:: bash bin/hpctl list You should get back an empty list, because we haven't launched any processes yet. Let's try a simple example to make sure we can start something and access it remotely: .. code-block:: bash bin/hpctl start examples/nginx.json If you list processes with ``bin/hpctl list`` now, you should see something like this: .. code-block:: json [ { "process_name": "nginx", "group": null, "socket_addresses": [ { "host": "34.217.208.197", "port": 31491, "is_public": true } ], "created": "2022-12-01T22:40:33+00:00", "status": "RUNNING" } ] Note that the process is exposing a single public port. In this example, it's ``34.217.208.197:31491``, but your host and port will be different. If you have ``jq`` installed, are on macOS, and want to open that server with a one-liner: .. code-block:: bash open http://$(bin/hpctl list | jq -r "(.[0].socket_addresses[0].host + \":\" + (.[0].socket_addresses[0].port|tostring))") Otherwise, you can copy-paste the host and port into a browser window. Either way, you should see an `nginx `_ "hello world" page. Now that we've verified that everything is working, let's stop the process: .. code-block:: bash bin/hpctl stop nginx Implementation and Configuration Details ---------------------------------------- For details on how the ``eks`` runtime creates processes, see :doc:`/architecture/k8s`. Authenticating with AWS ^^^^^^^^^^^^^^^^^^^^^^^ In order for Hydroplane to launch processes on an EKS cluster, it must be able to authenticate with that cluster. To authenticate, you'll need an AWS `access key ID and corresponding secret access key `_. These credentials will need to be stored in Hydroplane's :doc:`secret store `. .. _eks-secret-example: Authenticating with IAM Credentials ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Here's an example configuration that uses a secret stored in Hydroplane's secret store to authenticate with EKS. This config uses a secret named ``aws-creds`` that's stored in the secret store as a JSON object like so: .. code-block:: json {"AccessKeyId":"XXXXXXXXX","SecretAccessKey":"XXXXXXX"} These related credentials are referenced individually using the ``key`` field. .. code-block:: yaml runtime: runtime_type: eks cluster_name: my-cluster region: us-west-2 credentials: access_key: access_key_id: secret_name: aws-creds key: AccessKeyId secret_access_key: secret_name: aws-creds key: SecretAccessKey Authenticating with Role Assumption ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ While nothing stops you from authenticating directly with an access key and secret, it's generally considered a best practice to authenticate using `temporary IAM credentials `_. These credentials are associated with a given IAM role, and expire after a certain period of time. Hydroplane will attempt to automatically renew these temporary credentials periodically. Here's an example of using the ``aws-creds`` secret from above to authenticate and then assume the role ``EKSAdmin``. Most of this configuration will look familiar if you've looked at the configuration in the last section, but we've added an ``assume_role`` block to define the role assumption. .. code-block:: yaml runtime: runtime_type: eks cluster_name: my-cluster region: us-west-2 credentials: access_key: access_key_id: secret_name: aws-creds key: AccessKeyId secret_access_key: secret_name: aws-creds key: SecretAccessKey assume_role: role_arn: "arn:aws:iam::123456789012:role/EKSAdmin" A Note on IAM User Permissions ------------------------------ The authenticating IAM user must have enough permissions on the EKS cluster to create and destroy pods and services. If you've set up your cluster with `hydro-project/eks-setup `_, then any user you add using that script will have the appropriate privileges. AWS Authentication Configuration Reference ------------------------------------------ .. automodule:: hydroplane.models.aws :members: