OS Updates and Package Management: Ubuntu Repo Management With Aptly and AWS S3

Note: In light of the AWS S3 outage in us-east-1 on February 28, 2017, let’s discuss a few things. Amazon’s S3 has exemplary availability. Compare that with the time and cost of maintaining package distribution yourself. It’s easy to look at S3’s outage and conclude that it is better to handle the responsibility yourself. In the same way, it’s easy to see news of a plane crash and conclude that driving is more reliable. The feeling of control doesn’t always lead to the most reliable outcome. Aptly does provide the ability to serve a repository on its own. See how to front Aptly with nginx in an emergency like the one on Tuesday February 28.

It is an unfortunate fact that many organizations do not routinely perform comprehensive software patching. At Threat Stack, we have confirmed this with our own analysis of how frequently systems are updated, and Verizon’s DIBR shows us that the most commonly exploited vulnerabilities are months or years old.

But patching is one area where following the status quo is a very bad idea. As a best practice, your organization needs a patching strategy to make sure it remains secure, and with that in mind, this post explains how you can adopt a patching strategy that suits your organization’s needs and values.

The Need for a Patch Management Strategy

First, however, let’s take a look at some of the reasons why organization’s don’t patch their systems. Most of these have to do with time: Patching can be time-consuming work that competes with other priorities. Time needs to be spent testing updates to ensure that they won’t negatively affect running systems. If your method of updating is terminate and replace, DBs and other systems with persistent data need to be handled properly to prevent data loss. Even when the process is fully automated, it might take time for data replication to finish. Finally, kernel updates in most organizations require reboots, which means you need to assess the impact of temporary outages before performing an update.

OK, but in the face of this, you are still vulnerable to threats, so it’s critical that you fit a patch management strategy into your workflow. While it will cost you some outlay in terms of time, you will be rewarded with improved security. The challenge for you is to choose a strategy that is right for your organization.

Broadly speaking, patch management strategies fall into two camps (three if you include not patching):

  • Update immediately as released
  • Update on a regular cadence

Both approaches have their pros and cons. Neither is wrong and neither is right. Each represents a judgement about the importance of Stability vs. the importance of Security.

Whether you want more control over your patching schedule (rather than having hosts take unattended updates off hours), or whether you want to patch at a faster cadence, a common reason for not being able to do either effectively is the time it takes to run and manage your own package repository. On RedHat-based systems there is Spacewalk for managing repositories and deploying updates. If you use Foreman for managing systems, the Katello plugin handles repository management and package update deployments. Artifactory is a SaaS service if you prefer to not host your own mirror. In this post we are going to focus on managing Debian and Ubuntu APT repositories using Aptly.

Package Management Camps

Let’s take a deep look at the two package management camps along with their pros and cons. If you don’t have a patch management strategy at present, then it’s important to identify one that fits your needs.

We are also going to discuss package management in terms of environments where existing hosts are updated and remain running afterwards. If you are truly doing immutable infrastructure, you will probably have to read this and think of it in terms of your base image build (container or instance) and release pipeline.

Immediate Patching

Immediate patching means that systems are updated within a relatively short time after the vendor has released an update. This is commonly done by having a cron job that executes a package manager update command. Or, you might want slightly more control and perform updates after some testing is done by running an orchestration tool like Ansible or SaltStack.

The main characteristics of immediate patching are as follows:

  • Updates are dictated by the vendor’s schedule.
  • Updates are made relatively quickly after they are released.
  • The speed of updating and the resulting security are most important to your organization.

With immediate patching, updates are handled rapidly, and if they are automated, they are performed reliably so updates are not missed. This assures that you are protected from the latest software vulnerabilities that have been introduced.

You can’t plan for the updates and associated work, however, and this can be disruptive to your workflow. If the update process is automated via cron, the amount of work is reduced compared to if you are using an orchestration tool to perform updates. But now you are introducing the possibility that unplanned patching will occur off hours (for example, overnight or on weekends), and bugs do happen when a vendor releases fixes. You end up with a tradeoff between unplanned failure and unplanned work. Finally, some updates require intervention no matter how you deploy. For example, kernel updates require reboots. And services may need controlled restarts that require you to assess how the outage is to be handled.

Pros:

  • Your systems are brought up to date rapidly and are defended against the latest threats.
  • Updates are not likely to be overlooked.

Cons:

  • The update process may be time intensive.
  • There is an increased risk of failure.
  • Updates cannot be planned and therefore can be disruptive to your workflow.

Cadence Patching

Cadence patching means that you update systems at regular intervals. How regular that cadence is can vary widely. Your cadence could be monthly or even quarterly. Or you could have a weekly “patch Thursday” when all patches since the previous week are implemented. (With that frequency, your approach almost resembles immediate patching. However, you could hold off implementing a patch that a vendor released if it’s too close to your patch date and you want more time to test it.)

The main characteristics of cadence patching are as follows:

  • Updates are dictated by your schedule, not the vendor’s.
  • You make an update a period of time after the vendor releases it so you can confidently assess its stability.
  • Your organization values the stability of running systems over the speed of updates.

Updates made on a regular cadence are handled in a manner that the organization deems to be timely with an emphasis on keeping existing systems stable. The cadence period will vary across organizations depending on the threats an organization deems most important. For example, local vulnerabilities might be deprioritized over remote vulnerabilities. The organization may feel that they have enough access controls to limit who has local access and enough monitoring in place to keep an eye on those who do. Vulnerabilities to internal systems might be deprioritized over vulnerabilities to edge-facing systems because internal systems have limited access paths while external systems can be reached by anyone on the internet.

Typically, longer cadences are used because of the amount of work involved, the availability of resources like time and manpower, and your assessments of vulnerability. When software vulnerability updates are released, they should be analyzed, and a risk assessment should be performed.

Note: Cadence patching should include a procedure for performing out-of-cycle updates when a vulnerability is determined to be too critical to wait. For example, Heartbleed should have forced out-of-cycle updating OpenSSL on edge web servers. Otherwise, updating systems is a routine task that can be planned. This ensures that other work that needs to be done is not interrupted.

Pros:

  • Cadence patching is less time intensive than immediate patching.
  • Work is predictable and can be scheduled.
  • You have time to test and reduce failure.

Cons:

  • Systems are left unprotected for longer periods of time.

Which is Right For Your Organization?

No one can tell you which approach your organization should adopt. You need to make a judgement based on your risk assessments, risk tolerance, engineering values, and available resources. For example, if you value stability, take into account how often have you seen a bad patch come out that has massively broken something. What is the likelihood of that happening?

At the same time, some organizations are routinely attacked because they are large and important targets. They need to be protected from the latest vulnerability as quickly as possible. But many organizations are more likely to be hit with attempts to exploit older vulnerabilities because older vulnerabilities end up yielding better payoffs with less work. Your attackers probably calculate ROI just as you are doing right now as you try to determine which approach you should choose.

APT Repository Mirrors With Aptly

Whether you choose immediate patching or cadence patching, managing your own repository can make your life easier. If you prefer immediate patching, it gives you finer control over when updates occur. For example, it lets you update your mirror at midnight, update a batch of test systems afterwards from that mirror, and finally let the rest of your systems take updates after you’ve gotten in and had your coffee, or roll back the mirror update while you assess failures. If you prefer cadence patching, this may reduce some of the work involved in managing a package repository mirror and give you incentive to shorten your cadence.

Aptly is a tool for managing APT repositories. Unlike using a tool like rsync to mirror a package repository, Aptly separates the steps of mirroring and publishing, and it allows the creation of point-in-time snapshots. This allows you to download updates without disturbing the currently published repository. The newly updated repository can then be snapshot and published to your mirror when appropriate. You can even publish the same snapshot to multiple package repositories, allowing you to publish to one repository for test systems and publish the same package contents to a second repository for wider package distribution. Combined with AWS S3 for hosting your package repository, you don’t even need to deploy a new instance to host the service.

Setting Up and Managing a Mirror With Aptly

Use the following procedure to manage a mirror with Aptly:

  1. Create an S3 bucket where your package repository will live. Note: Due to a bug in apt-s3-transport, which we will use for package publishing and retrieval, valid bucket names with “.” in them will fail. To avoid failure, use example-com-package-repo instead of example.com-package-repo:

    [email protected]:~$ aws s3 mb s3://example-com-package-repo
    make_bucket: example-com-package-repo
    
  2. Once you have created the bucket, create a new host for using Aptly and managing the repository. You could reuse an existing host if you want to (for example, build a host from your CI/CD system if you want to track repository releases from there). The important thing is that the host must have the IAM role provided below in order to manage the repository. The alternative is to leave AWS access credentials in a config file, which we would prefer not to do.

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "Stmt1405592139000",
          "Effect": "Allow",
          "Action": [
            "s3:DeleteObject",
            "s3:GetObject",
            "s3:ListBucket",
            "s3:PutObject",
            "s3:PutObjectAcl"
          ],
          "Resource": [
            "arn:aws:s3:::example-com-package-repo/*", "arn:aws:s3:::example-com-package-repo"
          ]
        }
      ]
    }
    
  3. On the host where Aptly will be managed, set up GPG for package repository verification and signing. Add the Ubuntu package signing keys which are listed here. Aptly needs these to verify the package repository it will be mirroring.

    $ gpg --no-default-keyring --keyring trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 40976EAF437D05B5 3B4FE6ACC0B21F32
    gpg: directory `/home/ubuntu/.gnupg' created
    gpg: new configuration file `/home/ubuntu/.gnupg/gpg.conf' created
    gpg: WARNING: options in `/home/ubuntu/.gnupg/gpg.conf' are not yet active during this run
    gpg: keyring `/home/ubuntu/.gnupg/secring.gpg' created
    gpg: keyring `/home/ubuntu/.gnupg/trustedkeys.gpg' created
    gpg: requesting key 437D05B5 from hkp server keys.gnupg.net
    gpg: requesting key C0B21F32 from hkp server keys.gnupg.net
    gpg: /home/ubuntu/.gnupg/trustdb.gpg: trustdb created
    gpg: key 437D05B5: public key "Ubuntu Archive Automatic Signing Key <[email protected]>" imported
    gpg: key C0B21F32: public key "Ubuntu Archive Automatic Signing Key (2012) <[email protected]>" imported
    gpg: no ultimately trusted keys found
    gpg: Total number processed: 2
    gpg:               imported: 2  (RSA: 1)
    
  4. Once the Ubuntu package signing keys are imported, create your own package signing key to be used to sign your mirrors:

    $ gpg --gen-key
    gpg (GnuPG) 1.4.20; Copyright (C) 2015 Free Software Foundation, Inc.
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    
    Please select what kind of key you want:
       (1) RSA and RSA (default)
       (2) DSA and Elgamal
       (3) DSA (sign only)
       (4) RSA (sign only)
    Your selection? 1
    RSA keys may be between 1024 and 4096 bits long.
    What keysize do you want? (2048)
    Requested keysize is 2048 bits
    Please specify how long the key should be valid.
             0 = key does not expire
            = key expires in n days
          w = key expires in n weeks
          m = key expires in n months
          y = key expires in n years
    Key is valid for? (0) 0
    Key does not expire at all
    Is this correct? (y/N) y
    
    You need a user ID to identify your key; the software constructs the user ID
    from the Real Name, Comment and Email Address in this form:
        "Heinrich Heine (Der Dichter) <[email protected]>"
    
    Real name: Example Corp Operations
    Email address: [email protected]
    Comment: apt signing key
    You selected this USER-ID:
        "Example Corp Operations (apt signing key) <[email protected]>"
    
    Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
    You need a Passphrase to protect your secret key.
    
    gpg: gpg-agent is not available in this session
    You don't want a passphrase - this is probably a *bad* idea!
    I will do it anyway. You can change your passphrase at any time,
    using this program with the option "--edit-key".
    
    We need to generate a lot of random bytes. It is a good idea to perform
    some other action (type on the keyboard, move the mouse, utilize the
    disks) during the prime generation; this gives the random number
    generator a better chance to gain enough entropy.
    
    Not enough random bytes available. Please do some other work to give
    the OS a chance to collect more entropy! (Need 247 more bytes)
    .+++++
    +++++
    We need to generate a lot of random bytes. It is a good idea to perform
    some other action (type on the keyboard, move the mouse, utilize the
    disks) during the prime generation; this gives the random number
    generator a better chance to gain enough entropy.
    
    Not enough random bytes available. Please do some other work to give
    the OS a chance to collect more entropy! (Need 56 more bytes)
    ..+++++
    .+++++
    gpg: key 64560093 marked as ultimately trusted
    public and secret key created and signed.
    
    gpg: checking the trustdb
    gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
    gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
    pub   2048R/2D365D26 2017-02-25
          Key fingerprint = DA88 A6C8 F218 EE8D 7149  A5DF 5220 F60D 2D36 5D26
    uid                  Example Corp Ops Team (apt repo signing key) <[email protected]>
    sub   2048R/4E0561C1 2017-02-25  
    
  5. With the key created, you will need to keep track of both the public key and the private key. The public key will need to be distributed to all your hosts so they can verify your package repository. The private key will need to be stored safely so future updates can be signed:

    $ gpg --export --armor 2D365D26 > [email protected]
    $ gpg --export-secret-key --armor 2D365D26 > [email protected]
    $ ls -al [email protected]*
    -rw-rw-r-- 1 ubuntu ubuntu 1739 Feb 26 20:34 [email protected]
    -rw-rw-r-- 1 ubuntu ubuntu 3508 Feb 25 17:00 [email protected]
    
  6. With GPG signing keys in place, start setting up Aptly. The packages (aptly and apt-s3-transport) need to be installed. It is wise to use the latest Aptly release which was not available in Ubuntu 16.04. The latest version supports signing repositories with SHA-256 instead of SHA-1. In addition to SHA-1 having been demonstrated as exploitable, APT in 16.04 throws warnings, and in 16.10 and on APT will refuse to install packages from repositories signed with SHA-1.

  7. Create a config file for Aptly. Running aptly config show will create a very basic config file:

    $ aptly config show
    
    {
      "rootDir": "/home/ubuntu/.aptly",
      "downloadConcurrency": 4,
      "downloadSpeedLimit": 0,
      "architectures": [],
      "dependencyFollowSuggests": false,
      "dependencyFollowRecommends": false,
      "dependencyFollowAllVariants": false,
      "dependencyFollowSource": false,
      "gpgDisableSign": false,
      "gpgDisableVerify": false,
      "downloadSourcePackages": false,
      "ppaDistributorID": "ubuntu",
      "ppaCodename": "",
      "skipContentsPublishing": false,
      "S3PublishEndpoints": {},
      "SwiftPublishEndpoints": {}
      }
    }
    

    Two values you might want to adjust are rootDir, to put packages onto a disk with enough room before they are published to S3, and architectures to limit the OS architectures mirrored.

  8. You now need to add S3 publishing endpoints to S3PublishEndpoints. An endpoint needs a name and then a region, bucket, and acl associated with it at a minimum. I also choose to separate each repository I mirror under different prefixes. This makes it easier to delete and retry if you find that your repository has an issue you can’t solve:

    "xenial-security-main-repo": {
      "region": "us-east-1",
      "bucket": "example-com-package-repo",
      "prefix": "ubuntu-xenial-security-main",
      "acl": "private"
    }
    

    This is the publishing endpoint you will use for the Ubuntu Xenial security repository for the main component. Notice that the prefix is named slightly differently. At times, I have had trouble fully understanding which arguments are to be passed to which commands when, so this makes it easier for me to distinguish things as I read my own documentation. Additionally, the acl is set to private. We will be requiring AWS IAM credentials to access this repository. You don’t have to, but it is a good practice, particularly if private packages produced internally are to be added to the repository.

    Here is a sample final config file that mirrors Aptly to a larger disk, only mirrors the amd64 architecture, and mirrors the four repositories we want to manage:

    {
      "rootDir": "/srv/aptly",
      "downloadConcurrency": 4,
      "downloadSpeedLimit": 0,
      "architectures": ["amd64"],
      "dependencyFollowSuggests": false,
      "dependencyFollowRecommends": false,
      "dependencyFollowAllVariants": false,
      "dependencyFollowSource": false,
      "gpgDisableSign": false,
      "gpgDisableVerify": false,
      "downloadSourcePackages": false,
      "ppaDistributorID": "ubuntu",
      "ppaCodename": "",
      "skipContentsPublishing": false,
      "S3PublishEndpoints": {
        "xenial-security-main-repo": {
          "region": "us-east-1",
          "bucket": "example-com-package-repo",
          "prefix": "ubuntu-xenial-security-main",
          "acl": "private"
        },
        "xenial-security-universe-repo": {
          "region": "us-east-1",
          "bucket": "example-com-package-repo",
          "prefix": "ubuntu-xenial-security-universe",
          "acl": "private"
        },
        "xenial-updates-main-repo": {
          "region": "us-east-1",
          "bucket": "example-com-package-repo",
          "prefix": "ubuntu-xenial-updates-main",
          "acl": "private"
        },
        "xenial-updates-universe-repo": {
          "region": "us-east-1",
          "bucket": "example-com-package-repo",
          "prefix": "ubuntu-xenial-updates-universe",
          "acl": "private"
        }
      }
    }
    
  9. Now create a mirror in Aptly and point it to an Official Ubuntu package repository. Then update the mirror to retrieve the packages. In this example, you will use the AWS Ubuntu mirror of xenial-security for the main component. Notice that the mirror is named xenial-security-main-mirror. That helps distinguish it from the S3 publishing endpoint xenial-security-main-repo later on:

    $ aptly mirror create xenial-security-main-mirror http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial-security main
    Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/xenial-security/InRelease...
    gpgv: Signature made Sun 26 Feb 2017 06:52:31 PM UTC using DSA key ID 437D05B5
    gpgv: Good signature from "Ubuntu Archive Automatic Signing Key <[email protected]>"
    gpgv: Signature made Sun 26 Feb 2017 06:52:31 PM UTC using RSA key ID C0B21F32
    gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <[email protected]>"
    
    Mirror [xenial-security-main-mirror]: http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial-security successfully added.
    You can run 'aptly mirror update xenial-security-main-mirror' to download repository contents.
    
    $ aptly mirror update xenial-security-main-mirror
    Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/xenial-security/InRelease...
    gpgv: Signature made Sun 26 Feb 2017 06:52:31 PM UTC using DSA key ID 437D05B5
    gpgv: Good signature from "Ubuntu Archive Automatic Signing Key <[email protected]>"
    gpgv: Signature made Sun 26 Feb 2017 06:52:31 PM UTC using RSA key ID C0B21F32
    gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <[email protected]>"
    Downloading & parsing package files...
    Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/xenial-security/main/binary-amd64/Packages.gz...
    Building download queue...
    Download queue: 1478 items (7.75 GiB)
    
  10. With packages downloaded, create a snapshot of the repository. This will record the current state of the repository and will be what you publish to S3. Using a snapshot has the advantage that if the repository in S3 is ever damaged or accidentally deleted, you can redeploy this specific repository state. After the snapshot is created, publish it to S3:

    $ DATE=$(date +%Y%m%d%H%M)
    $ aptly snapshot create xenial-security-main-snap-${DATE} from mirror xenial-security-main-mirror
    
    Snapshot xenial-security-main-snap-201702262142 successfully created.
    You can run 'aptly publish snapshot xenial-security-main-snap-201702262142' to publish snapshot as Debian repository.
    
    $ aptly publish snapshot xenial-security-main-snap-${DATE} s3:xenial-security-main-repo:
    Loading packages...
    Generating metadata files and linking package files…
    
  11. At this point, the repository is published to S3. How do you update the repository? The process is similar: first update the mirror and snapshot it again:

    $ aptly mirror update xenial-security-main-mirror
    Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/xenial-security/InRelease...
    gpgv: Signature made Sun 26 Feb 2017 06:52:31 PM UTC using DSA key ID 437D05B5
    gpgv: Good signature from "Ubuntu Archive Automatic Signing Key <[email protected]>"
    gpgv: Signature made Sun 26 Feb 2017 06:52:31 PM UTC using RSA key ID C0B21F32
    gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <[email protected]>"
    Downloading & parsing package files...
    Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/xenial-security/main/binary-amd64/Packages.gz...
    Building download queue...
    Download queue: 4 items (26 MiB)
    
    $ DATE=$(date +%Y%m%d%H%M)
    $ aptly snapshot create xenial-security-main-snap-${DATE} from mirror xenial-security-main-mirror
    
    Snapshot xenial-security-main-snap-201702281711 successfully created.
    You can run 'aptly publish snapshot xenial-security-main-snap-201702281711' to publish snapshot as Debian repository.
    
  12. Publish the new snapshot by switching the S3 endpoint to the new snapshot. Pay close attention to the command. The command switched the repository named xenial-security to the latest snapshot. That is the name of the repository you gave in the aptly mirror create command when you first started this process:

    $ aptly publish switch xenial-security s3:xenial-security-main-repo:  xenial-security-main-snap-${DATE}
    Loading packages...
    Generating metadata files and linking package files...
    

Using the New Repository Mirror

To use the repository, hosts will need a bit of configuration. To start, they will need an IAM role that lets them access the repository. Remember, you have set the acl on the bucket to private, so hosts will rely on their IAM role for access.

  1. Create an IAM policy and add it to the appropriate IAM role:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "Stmt1405592139000",
          "Effect": "Allow",
          "Action": [
            "s3:GetObject",
            "s3:ListBucket",
          ],
          "Resource": [
            "arn:aws:s3:::example-com-package-repo/*", "arn:aws:s3:::example-com-package-repo"
          ]
        }
      ]
    }
    
  2. With the policy in place, ensure that apt-s3-transport is installed on hosts so APT can be configured to use S3 URIs. Note: apt-s3-transport does not adhere to the S3 URI format used by the AWS SDKs. So, s3://<bucket_name>/prefix> does not work. Instead the format is s3://.s3.amazonaws.com/<prefix>.

  3. Finally, add the S3 bucket as an APT source, and disable the upstream repository that you are now mirroring:

    $ cat /etc/apt/sources.list:
    #deb http://security.ubuntu.com/ubuntu xenial-security main
    deb s3://example-com-package-repo.s3.amazonaws.com/ubuntu-xenial-security-main/ xenial-security main
    

Conclusion

Now that you have control over when your repository is updated, you may feel more comfortable having package updates running regularly via cron because you now have better control over when updates are made available. You may find this easy enough to do via Jenkins such that you are willing to increase your cadence. You can also publish a snapshot to multiple S3 endpoints. This would allow you to point some hosts to a repository that gets updates ahead of other systems and therefore lets it be used for testing the stability of the latest updates.

In any event, determine what patching style is best for you and just start. You can increase or decrease your patching speed depending on what works best for you. Remember: The worst thing you can do is not patch your systems.