Katello: Import CentOS Errata into Pulp

Working with Katello – part 2. We will be importing CentOS errata into Pulp.

This article is part of the Homelab Project with KVM, Katello and Puppet series.

Homelab

We have Katello installed on a CentOS 7 server:

katello.hl.local (10.11.1.4) – see here for installation instructions

See the image below to identify the homelab part this article applies to.

Credits

This is possible thanks to the following sites:

  1. https://cefs.steve-meier.de/ – CEFS: CentOS Errata for Spacewalk
  2. https://github.com/rdrgmnzs/pulp_centos_errata_import – the script that imports CentOS errata into Pulp/Katello

Installation

Install the following packages:

# yum install git \
  pulp-admin-client \
  pulp-rpm-admin-extensions \
  pulp-rpm-consumer-extensions \
  pulp-rpm-handlers \
  pulp-rpm-yumplugins \
  pulp-rpm-admin-extensions \
  pulp-consumer-client \
  python-pulp-agent-lib \
  perl-Text-Unidecode \
  perl-XML-Simple \
  perl-XML-Parser

Configure CentOS Errata Import

Make sure that the CentOS repositories have been synced. We’ve done this in the previous article.

Clone the repository:

# cd /opt && git clone https://github.com/rdrgmnzs/pulp_centos_errata_import.git
# cd ./pulp_centos_errata_import

Download the latest errata archive and extract the file:

# wget -N https://cefs.steve-meier.de/errata.latest.xml.bz2
# bunzip2 ./errata.latest.xml.bz2

Configure Pulp authetication:

# mkdir -m0700 ~/.pulp
# cat /etc/pki/katello/certs/pulp-client.crt /etc/pki/katello/private/pulp-client.key > ~/.pulp/user-cert.pem
# chmod 0400 ~/.pulp/user-cert.pem

Import CentOS Errata into Pulp

Chances are that we won’t know which repository IDs to use when importing CentOS errata, therefore we can run the following command without specifying any (it may take longer):

# perl ./errata_import.pl --errata=errata.latest.xml

Alternatively, we can view a list of Pulp repositories by using following command:

# pulp-admin repo list | less

And then import errata into repositories required, e.g.:

# perl ./errata_import.pl \
  --errata=errata.latest.xml \
  --include-repo=83418f8f-e069-4b24-8e67-4b07af5e87d2 \
  --include-repo=0abbd033-67cc-4e42-bff0-237d52f1bcdb

There was a case when we got the error saying “Skipping errata […] — No packages found”. If that happens, try passing the repository ID you want to load the errata for with the –include-repo flag.

After importing errata, use the mighty Hammer to force post-sync actions (Katello 3.0 and up):

# hammer settings set \
  --name "force_post_sync_actions" \
  --value "true"

We have to sync the repositories so that errata is published (and visible in the Katello interface).

# hammer repository synchronize \
  --name "base_x86_64" \
  --product "el7_repos"

If all goes well, we should see some errata:

# hammer repository info \
  --name "base_x86_64" \
  --product "el7_repos"
ID:                 1
Name:               base_x86_64
Label:              base_x86_64
Organisation:       Lisenet
Red Hat Repository: no
Content Type:       yum
Checksum Type:      sha256
Mirror on Sync:     no
URL:                http://mirror.centos.org/centos/7/os/x86_64/
Publish Via HTTP:   yes
Published At:       http://katello.hl.local/pulp/repos/lisenet/Library/custom/el7_repos/base_x86_64/
Relative Path:      lisenet/Library/custom/el7_repos/base_x86_64
Download Policy:    on_demand
Product:            
    ID:   1
    Name: el7_repos
GPG Key:            
    ID:   1
    Name: RPM-GPG-KEY-CentOS-7
Sync:               
    Status:         Success
    Last Sync Date: 28 days
Created:            2018/02/20 20:56:52
Updated:            2018/02/21 23:16:44
Content Counts:     
    Packages:       9591
    Package Groups: 84
    Errata:         830

Note how “Mirror on Sync” is set to “no”. This is to prevent loaded errata from being cleared on next sync.

Here is some info for a randomly selected errata ID:

# hammer erratum info --id 4191
ID:          4191
Errata ID:   CESA-2018:0260
Title:       CentOS systemd Security Update
Type:        security
Severity:    Moderate
Issued:      2018-02-01
Updated:     2018-02-01
Description: Not available
Summary:     
Solution:

34 thoughts on “Katello: Import CentOS Errata into Pulp

  1. I’m still having issues with importing,

    It seems that it imported ~4k erratas for EPEL, but nothing for the others (especially base)

    running it displays a whole bunch of :

    NOTICE: Skipping errata CEBA-2018:0420 (CentOS openscap BugFix Update) -- No packages found
    NOTICE: Skipping errata CEBA-2018:0421 (CentOS xorg-x11-server BugFix Update) -- No packages found
    NOTICE: Skipping errata CEBA-2018:0422 (CentOS nfs-utils BugFix Update) -- No packages found
    NOTICE: Skipping errata CEBA-2018:0510 (CentOS sssd BugFix Update) -- No packages found
    NOTICE: Skipping errata CEBA-2018:0511 (CentOS ksh BugFix Update) -- No packages found
    NOTICE: Skipping errata CEBA-2018:0513 (CentOS gcc BugFix Update) -- No packages found
    NOTICE: Skipping errata CEBA-2018:0514 (CentOS irqbalance BugFix Update) -- No packages found
    NOTICE: Skipping errata CEBA-2018:0597 (CentOS tzdata BugFix Update) -- No packages found
    INFO: Errata for CEBA-2018:0658 already exists
    INFO: Errata for CEBA-2018:0659 already exists
    INFO: Errata for CEBA-2018:0661 already exists
    INFO: Errata for CEBA-2018:0662 already exists
    INFO: Errata for CEBA-2018:0663 already exists
    INFO: Errata for CEBA-2018:0664 already exists
    INFO: Errata for CEBA-2018:0665 already exists
    INFO: Errata for CEBA-2018:0668 already exists
    INFO: Errata for CEBA-2018:0669 already exists

    followed by another whole bunch of (which take a VERY LONG TIME):

    INFO: Creating errata for CEBA-2018:0728 (CentOS python-slip BugFix Update) (3 of 3)
    INFO: +----------------------------------------------------------------------+
                                  Unit Upload
    +----------------------------------------------------------------------+
    
    Extracting necessary metadata for each request...
    ... completed
    
    Creating upload requests on the server...
    [==================================================] 100%
    Initializing upload
    ... completed
    
    Starting upload of selected units. If this process is stopped through ctrl+c,
    the uploads will be paused and may be resumed later using the resume command or
    canceled entirely using the cancel command.
    
    Importing into the repository...
    This command may be exited via ctrl+c without affecting the request.
    
    
    [\]
    Running...
    
    Task Succeeded
    
    
    Deleting the upload request...
    ... completed

    Command I’m trying now is:

    # perl ./errata_import.pl --errata=errata.latest.xml --include-repo=c659ccfb-afeb-431c-891e-3228e683008e

    the ID is for base repo. I tried without it as well before that which gave me 4k erratas on EPEL.

    Any ideas?

    Thanks

    • I think there is some confusion here. The script imports CentOS errata, but it has nothing to do with EPEL errata. While EPEL repository provides security metadata, CentOS repositories don’t supply it. This is the reason why we want to import CentOS errata from CentOS-Announce mailing list.

      From the logs that you’ve posted, it looks like CentOS errata has already been applied. Try forcing post-sync actions and then re-sync the repository.

    • That’s why I was confused too. Right now I see available Errata like “FEDORA-EPEL-2018-ad387c7768” in my system but nothing under base repo.

      Once I applied force post sync actions and did repo sync, I started seeing base repo errata.

      By the way thanks so much for this series. I’m learning a lot from it.

  2. Conflicts for Katello 3.7 when installing the packages
    ——————-
    –> Processing Conflict: katello-agent-3.3.4-1.el7.noarch conflicts pulp-consumer-client
    –> Finished Dependency Resolution
    Error: katello-agent conflicts with pulp-consumer-client-2.16.4-1.el7.noarch
    You could try using –skip-broken to work around the problem
    You could try running: rpm -Va –nofiles –nodigest

    • I used Katello 3.6 and didn’t have this problem I’m afraid. I later upgraded Katello from 3.6 to 3.7 and errata routine still worked. Also, I don’t have the katello-agent package installed on the server.

      It looks like some packages are obsoleted by katello-agent: pulp-rpm-handlers and python-pulp-agent-lib. This may very well be the case here.

    • @Eric, et al.:

      As of today, running the “yum” command, verbatim, as shown in the article, results in the following error:

      # yum install ...
      
      --> Processing Conflict: katello-agent-3.3.5-1.el7.noarch conflicts pulp-consumer-client
      --> Finished Dependency Resolution
      --> Finding unneeded leftover dependencies
      Found and removing 0 unneeded dependencies
      Error: katello-agent conflicts with pulp-consumer-client-2.16.4-1.el7.noarch
       You could try using --skip-broken to work around the problem
       You could try running: rpm -Va --nofiles --nodigest

      This is due to various changes since this article was first posted. In order to complete this step successfully, simply remove “pulp-rpm-handlers” and “python-pulp-agent-lib” from the list of packages to be installed; i.e.:

      # yum install git pulp-admin-client pulp-rpm-admin-extensions \
          pulp-rpm-consumer-extensions pulp-rpm-yumplugins pulp-consumer-client \
          perl-Text-Unidecode perl-XML-Simple perl-XML-Parser

      Additionally, you should be able to remove “git” from the package list as it is likely already installed by this point (if you have been following the previous articles).

      (N.B.: The comment form likely messed up the formatting of the above; hopefully you can figure it out!)

      @Tomas:

      Thanks for the great series of articles!

    • Thanks for your feedback!

      Try removing pulp-rpm-handlers and python-pulp-agent-lib from the list of packages to be installed as per suggestion in another comment.

  3. Very nice article!

    I am on 3.8 and it looks like force_post_sync_actions is no longer there..

    $ hammer settings list | grep force_post_sync_actions
    $ hammer settings set --name "force_post_sync_actions" --value "true"      
    Could not update the setting:
      Error: setting not found.
    • Thanks for your feedback. I’ve got several comments saying that there have been some changes between Katello versions. This is unfortunate but inevitable, because the project is in active development.

      My homelab is on Katello 3.6. According to this changelog, force_post_sync_actions has been deprecated, and has since been replaced by the skip-metadata-check hammer flag.

      Users have to set skip-metadata-check to force post-sync actions.

    • Hi,
      Did you ever get this working? I am having troubles importing errata also.
      I’ve tried the following but still no luck:
      hammer repository synchronize –skip-metadata-check true –id 3
      i’ve also tried to recreate the repository using: –mirror-on-sync=false to prevent loaded errata from being cleared on next sync but no luck either. Thanks.
      Phat

    • Hi all:
      The post is 2 years earlier there!
      I just get katello-3.14.1 installed and try to import the ERRATA, but however unfortunately I find neither force_post_sync_actions nor skip-metadata-check are there

      [root@foreman pulp_centos_errata_import]# hammer settings list | grep force_post_sync_actions
      [root@foreman pulp_centos_errata_import]# hammer settings list | grep skip-metadata-check
      [root@foreman pulp_centos_errata_import]#

      any idea ?

    • I think that force_post_sync_actions has been replaced with skip-metadata-check hammer flag to force post-sync actions.

  4. Hi,
    I am on 3.8 and tried to use the ‘–skip-metadata-check true’ to force post-sync actions, however it does not seem to be getting the errata data – any idea?

    hammer repository synchronize –product ‘Centos 7’ –name ‘Centos 7 – Base – x86_64’ –skip-metadata-check true –organization MyOrg

    hammer repository info –id 1
    ID: 1
    Name: Centos 7 – Base – x86_64
    Label: Centos_7_-_Base_-_x86_64
    Organization: MyOrg
    Red Hat Repository: no
    Content Type: yum
    Mirror on Sync: yes
    URL: http://mirror.centos.org/centos/7/os/x86_64/
    Publish Via HTTP: yes
    Published At: katello.example.com/pulp/repos/Vendavo/Library/custom/Centos_7/Centos_7_-_Base_-_x86_64/
    Relative Path: MyOrg/Library/custom/Centos_7/Centos_7_-_Base_-_x86_64
    Download Policy: on_demand
    Product:
    ID: 1
    Name: Centos 7
    GPG Key:

    Sync:
    Status: Success
    Last Sync Date: less than a minute
    Created: 2018/11/13 20:54:23
    Updated: 2018/11/13 21:09:36
    Content Counts:
    Packages: 9911
    Source RPMS: 0
    Package Groups: 88
    Errata: 0

  5. Sorry, please disregard my previous comment, apparently all errata appeared after disabling ‘Mirror on Sync’.

    • Thanks. Yes, as mentioned in the blog post, “Mirror on Sync” needs setting to “no” in order to prevent loaded errata from being cleared on next sync

  6. Hello

    i follow the above all is workig as expected, i could see the errata when pulp-admin repo list | less but it is not synching, mirror on sync is turned off.

    Id: 3-CentOS7_Content_View-v1_0-3c9a1001-6382-4a4c-bbea-d9dc35d
    88411
    Display Name: OS_x86_64
    Description: None
    Content Unit Counts:
    Distribution: 1
    Erratum: 887
    Package Group: 88

    hammer repository info –name “OS_x86_64” –product “CentOS 7 x86_64” –organization pvt

    Name: OS_x86_64
    Label: OS_x86_64
    Sync:
    Status: Success
    Last Sync Date: 10 minutes
    Created: 2019/06/13 17:55:36
    Updated: 2019/06/14 07:39:13
    Content Counts:
    Packages: 10019
    Source RPMS: 0
    Package Groups: 88
    Errata: 0

  7. Hi,

    After completion of errata import, im able to see the errata in content view but the same time which is not visible for content hosts or any individual host. how can we fix it.

    Foreman 1.24 and katello 3.14
    and client side is
    katello-host-tools-fact-plugin-3.0.2-1.el7.noarch
    katello-ca-consumer-insightsof.ctrls.in-1.0-1.noarch
    katello-client-repos-3.4.0-3.el7.noarch
    katello-host-tools-3.5.1-2.el7.noarch

    There are 292 total Errata in this organization but none match the above filters.

  8. Hi,

    First of all, thanks for this, most of the steps worked, and as u mentioned the settings command is replaced by :hammer repository synchronize –skip-metadata-check true –name “base_x86_64_7.8” –product “el7_repos”

    I had a question, after the above synch of the repo do we have to publish and promote the CV again?

    • Also, beware if you install katello 3.16 or later this process will not work. Starting 3.16 the pulp data =base has been moved to postgresql db, so the pulp-admin will not report any repositories and hence importing/updating errata will not happen.

      [root@katello tools]# mongo
      MongoDB shell version v3.4.9
      connecting to: mongodb://127.0.0.1:27017
      MongoDB server version: 3.4.9
      > use pulp_database;
      switched to db pulp_database
      > db.repos.find()

      { “_id” : ObjectId(“5f1cad6e4a54e704e2a80ecb”), “repo_id” : “1-centos-7-v1_0-puppet-a7227dc4-5668-43ae-bbf6-d3a0a67b0f32”, “display_name” : “centos-7 Puppet Environment”, “notes” : { “_repo-type” : “puppet-repo” }, “scratchpad” : { }, “content_unit_counts” : { }, “_ns” : “repos” }
      And the actual Yum repos are stored in PostgreSQL:

      [root@katello tools]# su – postgres
      [postgres@katello ~]$ psql pulpcore
      psql (12.1)
      Type “help” for help.
      pulpcore=# select pulp_id, name from core_repository;

      pulp_id | name
      ————————————–+—————
      e0c17697-0649-49ad-b6c3-50606b1b5531 | base-38848
      a5b2c791-36ce-43d9-aa73-b9cd8adc6aeb | updates-47970
      bd454296-a9f4-47a8-af00-9311585ec40c | epel-98560
      (3 rows)

  9. Experiencing issues with this – any updates for Katello 4.2?

    ran the perl script and included the repo names found from querying the postgresql repository as mentioned in above comment.

    Script output states “No Packages found” for all errata”

    Repos are CentOS 7 x86_64 Base and Updates

    Foreman 3.0.1
    Katello 4.2

  10. I am having issues while trying to import errata to katello:

    INFO: Creating errata for CESA-2020:4060 (Important CentOS kernel Security Update) (13 of 14)
    Use of uninitialized value in concatenation (.) or string at /opt/pulp_errata_import/errata_import.pl line 247.
    Use of uninitialized value in concatenation (.) or string at /opt/pulp_errata_import/errata_import.pl line 247.
    Use of uninitialized value in concatenation (.) or string at /opt/pulp_errata_import/errata_import.pl line 247.
    Use of uninitialized value in concatenation (.) or string at /opt/pulp_errata_import/errata_import.pl line 247.

    INFO: Command: erratum
    Description: creates a new erratum

    Available Arguments:

    –bg – if specified, the client process will end immediately
    (the task will continue to run on the server)
    –repo-id – (required) unique identifier; only alphanumeric, ., -,
    and _ allowed
    -v – display extra information about the upload process
    –erratum-id, -i – (required) id of the erratum to create
    –title, -n – (required) title of the erratum
    –description, -d – (required) description of the erratum
    –version – (required) version of the erratum
    –release – (required) release of the erratum
    –type, -t – (required) type of the erratum; examples: “bugzilla”,
    “security”, “enhancement”
    –status, -s – (required) status of the erratum; examples: “final”
    –updated, -u – (required) timestamp the erratum was last updated;
    expected format “YYYY-MM-DD HH:MM:SS”
    –issued – (required) timestamp the erratum was issued; expected
    format “YYYY-MM-DD HH:MM:SS”
    –reference-csv, -r – path to a CSV file containing reference information. An
    example of a reference would be information of a
    bugzilla issue. Format for each entry is:
    “href,type,id,title”
    –pkglist-csv, -p – (required) path to a CSV file containing package list
    information. Format for each entry is:
    “name,version,release,epoch,arch,filename,checksum,check
    sum_type,sourceurl”
    –from – (required) erratum issuer, used in the ‘from_str’ for
    the erratum; typically an email address
    –pushcount – pushcount for the erratum; an integer, defaults to 1 if
    unspecified
    –reboot-suggested – if specified, the erratum will be marked as
    ‘reboot-suggested’
    –restart-suggested – if specified, the erratum will be marked as
    ‘restart-suggested’
    –relogin-suggested – if specified, the erratum will be marked as
    ‘relogin-suggested’
    –severity – severity of the erratum
    –rights – rights for the erratum
    –summary – summary for the erratum
    –solution – solution for the erratum
    The following options are required but were not specified:
    –release

    /bin/perl /opt/pulp_errata_import/errata_import.pl –errata=/opt/pulp_errata_import/errata.latest.xml –include-repo=4bd040c9-d19c-4da8-8564-43e179581ef2

    It just started to happen at around december 3rd, before then, it was working like a charm.

    I am using katello-3.14.1-1.el7.noarch and Centos7 at the moment. pulp-server-2.21.5-1.el7.noarch

    • Have you tried specifying the release?

      The following options are required but were not specified:
      –release
      
    • Found the problem, our WAF was blocking a specific request made by:

      pulp-admin rpm repo content rpm –repo-id=4bd040c9-d19c-4da8-8564-43e179581ef2 –match=filename=kernel-tools-libs-devel-3.10.0-1160.el7.x86_64 –fields=name,version,release,epoch,arch,checksum,checksumtype

      When that got blocked, the script just received a 403 error instead of the table it was expecting thus making the variables it uses at around that erroring line 247 being blank

      Its all good now after whitelisting the pulp urls

    • Thanks, that makes sense. From my experience, WAFs are notorious for blocking legitimate traffic. Great care should be taken when configuring WAF rules.

Leave a Reply

Your email address will not be published. Required fields are marked *