Catégorie : GEEK

Mise à jour Drupal 8.9.17 vers 10.2.3.

 

 

Bonjour à tous,

Utilisant DRUPAL pour un de mes projets (CRX-RADIO-CLOUD), j’ai été déjà amené à faire des mises à jour. Pour ce faire j’ai toujours utilisé COMPOSER, un outil que j’aime bien, cependant je dois dire que là j’en ai « bavé » à chercher des solutions à différentes erreurs/bugs et versions de librairies, je partage ici mes notes de cette mise à jour « massive ».

A savoir que DRUPAL utilise différentes version de PHP suivant la version ici 8, 9 puis 10, J’ai donc du suivre aussi la mise à jour de PHP (7.4,8.0,8.1). Heureusement j’avais à disposition toutes les versions et modules APACHE donc pas de soucis ici pour le prérequis PHP du serveur utilisé pour faire la mise à jour :

Voici mon « upgrade path » ici : 

8.9.17 à 9.0
9.0 à 9.1.0
9.1.0 à 9.2.21
9.2.1 à 9.2.4
9.2.4 à 9.4.15
9.4.15 à 9.5.0
9.5.0 à 10.2.3
10.2.3 à 10.2.4

=> A savoir que mettre à jour DRUPAL via COMPOSER prend énormément de mémoire, je dois donc ajouter une SWAP sur mon serveur pour absorber la charge mémoire : ( >=4G )

# mkdir -p /var/_swap
# cd /var/_swap
# fallocate -l 4G swapfile
# chmod 600 swapfile
# mkswap swapfile
# swapon swapfile


Tableau des commandes COMPOSER ‘utiles’ ici :

#affiche les infos de versions : 
composer outdated "drupal/*"

#enlève une lib au composer.json (+ --no-update): 
composer remove 'lib/lib:^x.x'                              

#ajout une lib au composer.json  (+ --no-update): 
composer require 'lib/lib:^x.x.x'       

#mise à jour std, après avoir modifier le composer.json en conséquence :

composer require drupal/core-recommended:[VERSION] \
drupal/core-composer-scaffold:[VERSION] \
drupal/core-project-message:[VERSION] \
--update-with-all-dependencies


Mise à jour DRUPAL 8.9.17 à 9.0 :

Pour commencer, je suis donc parti d’un PHP7.3 que j’ai du passé en 7.4 (partie CLI), puis une bascule du module PHP/APACHE.

Après un DUMP complet de la base et un SNAPSHOT des données, je peux démarrer cette mise à jour :

#bascule du PHP/CLI en v7.4, préalablement installé : 

# update-alternatives --display php
# update-alternatives --set php /usr/bin/php7.4


Lancement de la mise à jour :

# composer require 'drupal/core-recommended:^9' 'drupal/core-composer-scaffold:^9' 'drupal/core-project-message:^9' --update-with-dependencies --no-update
# composer require 'drupal/core-dev:^9' --dev --update-with-dependencies --no-update
# composer require "drupal/core:9.0.0 as 8.9.17" --no-update && composer update

A la suite d’une erreur PHP ‘fatal error: require(): failed opening required vendor/composer/../symfony/polyfill-intl-normalizer/bootstrap.php’
=> je renomme le dossier vendor en vendor_, pour forcer sa mise à jour complète en réinstallant toutes les librairies :

# mv vendor vendor_

Je peux alors lancer la mise à jour, l’option ‘no-update‘ permet de préparer le fichier composer.json,
avec l’ajout des versions de libs qui va être utilisé par le gestionnaire de paquets COMPOSER :

# composer require 'drupal/core-recommended:^9' 'drupal/core-composer-scaffold:^9' 'drupal/core-project-message:^9' --update-with-dependencies --no-update

# composer require 'drupal/core-dev:^9' --dev --update-with-dependencies --no-update

# composer require "drupal/core:9.0.0 as 8.9.17" --no-update && composer update

Une fois exécuté, je passe au lancement du script en HTTPS : /update.php  qui permet le passage des mises à jour sur la base de données.
Ce script est très important car il remonte aussi les soucis de d’extensions ou thèmes non compatibles, c’est aussi pour ca que je préfère ne pas utiliser DRUSH qui lui gère cela automatiquement.


Mise à jour DRUPAL de 9.0 à 9.1.0 :

Je viens modifier la version de 9.0 à 9.1.0 dans le composer.json :

Modification du composer json : 

    "require": {
        "composer/installers": "^1.0.24",
        "drupal/core": "9.1.0",
        "drupal/codesnippet": "^1.6",
        "drupal/colorbox": "^1.4",
        "drupal/bootstrap": "^3.15",
        "drupal/bootstrap_library": "^1.11",
        "egulias/email-validator": "^2.0",
        "drupal/core-recommended": "9.1.0",
        "drupal/core-composer-scaffold": "9.1.0",
        "drupal/core-project-message": "9.1.0",

+ rm -f composer.lock

Ensuite je passe le COMPOSER de la version 1 à 2 puis le PHP CLI de la version 7.4 à 8.0 + du module APACHE  :

Passage de COMPOSER en Version 2 : 

# composer self-update --2     ( peux etre rollback via :  composer self-update --rollback     ). 

Patch important pour passer en 9.1 : 

composer config --unset scripts.post-package-install
composer config --unset scripts.post-package-update
composer require drupal/core-vendor-hardening:^9


Use the `composer fund` command to find out more!
Scaffolding files for drupal/core:
  - Copy [web-root]/.htaccess from assets/scaffold/files/htaccess
  - Copy [web-root]/INSTALL.txt from assets/scaffold/files/drupal.INSTALL.txt
  - Copy [web-root]/README.txt from assets/scaffold/files/drupal.README.txt
  - Copy [web-root]/robots.txt from assets/scaffold/files/robots.txt
  - Copy [web-root]/update.php from assets/scaffold/files/update.php

In Filesystem.php line 288:

  Could not delete /dataglusterfs_mount/project.crx.cloud/html/sites/default/default.services.yml:


root@crx-webng03 mods-enabled # a2dismod php7.4
Module php7.4 disabled.
To activate the new configuration, you need to run:
  systemctl restart apache2


root@crx-webng03 mods-enabled # a2enmod php8.0
Considering dependency mpm_prefork for php8.0:
Considering conflict mpm_event for mpm_prefork:
Considering conflict mpm_worker for mpm_prefork:
Module mpm_prefork already enabled
Considering conflict php5 for php8.0:
Module php8.0 already enabled


+ PHP CLI : 
# update-alternatives --display php
# update-alternatives --set php /usr/bin/php[VERSION]


Mise à jour DRUPAL de 9.1.0 à 9.2.21 :

Rien de particulier si ce n’est que j’installe en + drupal/upgrade_status qui va me permettre de récupérer un status de la mise à jour sous forme de rapport :

composer require drupal/upgrade_status

Création d'un report : 
root@crx-webng03 html # composer require drupal/upgrade_status --with-all-dependencies



Modification :  ^9.1  : 

cat composer.json
{
    "name": "drupal/drupal",
    "description": "Drupal is an open source content management platform powering millions of websites and applications.",
    "type": "project",
    "license": "GPL-2.0-or-later",
    "require": {
        "composer/installers": "^1.0.24",
        "drupal/core": "^9.1",
        "drupal/codesnippet": "^1.6",
        "drupal/colorbox": "^1.4",
        "drupal/bootstrap": "^3.15",
        "drupal/bootstrap_library": "^1.11",
        "egulias/email-validator": "^2.0",
        "drupal/core-recommended": "^9.1",
        "drupal/core-composer-scaffold": "^9.1",
        "drupal/core-project-message": "^9.1",
        "drupal/remove_meta_info": "^1.0",
        "drupal/core-vendor-hardening": "^9",
        "drupal/upgrade_status": "^4.1"

composer update

composer require drupal/core-recommended:9.2.21 drupal/core-composer-scaffold:9.2.21 drupal/core-project-message:9.2.21 --update-with-all-dependencies

Je lance ensuite cette commande très pratique qui va m’afficher ce qui reste à mettre à jour :

root@crx-webng03 html # composer outdated "drupal/*"
Do not run Composer as root/super user! See https://getcomposer.org/root for details
Continue as root/super user [yes]? yes
Legend:
! patch or minor release available - update recommended
~ major release available - update possible

Direct dependencies required in composer.json:
drupal/bootstrap              3.24.0 ! 3.26.0 Built to use Bootstrap, a sleek, intuitive, and powerful front-end framework for faster and easier web development.
drupal/bootstrap_library      1.15.0 ~ 2.0.2  Provides Bootstrap Integration.
drupal/colorbox               1.10.0 ~ 2.0.1  A light-weight, customizable lightbox plugin for jQuery.
drupal/core                   9.2.21 ! 9.5.11 Drupal is an open source content management platform powering millions of websites and applications.
drupal/core-composer-scaffold 9.2.21 ~ 10.2.3 A flexible Composer project scaffold builder.
drupal/core-dev               9.1.0  ~ 10.2.3 require-dev dependencies from drupal/drupal; use in addition to drupal/core-recommended to run tests from drupal/core.
drupal/core-project-message   9.2.21 ~ 10.2.3 Adds a message after Composer installation.
drupal/core-recommended       9.2.21 ~ 10.2.3 Locked core dependencies; require this project INSTEAD OF drupal/core.
drupal/core-vendor-hardening  9.5.11 ~ 10.2.3 Hardens the vendor directory for when it's in the docroot.

Transitive dependencies not required in composer.json:
Everything up to date


Mise à jour DRUPAL de 9.2.1 à 9.2.4 :

root@crx-webng03 html # composer require drupal/core-recommended:^9.2.4 drupal/core-composer-scaffold:^9.2.4 --update-with-dependencies
Do not run Composer as root/super user! See https://getcomposer.org/root for details
Continue as root/super user [yes]? yes
./composer.json has been updated
Running composer update drupal/core-recommended drupal/core-composer-scaffold --with-dependencies
Loading composer repositories with package information
Dependency drupal/core is also a root requirement. Package has not been listed as an update argument, so keeping locked at old version. Use --with-all-dependencies (-W) to include root dependencies.
Dependency egulias/email-validator is also a root requirement. Package has not been listed as an update argument, so keeping locked at old version. Use --with-all-dependencies (-W) to include root dependencies.
Updating dependencies
Lock file operations: 0 installs, 1 update, 0 removals
  - Upgrading drupal/core-composer-scaffold (9.2.21 => 9.5.11)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 0 installs, 1 update, 0 removals
  - Upgrading drupal/core-composer-scaffold (9.2.21 => 9.5.11): Extracting archive
Package doctrine/reflection is abandoned, you should avoid using it. Use roave/better-reflection instead.
Package laminas/laminas-zendframework-bridge is abandoned, you should avoid using it. No replacement was suggested.
Package symfony/debug is abandoned, you should avoid using it. Use symfony/error-handler instead.
Package behat/mink-goutte-driver is abandoned, you should avoid using it. Use behat/mink-browserkit-driver instead.
Package fabpot/goutte is abandoned, you should avoid using it. Use symfony/browser-kit instead.
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
Hardening vendor directory with .htaccess and web.config files.
> Drupal\Core\Composer\Composer::ensureHtaccess
87 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
Cleaning installed packages.
Found 25 security vulnerability advisories affecting 6 packages.
Run "composer audit" for a full list of advisories.
root@crx-webng03 html #

 

Mise à jour DRUPAL de 9.2.4 à 9.5.0 (en passant par la 9.4.15) :

 ^9.5
 root@crx-webng03 html # 
 composer require drupal/core-recommended:9.5.0 drupal/core-composer-scaffold:9.5.0 drupal/core-project-message:9.5.0 --update-with-all-dependencies

  Problem 1
    - drupal/codesnippet is locked to version 1.9.0 and an update of this package was not requested.
    - drupal/codesnippet 1.9.0 requires drupal/ckeditor * -> found drupal/ckeditor[dev-1.0.x, 1.0.0-rc1, ..., 1.0.x-dev (alias of dev-1.0.x)] but these were not loaded, likely because it conflicts with another require.

FAILED composer require drupal/core-recommended:9.5.0 drupal/core-composer-scaffold:9.5.0 drupal/core-project-message:9.5.0 -W

=> Désinstallation de drupal/codesnippet. 

=> Relance : 

root@crx-webng03 html # composer require drupal/core-recommended:9.5.0 drupal/core-composer-scaffold:9.5.0 drupal/core-project-message:9.5.0 --update-with-all-dependencies

#note : j'ai activé ckeditor5 (pour ne plus utiliser la V4 dans DRUPAL). 

=> Au final je dois rollback en 9.2.4, je décide alors de passer par la 9.4.15 (intermédiaire ici avance la 9.5) :

root@crx-webng03 html # cat composer.json
{
    "name": "drupal/drupal",
    "description": "Drupal is an open source content management platform powering millions of websites and applications.",
    "type": "project",
    "license": "GPL-2.0-or-later",
    "require": {
        "composer/installers": "^1.0.24",
        "drupal/core": "9.4.15",
        "drupal/colorbox": "^1.4",
        "drupal/bootstrap": "^3.15",
        "drupal/bootstrap_library": "^1.11",
        "egulias/email-validator": "^3.2",
        "drupal/core-recommended": "9.4.15",
        "drupal/core-composer-scaffold": "9.4.15",
        "drupal/core-project-message": "9.4.15",
        "drupal/remove_meta_info": "^1.0",
        "drupal/core-vendor-hardening": "^9",
        "drupal/upgrade_status": "^4.1",
        "drupal/highlightjs_input_filter": "^1.1",
   
...
 1280  composer require drupal/core-recommended:9.4.15 drupal/core-composer-scaffold:9.4.15 drupal/core-project-message:9.4.15 --update-with-all-dependencies

root@crx-webng03 html # composer require 'drupal/addtoany:^2.0'


Bootstrap (Thème)	8.x-3.24	8.x-3.30 (Notes de version)
Requires Drupal core: 9.5.0 to 10.2.4

...


Mise à jour DRUPAL 9.4.15 à 9.5.0 : 

root@crx-webng03 html # cat composer.json
{
    "name": "drupal/drupal",
    "description": "Drupal is an open source content management platform powering millions of websites and applications.",
    "type": "project",
    "license": "GPL-2.0-or-later",
    "require": {
        "composer/installers": "^1.0.24",
        "drupal/core": "9.5.0",
        "drupal/colorbox": "^2.0",
        "drupal/bootstrap": "^3.30",
        "drupal/bootstrap_library": "^2.0",
        "egulias/email-validator": "^3.2",
        "drupal/core-recommended": "9.5.0",
        "drupal/core-composer-scaffold": "9.5.0",
        "drupal/core-project-message": "9.5.0",
        "drupal/remove_meta_info": "^1.0",
        "drupal/core-vendor-hardening": "^9",
        "drupal/upgrade_status": "^4.1",
        "drupal/highlightjs_input_filter": "^1.1",
        "drupal/addtoany": "^2.0"
    },

 1317  composer require drupal/core-recommended:9.5.0 drupal/core-composer-scaffold:9.5.0 drupal/core-project-message:9.5.0 --update-with-all-dependencies



=> composer require 'drupal/bootstrap:^3.30'

Scaffolding files for drupal/core:
  - Copy [project-root]/.gitattributes from assets/scaffold/files/gitattributes
  - Copy [web-root]/.htaccess from assets/scaffold/files/htaccess
  - Copy [web-root]/robots.txt from assets/scaffold/files/robots.txt
  - Copy [web-root]/web.config from assets/scaffold/files/web.config
  - Copy [web-root]/sites/example.settings.local.php from assets/scaffold/files/example.settings.local.php


...
87 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
Scaffolding files for drupal/core:

In Filesystem.php line 288:

  Could not delete /dataglusterfs_mount/project.crx.cloud/html/sites/default/default.services.yml:


Mise à jour DRUPAL 9.5.0 à 10.2.3 puis 10.2.4 : 

Surement l’étape la plus difficile ici car le changement de version majeure implique l’obsolescence de pas mal de lib/thèmes côté DRUPAL :

root@crx-glust01 html # cat composer.json
{
    "name": "drupal/drupal",
    "description": "Drupal is an open source content management platform powering millions of websites and applications.",
    "type": "project",
    "license": "GPL-2.0-or-later",
    "require": {
        "composer/installers": "^1.0.24",
        "drupal/core": "10.2.4",
        "drupal/colorbox": "^2.0",
        "drupal/bootstrap": "^3.30",
        "drupal/bootstrap_library": "^2.0",
      "egulias/email-validator": "~4.0.2",
        "drupal/core-recommended": "10.2.4",
        "drupal/core-composer-scaffold": "10.2.4",
        "drupal/core-project-message": "10.2.4",
        "drupal/remove_meta_info": "^1.0",
        "drupal/core-vendor-hardening": "^9",
        "drupal/upgrade_status": "^4.1",
        "drupal/highlightjs_input_filter": "^1.1",
        "drupal/addtoany": "^2.0",
        "symfony/console": "^6.4",
        "guzzlehttp/guzzle": "^7.5",
        "drupal/classy": "*",
        "drupal/bartik": "^1.0",
        "drupal/seven": "^1.0",
        "drupal/bartik-bartik": "^1.0",
        "drupal/rdf": "^2.1",
        "drupal/quickedit": "^1.0",
        "drupal/color": "^1.0",
        "drupal/ckeditor": "*"
    },

...

composer require 'egulias/email-validator:^3.2.1' --update-with-dependencies --no-update
=> egulias/email-validator ~3.2.1

composer require 'symfony/console:^6.4' --update-with-dependencies --no-update
composer require 'egulias/email-validator:^4.0.2' --update-with-dependencies --no-update
 
=> egulias/email-validator[4.0.2]
=> drupal/addtoany 2.0.5
=> symfony/console ^6.4

composer remove 'drupal/core-dev:9.1.0'
composer require 'guzzlehttp/guzzle:^7.8.1' --update-with-dependencies --no-update
=> guzzlehttp/guzzle ~7.8.1

root@crx-webng03 html # composer prohibits drupal/core 10.2.3
root@crx-webng03 html # composer require drupal/core-recommended:10.2.3 drupal/core-composer-scaffold:10.2.3 drupal/core-project-message:10.2.3 --update-with-all-dependencies

composer require symfony/console:6.4 --no-update 

root@crx-webng03 html # composer require drupal/core-recommended:10.2.3 drupal/core-composer-scaffold:10.2.3 drupal/core-project-message:10.2.3 --update-with-all-dependencies
Do not run Composer as root/super user! See https://getcomposer.org/root for details
Continue as root/super user [yes]? yes
./composer.json has been updated
Running composer update drupal/core-recommended drupal/core-composer-scaffold drupal/core-project-message --with-all-dependencies
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires symfony/console ^6, found symfony/console[v6.0.0-BETA1, ..., 6.4.x-dev] but these were not loaded, likely because it conflicts with another require.
  Problem 2
    - Root composer.json requires drupal/core 10.2.3 -> satisfiable by drupal/core[10.2.3].
    - drupal/core 10.2.3 requires symfony/console ^6.4 -> found symfony/console[v6.4.0-BETA1, ..., 6.4.x-dev] but these were not loaded, likely because it conflicts with another require.
  Problem 3
    - Root composer.json requires drupal/core-recommended 10.2.3 -> satisfiable by drupal/core-recommended[10.2.3].
    - drupal/core-recommended 10.2.3 requires guzzlehttp/guzzle ~7.8.1 -> found guzzlehttp/guzzle[7.8.1, 7.8.x-dev] but these were not loaded, likely because it conflicts with another require.
  Problem 4
    - drupal/core 10.2.3 requires symfony/console ^6.4 -> found symfony/console[v6.4.0-BETA1, ..., 6.4.x-dev] but these were not loaded, likely because it conflicts with another require.
    - drupal/addtoany 2.0.5 requires drupal/core ^9.4 || ^10.0 -> satisfiable by drupal/core[10.2.3].
    - drupal/addtoany is locked to version 2.0.5 and an update of this package was not requested.

root@crx-webng03 html # composer require symfony/console:6.4.4 --no-update
Do not run Composer as root/super user! See https://getcomposer.org/root for details
Continue as root/super user [yes]? yes
./composer.json has been updated

composer prohibits drupal/core 10.2.3

composer require symfony/*:6.4 --no-update

--------

root@crx-webng03 html # composer prohibits drupal/core 10.2.3
Do not run Composer as root/super user! See https://getcomposer.org/root for details
Continue as root/super user [yes]? yes
drupal/drupal           -      requires         drupal/core (9.5.0)
drupal/core-recommended 9.5.0  requires         drupal/core (9.5.0)
drupal/core             10.2.3 requires         symfony/console (^6.4)
drupal/drupal           -      requires         symfony/console (4.4.49)
drupal/core             10.2.3 requires         symfony/dependency-injection (^6.4)
drupal/drupal           -      does not require symfony/dependency-injection (but v4.4.49 is installed)
drupal/core             10.2.3 requires         symfony/event-dispatcher (^6.4)
drupal/drupal           -      does not require symfony/event-dispatcher (but v4.4.44 is installed)
drupal/core             10.2.3 requires         symfony/filesystem (^6.4)
drupal/drupal           -      does not require symfony/filesystem (but v4.4.42 is installed)
drupal/core             10.2.3 requires         symfony/finder (^6.4)
drupal/drupal           -      does not require symfony/finder (but v4.4.44 is installed)
drupal/core             10.2.3 requires         symfony/http-foundation (^6.4)
drupal/drupal           -      does not require symfony/http-foundation (but v4.4.49 is installed)
drupal/core             10.2.3 requires         symfony/http-kernel (^6.4)
drupal/drupal           -      does not require symfony/http-kernel (but v4.4.51 is installed)
drupal/core             10.2.3 requires         symfony/mime (^6.4)
drupal/drupal           -      does not require symfony/mime (but v5.4.13 is installed)
drupal/core             10.2.3 requires         symfony/routing (^6.4)
drupal/drupal           -      does not require symfony/routing (but v4.4.44 is installed)
drupal/core             10.2.3 requires         symfony/serializer (^6.4)
drupal/drupal           -      does not require symfony/serializer (but v4.4.47 is installed)
drupal/core             10.2.3 requires         symfony/validator (^6.4)
drupal/drupal           -      does not require symfony/validator (but v4.4.48 is installed)
drupal/core             10.2.3 requires         symfony/process (^6.4)
drupal/drupal           -      does not require symfony/process (but v4.4.44 is installed)
drupal/core             10.2.3 requires         symfony/yaml (^6.4)
drupal/drupal           -      does not require symfony/yaml (but v4.4.45 is installed)
drupal/core             10.2.3 requires         twig/twig (^3.5.0)
drupal/drupal           -      does not require twig/twig (but v2.15.6 is installed)
drupal/core             10.2.3 requires         doctrine/annotations (^1.14)
drupal/drupal           -      does not require doctrine/annotations (but 1.13.3 is installed)
drupal/core             10.2.3 requires         guzzlehttp/guzzle (^7.5)
drupal/drupal           -      requires         guzzlehttp/guzzle (6.5.8)
drupal/core             10.2.3 requires         guzzlehttp/psr7 (^2.4.5)
drupal/drupal           -      does not require guzzlehttp/psr7 (but 1.9.1 is installed)
drupal/core             10.2.3 requires         asm89/stack-cors (^2.1)
drupal/drupal           -      does not require asm89/stack-cors (but 1.3.0 is installed)
drupal/core             10.2.3 requires         psr/log (^3.0)
drupal/drupal           -      does not require psr/log (but 1.1.4 is installed)
Not finding what you were looking for? Try calling `composer require "drupal/core:10.2.3" --dry-run` to get another view on the problem.
root@crx-webng03 html #


?? asm89/stack-cors
core_version_requirement: ^8 || ^9

Notes :

=> (à moi même)   C’est mal de lancer COMPOSER en root (   ajout d’un shell temporaire sur le www-data c’est mieux )       🙂

=> j’ai du désinstaller des thèmes/modules qui n’étaient plus valables en v10, pour le reste, j’ai du ajouter ce tag (sur les fichiers info.yml) :
‘  core_version_requirement: ^8 || ^9  ‘

=> j’ai aussi du réinstaller certaines >LIB en V10 pour ensuite les désactivées depuis l’interface d’administration :  ( messages remontés via update.php ) :

 1432  composer require drupal/core-recommended:10.2.3 drupal/core-composer-scaffold:10.2.3 drupal/core-project-message:10.2.3 --update-with-all-dependencies --no-update
 1461  composer require drupal/classy
 1463  composer require 'drupal/bartik:^1.0'
 1464  composer require 'drupal/seven:^1.0'
 1470  composer require 'drupal/bartik-bartik:^1.0'
 1472  composer require 'drupal/rdf:^2.1'
 1473  composer require 'drupal/quickedit:^1.0'
 1474  composer require 'drupal/color:^1.0'
 1495  composer require drupal/ckeditor
 1504  composer require drupal/core-recommended:10.2.4 drupal/core-composer-scaffold:10.2.4 drupal/core-project-message:10.2.4 --update-with-all-dependencies

Bon courage à vous pour vos mises à jour DRUPAL !

Voir aussi : https://bastien.barbe.pw/mise-a-jour-drupal-8-5-5-vers-8-5-6/

Update GITLAB-CE

Hello to all,

Hosting a GITLAB-CE solution (DEBIAN12 OS), I had to perform an update 14 to 16.
This is relatively long because you have to update in stages.

Fortunately the official site offers a tool to generate all the versions to use,
to arrive at the target version.

After migrate to DEBIAN12 i lost all gitlab repo, and finnaly i’ve done this update manually
version after version. Here is the process :

I retreive my gitlab version  here 14.2.1 and target to 16.9.1 :

# dpkg -l | grep gitlab 

Then i go to this site, to build my upgrade path :   ( 10 steps !!! )
https://gitlab-com.gitlab.io/support/toolbox/upgrade-path/?current=14.2.1&edition=ce

Basically after updating Debian 11 to 12, we have the GIT package which is ‘orphaned’,
we can see it via this command:

apt-show-versions | grep -i "no available version"

I go to this link to retreive all of packages needed :

https://packages.gitlab.com/app/gitlab/gitlab-ce/search?dist=debian&filter=all&page=1&q=

let’s go :

apt-get install gitlab-ce=14.3.6-ce.0
--------------------------------------	
	=> root@crx-git01 sources # wget https://packages.gitlab.com/gitlab/gitlab-ce/packages/debian/buster/gitlab-ce_14.3.6-ce.0_amd64.deb/download.deb
		
		mv download.deb gitlab-ce_14.3.6-ce.0_amd64.deb
		dpkg -i gitlab-ce_14.3.6-ce.0_amd64.deb

		...
		Keeping all older configuration backups
		Unpacking gitlab-ce (14.3.6-ce.0) over (14.2.1-ce.0) ...
		...
		...
		1709459914_2024_03_03_14.2.1_gitlab_backup.tar
		...

	Upgrade complete! If your GitLab server is misbehaving try running
		sudo gitlab-ctl restart

		root@crx-git01 sources # dpkg -l | grep git
		ii  gitlab-ce                      14.3.6-ce.0                    amd64        GitLab Community Edition (including NGINX, Postgres, Redis)


apt-get install gitlab-ce=14.9.5-ce.0
--------------------------------------
   => https://packages.gitlab.com/gitlab/gitlab-ce/packages/debian/buster/gitlab-ce_14.9.5-ce.0_amd64.deb

	root@crx-git01 sources # wget https://packages.gitlab.com/gitlab/gitlab-ce/packages/debian/buster/gitlab-ce_14.9.5-ce.0_amd64.deb/download.deb
		
	root@crx-git01 sources # wget https://packages.gitlab.com/gitlab/gitlab-ce/packages/debian/buster/gitlab-ce_14.9.5-ce.0_amd64.deb/download.deb
	--2024-03-03 11:08:46--  https://packages.gitlab.com/gitlab/gitlab-ce/packages/debian/buster/gitlab-ce_14.9.5-ce.0_amd64.deb/download.deb
	Resolving packages.gitlab.com (packages.gitlab.com)... 104.18.39.11, 172.64.148.245, 2606:4700:4400::6812:270b, ...
	Connecting to packages.gitlab.com (packages.gitlab.com)|104.18.39.11|:443... connected.
	HTTP request sent, awaiting response... 302 Found
	Location: https://d20rj4el6vkp4c.cloudfront.net/7/8/debian/package_files/127567.deb?t=1709460826_348df00b8ce7a24a8a77d0cd338f45a39ee4bba5 [following]
	--2024-03-03 11:08:46--  https://d20rj4el6vkp4c.cloudfront.net/7/8/debian/package_files/127567.deb?t=1709460826_348df00b8ce7a24a8a77d0cd338f45a39ee4bba5
	Resolving d20rj4el6vkp4c.cloudfront.net (d20rj4el6vkp4c.cloudfront.net)... 52.222.153.188, 52.222.153.182, 52.222.153.31, ...
	Connecting to d20rj4el6vkp4c.cloudfront.net (d20rj4el6vkp4c.cloudfront.net)|52.222.153.188|:443... connected.
	HTTP request sent, awaiting response... 200 OK
	Length: 1034232620 (986M) [application/x-debian-package]
	Saving to: 'download.deb'

	download.deb                                      100%[==========================================================================================================>] 986.32M  10.5MB/s    in 90s

	2024-03-03 11:10:17 (10.9 MB/s) - 'download.deb' saved [1034232620/1034232620]

	root@crx-git01 sources #  ll
	total 1010000
	-rw-r--r-- 1 root root 1034232620 Jun  1  2022 download.deb
	root@crx-git01 sources # dpkg -i download.deb
		
	root@crx-git01 sources # dpkg -l |grep git
	ii  gitlab-ce                      14.9.5-ce.0                    amd64        GitLab Community Edition (including NGINX, Postgres, Redis)

apt-get install gitlab-ce=14.10.5-ce.0
--------------------------------------
Running configuration backup
Creating configuration backup archive: gitlab_config_1709462867_2024_03_03.tar
/etc/gitlab/
/etc/gitlab/trusted-certs/
/etc/gitlab/gitlab-secrets.json
/etc/gitlab/gitlab.rb
Configuration backup archive complete: /etc/gitlab/config_backup/gitlab_config_1709462867_2024_03_03.tar
Keeping all older configuration backups
Unpacking gitlab-ce (14.10.5-ce.0) over (14.9.5-ce.0) ...


root@crx-git01 sources # dpkg -l |grep git

iU  gitlab-ce                      14.10.5-ce.0                   amd64        GitLab Community Edition (including NGINX, Postgres, Redis)

apt-get install gitlab-ce=15.0.5-ce.0
--------------------------------------
...
root@crx-git01 sources # dpkg -l |grep git
ii  gitlab-ce                      15.0.5-ce.0                    amd64        GitLab Community Edition (including NGINX, Postgres, Redis)

apt-get install gitlab-ce=15.4.6-ce.0
--------------------------------------
...
root@crx-git01 sources # dpkg -l |grep git
ii  gitlab-ce                      15.4.6-ce.0                    amd64        GitLab Community Edition (including NGINX, Postgres, Redis)
...

apt-get install gitlab-ce=15.11.13-ce.0
--------------------------------------
root@crx-git01 sources # dpkg -l |grep git
iU  gitlab-ce                      15.11.13-ce.0                  amd64        GitLab Community Edition (including NGINX, Postgres, Redis)
...

apt-get install gitlab-ce=16.1.6-ce.0
--------------------------------------
root@crx-git01 sources # dpkg -l |grep git
iU  gitlab-ce                      16.1.6-ce.0                    amd64        GitLab Community Edition (including NGINX, Postgres, Redis)

...

apt-get install gitlab-ce=16.3.7-ce.0
--------------------------------------
root@crx-git01 sources # dpkg -l |grep git
ii  gitlab-ce                      16.3.7-ce.0                    amd64        GitLab Community Edition (including NGINX, Postgres, Redis)
...

apt-get install gitlab-ce=16.7.6-ce.0
--------------------------------------
...
root@crx-git01 sources # dpkg -l |grep git
ii  gitlab-ce                      16.7.6-ce.0                    amd64        GitLab Community Edition (including NGINX, Postgres, Redis)


apt-get install gitlab-ce=16.9.1-ce.0
--------------------------------------
...

 

Have fun with your gitlab updates !

USB Control and triggerhappy on LINUX for HAMRADIO

In the past, i was using a « multimedia controller knob USB »  (on Windows OS with AUTOHOTKEY script to map SYSCALL of sound to special key),
This system can control my RADIO TRANSCEIVER (FT897).

For exemple i can use one RASPBERRYPI for TRX/TCP-IP control with CRX-COM and another RASPBERRYPI from my remote for USB BUTTON control,
here i’m using the same RASPBERRY device for TRX CAT, CRX-COM and USB Control.

I’ve found a new system based on LINUX now to do that, AHK equivalent system is call « TRIGGERHAPPY » on LINUX.
I’m using a RASPBERRY PI4 + an USB « Cooidea USB Multimedia Control« .

In my case on RPI4, TRIGGERHAPPY is already install / natif.

The advantage of this solution is :
– i did’t have to use my PC/tablet here, only  RASPBERRYPI4 + USB CAT SYSTEM + USB MEDIA CONTROLLER.
– Can work via INTERNET/NETWORK.

For HAM radio, use « TRIGGERHAPPY » is very usefull, you can map key/button to call a specific command,
exemples : control relay by press a key/device button, change frequency, turn antenna (key press/relay map).

Here is a photo of the test system RASPBERRYPI4+USB MULTIMEDIA Button, on the bottom right :

1. First step identify KEY mapping with USB device : 

Simply connect a device (keyboard/media controller) to your raspberry.
For exemple here i connect the USB device, i launch a « dmesg » command to have information :

[ 6143.609009] usb 1-1.4: new full-speed USB device number 6 using xhci_hcd
[ 6146.029045] usb 1-1.4: new full-speed USB device number 7 using xhci_hcd
[ 6146.166273] usb 1-1.4: New USB device found, idVendor=0483, idProduct=5756, bcdDevice= 2.00
[ 6146.166290] usb 1-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 6146.166303] usb 1-1.4: Product: USB Multimedia Control
[ 6146.166314] usb 1-1.4: Manufacturer: Cooidea
[ 6146.166326] usb 1-1.4: SerialNumber: 205435345950
[ 6146.175278] input: Cooidea USB Multimedia Control Keyboard as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4:1.0/0003:0483:5756.0003/input/input3
[ 6146.240139] input: Cooidea USB Multimedia Control Consumer Control as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4:1.0/0003:0483:5756.0003/input/input4
[ 6146.240454] hid-generic 0003:0483:5756.0003: input,hidraw1: USB HID v1.11 Keyboard [Cooidea USB Multimedia Control] on usb-0000:01:00.0-1.4/input0

Now let make a small test, we have a command to debug the « TRIGGERHAPPY » process it’s the THD command,
So after plug the USB device, i simply turn up/down the volume button and press on it to call « mute » option :

root@rmtrpi-56-f4eyq bastien # thd --dump /dev/input/event*
EV_KEY  KEY_VOLUMEUP    1       /dev/input/event2
# KEY_VOLUMEUP  1       command
EV_KEY  KEY_VOLUMEUP    0       /dev/input/event2
# KEY_VOLUMEUP  0       command
EV_KEY  KEY_VOLUMEDOWN  1       /dev/input/event2
# KEY_VOLUMEDOWN        1       command
EV_KEY  KEY_VOLUMEDOWN  0       /dev/input/event2
# KEY_VOLUMEDOWN        0       command

root@rmtrpi-56-f4eyq bastien # thd --dump /dev/input/event*
EV_KEY  KEY_MUTE        1       /dev/input/event2
# KEY_MUTE      1       command
EV_KEY  KEY_MUTE        0       /dev/input/event2
# KEY_MUTE      0       command

2. Create the TRIGGERHAPPY configuration : 

I can create the button configuration for EVENTS mapping :

root@rmtrpi-56-f4eyq crx-com # cat /etc/triggerhappy/triggers.d/volumectrl.conf
#change band:
KEY_MUTE            1       /opt/crx/crx-com/qrg_whell.php mute
#frequency up:
KEY_VOLUMEUP        1       /opt/crx/crx-com/qrg_whell.php up

#frequency down:
KEY_VOLUMEDOWN      1       /opt/crx/crx-com/qrg_whell.php down

I’ve create a PHP script to map KEYPAM to WEBSOCKET command here is the script :

I can call it with 3 commands for the moment, UP/DOWN and BAND UP (in future i will probably move to an USB system with more buttons, like STEP/MODE changing).

root@rmtrpi-56-f4eyq crx-com # /opt/crx/crx-com/qrg_whell.php mute
@DEBUG:n: 15m , 21000
@DEBUG, SET payload:0210000001
root@rmtrpi-56-f4eyq crx-com # /opt/crx/crx-com/qrg_whell.php up
@DEBUG, SET payload:0210010001
root@rmtrpi-56-f4eyq crx-com # /opt/crx/crx-com/qrg_whell.php down
C:21001
N:21000
@DEBUG, SET payload:0210000001
root@rmtrpi-56-f4eyq crx-com # /opt/crx/crx-com/qrg_whell.php up
@DEBUG, SET payload:0210010001
root@rmtrpi-56-f4eyq crx-com #

To create the script, i’ve first made this test (send a WEBSOCKET cmd via CLI with WEBSOCAT),
here CRX-COM program talk with JSON :

<?php

$cmd='echo \'{"protocol":"yaesu1","setQrg":"0071200001","readbuff":0}\' | /usr/local/bin/websocat -t --ws-c-uri=wss://192.168.1.167:5209/ - ws-c:cmd:\'socat - ssl:192.168.1.167:5209,verify=0\'';
$e=shell_exec($cmd);

?>

After succefully send QRG / change BAND,
i’ve create this script :  /opt/crx/crx-com/qrg_whell.php

#!/usr/bin/php
<?php
/*
 *	crx-com/qrg_whell.php	
 *	A CLI client for crx-com WEBSOCKET (remote station/crx-radio-cloud).
 *	(c) f4eyq@crx.cloud - 2024
 *	date: 27/02/2024
 *	up:   27/02/2024
 *	version: 0.1b
 * 
 *	Events mapping call this TRIGGERHAPPY script with args{mute,up,down} : 
 *	KEY_MUTE	1	/opt/crx/crx-com/qrg_whell.php mute	
 *	KEY_VOLUMEUP	1	/opt/crx/crx-com/qrg_whell.php up	
 *	KEY_VOLUMEDOWN	1	/opt/crx/crx-com/qrg_whell.php down
 *	See /etc/triggerhappy/triggers.d/volumectrl.conf	
 * */
require_once('/opt/crx/crx-com/qrg_whell.qrg.php');
$GTRX_conf			=	array(
	'crx-com-hostname'	=>	'192.168.1.167',
	'crx-com-port'		=>	'5209',
	'crx-com-protocol'	=>	'yaesu1'	
);
$GREG_file_crxcomqrgwhell	=	'/tmp/REG_file_crxcomqrgwhell';
$GDEF_QRG			=	'7000';
$GDEF_BAND			=	'40m';
$inputAction			=	'init';
if(isset($argv[1])){
	if($argv[1]=='up'){$inputAction='qrgup';}
	if($argv[1]=='down'){$inputAction='qrgdown';}
	if($argv[1]=='mute'){$inputAction='bandup';}
}
#echo "@DEBUG:".$inputAction."\n";
initRegistry();
$reg=getRegistry();
if($inputAction=='qrgup' && isset($reg['current_qrg'])){
	#echo "C:".$reg['current_qrg']."\n";
	$reg['current_qrg'] = $reg['current_qrg'] + 1;
	#echo "N:".$reg['current_qrg']."\n";
	saveRegistry($reg);
	setQrgTrx($reg);
	exit(0);
}
if($inputAction=='qrgdown' && isset($reg['current_qrg'])){
	#echo "C:".$reg['current_qrg']."\n";
	$reg['current_qrg'] = $reg['current_qrg'] - 1;
	#echo "N:".$reg['current_qrg']."\n";
	saveRegistry($reg);
	setQrgTrx($reg);
	exit(0);
}
if($inputAction=='bandup' && isset($reg['current_band'])){
	$current_band	=	$reg['current_band'];
	$fd=0;
	$next=$nextb=$nextq='';
	foreach($frequency_default as $vbd=>$vqrg){
                if($fd){//put the next band:
			$nextb=$vbd;
			$nextq=$vqrg;
                 break;
                }
		if($vbd==$current_band && $fd==0){
			$fd++;		
		}			
	}
	if($nextb=='' && $nextq==''){
		foreach($frequency_default as $vbd=>$vqrg){//put the first.
                        $nextb=$vbd;
			$nextq=$vqrg;
			break;
		}
	}
	#echo "@DEBUG:n: $nextb , $nextq  \n";
	$reg['current_band'] = $nextb;
        $reg['current_qrg']  = $nextq;
	saveRegistry($reg);
	setQrgTrx($reg);
}
function getQrgPayload($qrg,$protocol){
	//@TODO: implement all CRX-COM protocol
	//At this time only yaesu1 is supported (FT817,818,857,897)
	// input QRG exemple :  7006Khz
	$inm=$qrg/1000; //Convert KHZ to MHZ : 
	$qrgmhz=explode('.',$inm);//create array 0=> 7,  1=> 006 (floating part is part [1] of this array.
	// FA    P1  P1  P1  P1  P1  P1  P1  P1  => yaesu1 (ft817,897 etc... into crx-com).
	$part1=$qrgmhz[0];// 7
	$part1=str_pad($part1,3, "0", STR_PAD_LEFT);// 007
	if(!isset($qrgmhz[1]))$qrgmhz[1]='00';
	$part2=$qrgmhz[1];// 006
	$part2=str_pad($part2,5, "0", STR_PAD_RIGHT);// 006 0000 , => len 7
	$pl = ''.$part1.''.$part2.'01';
	return $pl;
}
function setQrgTrx($reg){
	global $GTRX_conf;
	if(!isset($reg['current_qrg']))return false;
	$qrg=$reg['current_qrg'];//QRG in KHZ
	$pl	=	getQrgPayload($qrg,$GTRX_conf['crx-com-protocol']);
	#echo "@DEBUG, SET payload:".$pl."\n";

    $cmd='echo \'{"protocol":"'.$GTRX_conf['crx-com-protocol'].'","setQrg":"'.$pl.'",';
	$cmd.='"readbuff":0}\' | /usr/local/bin/websocat -t --ws-c-uri=wss://'.$GTRX_conf['crx-com-hostname'].':'.$GTRX_conf['crx-com-port'];
	$cmd.='/ - ws-c:cmd:\'socat - ssl:'.$GTRX_conf['crx-com-hostname'].':'.$GTRX_conf['crx-com-port'].',verify=0\'';

	$e=shell_exec($cmd);
	return true;
}
function saveRegistry($ireg){
       global  $GREG_file_crxcomqrgwhell;
       $reg=serialize($ireg);
       file_put_contents($GREG_file_crxcomqrgwhell,$reg);
}
function initRegistry(){
	global	$GREG_file_crxcomqrgwhell;
	global	$GDEF_QRG;
	global	$GDEF_BAND;
	if(!file_exists($GREG_file_crxcomqrgwhell)){
		$r=touch($GREG_file_crxcomqrgwhell);
		if(!$r){
			die('cannot create :'.$GREG_file_crxcomqrgwhell.' ,please check right on file');			
		}
		else{
			echo "init reg file for TRX control : ".$GREG_file_crxcomqrgwhell." \n\n";
			$arr=array(
				'current_band'	=>	$GDEF_BAND,
				'current_qrg'	=>	$GDEF_QRG
			);
			$reg=serialize($arr);			
			file_put_contents($GREG_file_crxcomqrgwhell,$reg);
		}
	}
}
function getRegistry(){
	global  $GREG_file_crxcomqrgwhell;
	$rs=file_get_contents($GREG_file_crxcomqrgwhell);
	if($rs && $rs<>''){
		$reg=unserialize($rs);
		if(!$reg)return array();
		return $reg;
	}
	return array();
}
?>

Frequency configuration is a simple script :     /opt/crx/crx-com/qrg_whell.qrg.php

<?php
$frequency_default = array();
$frequency_default["136khz"]="135";
$frequency_default["500khz"]="493";
$frequency_default["160m"]="1855";
$frequency_default["80m"]="3672";
$frequency_default["60m"]="5353";
$frequency_default["40m"]="7090";
$frequency_default["30m"]="10120";
$frequency_default["20m"]="14057";
$frequency_default["17m"]="18068";
$frequency_default["15m"]="21250";
$frequency_default["12m"]="24890";
$frequency_default["10m"]="28090";
$frequency_default["6m"]="50315";
$frequency_default["4m"]="70000";
$frequency_default["2m"]="144300";
$frequency_default["70cm"]="430000";
?>

The setup of WEBSOCAT here is very simple here :

#wget : https://github.com/vi/websocat/releases/download/v1.2.0/websocat_nossl_arm-linux-static

root@rmtrpi-56-f4eyq websocat-master # whereis websocat
websocat: /usr/local/bin/websocat
root@rmtrpi-56-f4eyq websocat-master # ll /usr/local/bin/websocat
lrwxrwxrwx 1 root root 50 Feb 27 10:16 /usr/local/bin/websocat -> /opt/websocat-master/websocat_1.2.0_nossl_arm_mini

73 to all,

See :
CRX-COM/QRG-WHELL : https://git.crx.cloud/f4eyq/crx-com/-/tree/master/QRG-WHELL
TRIGGERHAPPY : https://github.com/wertarbyte/triggerhappy/
WEBSOCAT : https://github.com/vi/websocat
CRX-COM :
https://project.crx.cloud/Remote_ham_radio_station_setup_guide
https://project.crx.cloud/crx-com

VFO-USB :
https://bastien.barbe.pw/ham-un-bouton-vfo-usb-pour-son-catsystem/

USB DEVICE can be use, type « multimedia controller knob USB »
on ALIEXPRESS  /   AMAZON :

See also :

=> Test Windows10 +MCHF WINDOWS10PC +TRX-MCHF (FT817 protocol) : https://bastien.barbe.pw/webcontrol-radio-transceiver/
=> Test Mobile Phone / Mumble / m.crx.cloud / CRX-COM (01/10/2021) : https://bastien.barbe.pw/ham-remote-station-on-mobile-phone/
=> Test WIFI/RASPBERRY/FT817 (04/04/2020) : https://bastien.barbe.pw/cat-system-en-wifi/
=> Test FT817+CRX-COM (15/06/2019) :  https://project.crx.cloud/crx-com-rasb-ft817

Webcontrol radio transceiver !

Hello to all,

CRX-COM is now in version 1.0.4 release 5. I’ve migrate the server.js to node v16, serialport v10. Docker file is also update.

For info, the program can be run :
– with manual nodejs setup (v16) with the bin.js file.
– with LINUX/WINDOWS/MAC via the binary files (via pkg).
– with Docker.
I’ve test it here with Windows 10, MCHF/RS-918 and related transceiver (ft817 protocol)
and Google Chrome.     73 to all,

Threat Hunting with PHP and BASH.

After testing fail2ban, I decided to create my own tool (MyIPSPHP) because I needed a tool that did not directly use IP lists (sometimes paid) and above all I needed a tool specific to my WEB applications.

The functioning of the tool that I propose to you here is based on PHP and BASH as well as on the GEOIP library. It also allows alerts to be sent to my phone via « ntfy.sh ». I used CHATGPT to improve the filters (add new shells/exploits).

Finally I use this tool on a mail server, an apache service, an nginx service and an iptables/ulog firewall, the operation is sensitively the same, we parse the logs and we detect possible alerts (via counting of DROP ports for example for iptables over a day).

Here is the code for NGINX log (for reverse proxy setup here), the APACHE version is very similar, note that I mix PHP and BASH calls, this does not pose a particular problem even if a « full » php or « full » bash version would be even better :  (here the date filter scope is 60min)

If you want to test, don’t forget to set RUNMODE to 0 to enable the no-action mode (only output the ban action).

#!/usr/bin/php
<?php
#MyIPSPHP (c) f4eyq@crx.cloud - v1.0
#sec_check_nginxlogs.php
error_reporting(E_ALL);
#sec_check_nginxlogs.php
############################################################
define('DEVICE_FW_NAME','myrp01');
define('NTFY_URI','ntfy.sh/mycustomalert1245');
define('NGINX_LOG','/var/log/nginx/*.log');
define('WHITELIST_IPS',array('192.168','82.99.99.99'));
define('BLOCK_TIME_S',3888000); // 45 days(86400*45)
define('RUNMODE',1);#Running mode : 1 => Real (detect and bloc IP adresse), 0 => Dry run.
############################################################
require_once('/opt/security/libs/geo/geoip2.phar');
use GeoIp2\Database\Reader;
require_once('/opt/security/libs/libfw.php');
define('IP_LIST_BL','/opt/masscan_ip.txt');
define('DB_GEOLITE2_COUNTRY','/opt/security/libs/geo/GeoLite2-Country.mmdb');
#Prepare date filter :
#ex:  24/Mar/2023:18:16
$date_apache_d=date('j');
$date_apache_m=date('M');
$date_apache_Y=date('Y');
$date_apache_h=date('H');
$filder_current_h="$date_apache_d/$date_apache_m/$date_apache_Y:$date_apache_h";
echo "Run security Nginx log monitoring : ".$filder_current_h.".\n";
if(RUNMODE == 1){
 echo "Running real mode. \n";
}
$log_path       =       NGINX_LOG;
function detect_vuln($log_path, $filder_current_h, $scan_vuln, $message,$plugin='Default') {
    $dos=false;
    $cmd = "sudo cat $log_path | grep '$filder_current_h' | grep -E '$scan_vuln' | awk '{ print $1}' | sort | uniq -c | sort | ".genPaternGrepv()." | grep -v '/'";

    #Scan all archives :
    #$cmd = "sudo zcat $log_path*gz | grep -E '$scan_vuln' | awk '{ print $1}' | sort | uniq -c | sort | ".genPaternGrepv()." | grep -v '/'";

    if($plugin=='Dos'){
        $cmd="sudo cat $log_path | grep '$filder_current_h' | awk '{ print $1}' | sort | uniq -c | sort | ".genPaternGrepv()." | grep -v '/'";
        $dos=true;
    }

    if($plugin=='Googlebot'){
        $scan_vuln = 'Googlebot';
        $ggrange = '66.249';
        $cmd = "sudo cat $log_path | grep '$filder_current_h' | grep -E -v '".$ggrange."' | grep -E '$scan_vuln' | awk '{ print $1}' | sort | uniq -c | sort | ".genPaternGrepv()." | grep -v '/'";
        #Scan all archives :
        #$cmd = "sudo zcat $log_path*gz | grep -E -v '".$ggrange."' | grep -E '$scan_vuln' | awk '{ print $1}' | sort | uniq -c | sort | ".genPaternGrepv()." | grep -v '/'";
    }

    $now      = time();
    $ban_end  = $now + BLOCK_TIME_S;

    $check = `$cmd`;
    $statsbrut = explode(PHP_EOL, $check ?? '');
    foreach($statsbrut as $k=>$line){
        preg_match('!\d+\.*\d*!', $line, $matches);
        if(isset($matches[0])){
           $hits=(int)$matches[0];
           preg_match('!\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}!', $line, $matches);
           if(isset($matches[0]) && !checkIfIPIsLog2($matches[0])){

              $ip=$matches[0];

              $t=0;

              if($dos)
                $t=8000;

              if($hits>$t){
                $str=dspMessage($hits,$ip,$message);
                if(RUNMODE == 1){

                        echo "$str \n";

                        logm($str);

                        sendAlert($str);
                        blockIp($ip);

                        $log="$ip|$ban_end";
                        addIPinLog($log);

                }
                else{
                        echo "Dry mode : \n";
                        echo $str."\n";
                }
              }
           }
        }
    }
}

###########################################
##Exploit/Vuln
# w00tw00 / mstshash / nmaplowercheck ...
# Detect  rdp/nmap call.
###########################################
$scan_vuln = '/boaform/admin|mstshash|nmaplowercheck|w00tw00t|\.\.\/.\.\/|/bin/sh|Apache/2|apache/2|cgi-bin/|/HNAP1|/Tri%6Eity.txt%2ebak';
$message = 'Alert try2call exploit on '.DEVICE_FW_NAME;
detect_vuln($log_path, $filder_current_h, $scan_vuln, $message);

###########################################
#Detect Webshell / autodiscover
# like  vuln.php, shell.php, alfashell.php
###########################################
$scan_vuln = '/IOptimize.php|/vuln.php|/marijuana.php|/shell.php|/alfashell.php|/autodiscover|/eval-stdin.php|/alfacgiapi/perl.alfa|/ALFA_DATA|/alfashell.php|/alfa.php|/c99.php|/r57.php|/wso.php|/b374k.php|/phpshell.php|/tryag.php|/FilesMan.php|/zehir.php|/izocin.php|/milw0rm.php|/cgitelnet.php|/darkmail.php|/GFS-webshell.php|/ANTICHAT_SHELL.php|/AntichatShell.php|/antichat.php|/php-backdoor.php|/php-backdoor2.php|/php-backdoor3.php|/php-webshells.php|/php-remoteview.php|/FSOCK.php|/jackal.php|/wsoshell.php|/php-spy-shells.php|/c99ud.php|/c100.php|/c99shell.php|/r57shell.php|/kacak.php|/ToolsBrasil.php|/cyber_shell.php|/WebShell.php|/simple-backdoor.php|/php-backdoor4.php';
$message = 'Alert try2call webshell on '.DEVICE_FW_NAME;
detect_vuln($log_path, $filder_current_h, $scan_vuln, $message);

####################################
# Detect webadmin Call :
####################################
$scan_vuln='/sqlbuddy/login.php|/phpma/|/phpMyAdmin/|/pmamy2/|/PHPMYADMIN/|/mymanage/|/manage/|/manager/|/mysql/|/mysql-admin/|/mysqladmin/|/mysqlmanager/|/myadmin/|/mydb/|/phpmyadmin2/|/phpmyadmin3/|/phpmyadmin4/|/phpMyAdmin-3/|/phpMyAdmin-4/|/phpMyAdmin-5/|/phpMyAdmin-6/|/phpmyadmin-old/|/phpmyadmin2/|/phpmyadmin3/|/phpmyadmin4/|/phpmyadmin5/|/phpMyExplorer/|/pma/|/PMA2/|/PMA3/|/PMA4/|/PMA5/|/PMA6/|/PMA2005/|/PMA2006/|/PMA2007/|/PMA2008/|/PMA2009/|/PMA2010/|/PMA2011/|/PMA2012/|/PMA2013/|/PMA2014/|/PMA2015/|/PMA2016/|/PMA2017/|/PMA2018/|/PMA2019/|/PMA2020/|/PMA2021/|/PMA2022/|/pma-old/|/pma2005/|/pma2006/|/pma2007/|/pma2008/|/pma2009/|/pma2010/|/pma2011/|/pma2012/|/pma2013/|/pma2014/|/pma2015/|/pma2016/|/pma2017/|/pma2018/|/pma2019/|/pma2020/|/pma2021/|/pma2022/|/PMA-old/|/adminer/|/dbadmin/|/sqladmin/|/myadmin/|/sqlyog/|/emysql/|/websql/|/adminsql/|/sqlweb/|/mysqlgui/|/dbtool/|/dbweb/|/sqleditor/|/sqlmanager/|/sqlbuddy/|/datadmin/|/dbVisualizer/|/sqlworkbench/|/heidiSQL/|/navicat/|/toad/|/phpsqliteadmin/|/sqliteadmin/|/SQLiteManager/|/adminer.php';

$message = 'Alert try2call web-admin on '.DEVICE_FW_NAME;
detect_vuln($log_path, $filder_current_h, $scan_vuln, $message);

####################################
# Detect Configuration collector
####################################
$scan_vuln='/phpinfo.php|/.env|/.env.bak|config.php.bak|settings.php.bak|wp-config.php.bak|settings.py|.sqlite|database.php.bak|config.inc.php.bak|config.yml.bak';
$message = 'Alert try2collect conf on '.DEVICE_FW_NAME;
detect_vuln($log_path, $filder_current_h, $scan_vuln, $message);

###########################
#Detect SQL injection :
###########################
$scan_vuln='SELECT%20|INSERT%20|UPDATE%20|UNION%20|AND%20|DROP%20|DESCRIBE%20|DELETE%20|TRUNCATE%20|SELECT%20INTO%20|OR%20';
$message = 'Alert detect SQL injection on '.DEVICE_FW_NAME;
detect_vuln($log_path, $filder_current_h, $scan_vuln, $message);

###########################
#Detect DDOS :
###########################
$plugin = 'Dos';
$message = 'Alert DDOS on '.DEVICE_FW_NAME;
detect_vuln($log_path, $filder_current_h, $scan_vuln, $message,$plugin);

##########################
#Detect fake google bot
##########################
$plugin  = 'Googlebot';
$message = 'Alert fake GoogleBot on '.DEVICE_FW_NAME;
detect_vuln($log_path, $filder_current_h, $scan_vuln, $message,$plugin);

###########################
#Uban process :
###########################
uBanProcess();
sec_check_nginxlogs.php

Here is the Firewall version :  (here the date filter scope is 24 hours)

<?php
#MyIPSPHP (c) f4eyq@crx.cloud - v1.0
#sec_check_fw.php
define('BLOCK_TIME_S',5184000); // 60 days ( 86400 x 60 )
define('TRIGGER_threshold_portscan',20);

#Prepare date filter :
##ex: Mar 27 09:43:47
$date_apache_d=date('j');
$date_apache_d=str_pad($date_apache_d,3,' ',STR_PAD_LEFT);
$date_apache_m=date('M');
$date_apache_Y=date('Y');
$filder_current_h = $date_apache_m.$date_apache_d;

#replay / all archives logs :
#$cmd = "zcat /var/log/auth.log*gz | grep 'invalid' | grep -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' |cut -d ':' -f 5 | cut -d ' ' -f 4 | sort | ".genPaternGrepv()." | uniq -c";

$cmd="cat $alog | grep '".$filder_current_h."' | grep 'DROP' | cut -d '=' -f 5 |cut -d ' ' -f 1 | ".genPaternGrepv()." | sort | uniq -c";

$check = `$cmd`;

$now      = time();
$ban_end  = $now + BLOCK_TIME_S;
$statsbrut = explode(PHP_EOL, $check ?? '');

#ex:Mar 28 05:53:51 server2343 znet-zfw DROP  IN=eno1 OUT= MAC=d0:30:99:d7:6b:fc:a0:b4:31:cf:1b:41:08:00 SRC=22.22.22.22 DST=23.23.23.23 LEN=28 TOS=00 PREC=0x00 TTL=34 ID=34354 PROTO=UDP SPT=21345 DPT=1194 LEN=8 MARK=0
foreach($statsbrut as $k=>$line){
        preg_match('!\d+\.*\d*!', $line, $matches);
        if(isset($matches[0])){
           $hits=(int)$matches[0];
           preg_match('!\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}!', $line, $matches);
           if(isset($matches[0]) && !checkIfIPIsLog2($matches[0])){
               if($hits>TRIGGER_threshold_portscan){
                 $ip=$matches[0];
                 #echo "$ip, $hits \n";
                 $log="$ip|$ban_end";
                 #echo "Ban end:".date('d/m/y H:i',$ban_end).' - '.$ip." - $hits \n";
                 if(RUNMODE == 1){
                        #now check if port is different / count uniq item ( DPT=(X) )
                        $cmd="cat /var/log/ulog/syslogemu.log | grep '".$filder_current_h."' | grep '$ip' | cut -d '=' -f 14 | cut -d ' ' -f 1 | sort | uniq | wc -l";
                        $check = `$cmd`;
                        $check = (int) $check;
                        if($check>=TRIGGER_threshold_portscan)  {
                                $message = 'Alert portscanning on '.DEVICE_FW_NAME;
                                $str=dspMessage($hits,$ip,$message);
                                echo "Ban end:".date('d/m/y H:i',$ban_end).' - '.$ip." - $hits => $check \n";
                                logm($str);
                                sendAlert($str);
                                blockIp($ip);
                                addIPinLog($log);
                        }
                 }
                 else{
                        echo "Dry mode : \n";
                        echo $str."\n";
                 }

               }
           }
        }
}

And the small php lib use for various task like ban/uban and new ip in blacklist db etc :

<?php
#MyIPSPHP (c) f4eyq@crx.cloud - v1.0
#libs/libfw.php
use GeoIp2\Database\Reader;
function logm($m){
   $m2=date('d/m/Y H:i:s') . ": \n" . $m;
   $e = `echo "$m2" >> /var/log/securitychecks.log`;
}
function genPaternGrepv(){
  $nb=count(WHITELIST_IPS);
  $str='';
  $i=1;
  foreach(WHITELIST_IPS as $k=>$ip){
     $str.="grep -v '$ip'";
     if($i<$nb){
       $str.=" | ";
     }
   $i++;
  }
  return $str;
}
function checkIfIPIsLog2($ip){
        $l=IP_LIST_BL;
        $test = trim(`cat $l |grep '$ip|';echo $1;`);
        if($test<>''){
          return true;
        }
        return false;
}
function addIPinLog($ip){
        $l=IP_LIST_BL;
        $test= `echo "$ip" >> $l`;
}
function remoteIPFromLog($ip,$ipl){
        $e=`grep -v '$ip' $ipl > temp && mv temp $ipl`;
}
function blockIp($ip){
        $a=`ip route add blackhole $ip`;
}
function uBanIp($ip){
        $a=`ip route del blackhole $ip`;
}
function sendAlert($str){
       $uri=NTFY_URI;
       $a=`curl -d "$str" $uri`;
}
function dspMessage($hits,$ip,$message){
        $str=$message."\nBlock Ip: $hits => $ip,";
        $str.="\nCountry : ".getIpInfo($ip,'cn').".\n";
        return $str;
}
function getIpInfo($ip,$t='cn'){
        $reader = new Reader(DB_GEOLITE2_COUNTRY);
        $record = $reader->country($ip);
        if($t=='cn')return $record->country->name;
        if($t=='iso')return $record->country->isoCode;
}
function uBanProcess(){
$now      = time();
#Uban process :
$ipl=IP_LIST_BL;
$cmd="cat $ipl";
$check=`$cmd`;
$statsbrut = explode(PHP_EOL, $check ?? '');
#Format :   141.98.10.172|1681208329
#              IP        |    UBAN time (end ban)
foreach($statsbrut as $k=>$line){
   $p=explode('|',$line);
   if(isset($p[0]))$p[0]=trim($p[0]);
   if(isset($p[0]) && $p[0]<>''){
        $ip=$p[0];
        $bt=(int)$p[1];
        if($bt<=$now){
          #delete ip route
          uBanIp($ip);
          #echo "Uban $ip\n";
          remoteIPFromLog($ip,$ipl);
        }
   }
}
}

?>
libs/libfw.php

Finaly i’m using a CRON to launch the script every 1min :

*/1 * * * * /opt/security/sec_check_nginxlogs.php >> /dev/null 2>&1

Configuration WSL/VISUALSTUDIO CODE.

Bonjour à tous,

Si votre éditeur vous renvoi un message indiquant que votre instance WSL n’est pas en V2, CTRL+MAJ+u  puis :
wsl_term

Lancez un powershell en tant qu’administrateur puis :

PS C:\Users\tux> wsl --list --verbose
NAME STATE VERSION
* Debian Stopped 1
PS C:\Users\bba>

Lancez ensuite cette commande pour convertir l’image WSL ici « Debian » :

PS C:\Users\tux> wsl --set-version Debian 2

Enfin vous pouvez décider que toutes vos images soient en V2 via cette commande :

PS C:\WINDOWS\system32> wsl --set-default-version 2
Pour plus d’informations sur les différences de clés avec WSL 2, visitez https://aka.ms/wsl2
L’opération a réussi.

Et voilà !

wsl --list --verbose
  NAME      STATE           VERSION
* Debian    Running         2

Sources :
https://docs.microsoft.com/fr-fr/windows/wsl/troubleshooting
https://aka.ms/wsl2

ANSIBET project is launch !

Hello everybody,

I developed a new concept to control the Ansible layer, my project is called « ANSIBET », phylosophy is make my ansible system experience better

The objective of this one is to bring a fast piloting of Ansible by freeing itself from the modification of files and by making more « user friendly » the interface of Ansible.

For the moment it is a « beta » version, the idea being progressively to build an advanced Ansible management system.

https://bastien.barbe.pw/ansibet/

 

Goodbye !

Configuration du NAT pour une machine hyper-v avec Windows 10/11.

Bonjour à tous,

Dans cet article je vais vous parler de configuration NAT pour une VM sous HYPERV avec Windows 10.
L’idée de cette configuration est de permettre le surf Internet d’une machine virtuelle ici DEBIAN LINUX.

hyper_vm_nat_net

1 Contrôle de la navigation Internet de la couche « virtualisation » :

Cette étape bien que facultative vous permet de vous s’assurez que votre machine WSL sort sur Internet,
Notez que ici ma machine WSL est de type « 1 ».

PS C:\Windows\System32\WindowsPowerShell\v1.0> wsl --list --running -v
NAME STATE VERSION
* Debian Running 1

N’utilisant que peu WSL, je commence par changer le mot de passe ROOT :

#Reset root pasword 
#run WINOWS cli 
wsl -u root 
passwd

Un simple ping en root fait l’affaire ensuite :

bba@CT-6SYSFC2:~$ su -
Password:
root@CT-6SYSFC2:~# ping www.google.fr
PING www.google.fr(par21s20-in-x03.1e100.net (2a00:1450:4007:818::2003)) 56 data bytes
64 bytes from par21s20-in-x03.1e100.net (2a00:1450:4007:818::2003): icmp_seq=1 ttl=114 time=16.2 ms
64 bytes from par21s20-in-x03.1e100.net (2a00:1450:4007:818::2003): icmp_seq=2 ttl=114 time=17.8 ms
..

2. Création d’un « v-switch » : 

Je créé ici un switch virtuel que j’appel « NATLINUX » :

New-VMSwitch –SwitchName "NATLINUX" –SwitchType Internal –Verbose

#pour enlever celui ci 
#Remove-VMSwitch -Name "NATLINUX"

Je récupère ensuite l’ID de l’interface de ce SW ici « 14 » :

PS C:\WINDOWS\system32> Get-NetAdapter

Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
----                      --------------------                    ------- ------       ----------             ---------
vEthernet (NATLINUX)      Hyper-V Virtual Ethernet Adapter #3          14 Up           00-15-5D-B0-01-08        10 Gbps

PS C:\WINDOWS\system32>

3. Création de la passerelle NAT et rattachement à la machine virtuelle : 

Je peux ensuite créer la passerelle de NAT via cette commande POWERSHELL :

New-NetIPAddress -IPAddress 172.16.229.10 -PrefixLength 24 -InterfaceIndex 14 -Verbose

Maintenant je « map » ma machine virtuelle LINUX pour utiliser cette passerelle :

Get-VM | Get-VMNetworkAdapter | Connect-VMNetworkAdapter –SwitchName "NATLINUX"

Je précise alors que ma machine virtuelle va sortir par l’adresse IP de cette passerelle NAT pour ceci je créé une règle de NAT :

New-NetNat -Name NATOutsideDebian -InternalIPInterfaceAddressPrefix 172.16.229.0/24

#Pour effacer cette règle : Remove-NetNat

Ensuite je peux lister la règle via cette commande :

PS C:\WINDOWS\system32> Get-NetNat


Name                             : NATNetwork
ExternalIPInterfaceAddressPrefix :
InternalIPInterfaceAddressPrefix : 192.168.1.33/32
IcmpQueryTimeout                 : 30
TcpEstablishedConnectionTimeout  : 1800
TcpTransientConnectionTimeout    : 120
TcpFilteringBehavior             : AddressDependentFiltering
UdpFilteringBehavior             : AddressDependentFiltering
UdpIdleSessionTimeout            : 120
UdpInboundRefresh                : False
Store                            : Local
Active                           : True

Et voilà  :

bba@debian:~$ ping www.google.fr
PING www.google.fr (216.58.214.67) 56(84) bytes of data.
64 bytes from fra15s10-in-f3.1e100.net (216.58.214.67): icmp_seq=1 ttl=119 time=16.5 ms
64 bytes from fra15s10-in-f67.1e100.net (216.58.214.67): icmp_seq=2 ttl=119 time=17.0 ms
64 bytes from fra15s10-in-f67.1e100.net (216.58.214.67): icmp_seq=3 ttl=119 time=16.0 ms
^C
--- www.google.fr ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 16.034/16.493/16.964/0.379 ms
bba@debian:~$

Il est possible maintenant d’utiliser différents outils comme un serveur X windows pour pouvoir utiliser la GUI de ma machine virtuelle depuis WINDOWS,
Ou encore y accéder et disposer d’une multitude d’outils.

A très bientôt !

 

 

 

Gestion des mises à jour LINUX suite, cibler les mises à jour de sécurité

Bonjour à tous,

Suite à mon petit article d’introduction à ANSIBLE et les mises à jour LINUX ici : automatisation-de-la-gestion-des-utilisateurs-et-des-mises-a-jour
Je vous propose ici de gérer une ou plusieurs mises à jour de sécurité en mode « automatique » toujours avec ANSIBLE.

Dans mon exemple ici je pars du principe que j’ai un ensemble de serveurs à mettre à jour ( en cas de mise à jour urgente à faire ),
Que j’ai un accès SSH sur ceux-ci et que je dispose de la couche ANSIBLE opérationnelle :

Je commence par identifier la mise à jour à faire via cette commande :

# apt-cache policy php8.0-memcache
php8.0-memcache:
  Installed: 8.0+4.0.5.2+3.0.9~20170802.e702b5f9+-1+0~20210302.23+debian10~1.gbpff8707
  Candidate: 8.0+4.0.5.2+3.0.9~20170802.e702b5f9+-7+0~20220117.28+debian10~1.gbp8ceec4
  Version table:
     8.0+4.0.5.2+3.0.9~20170802.e702b5f9+-7+0~20220117.28+debian10~1.gbp8ceec4 500
        500 https://packages.sury.org/php buster/main amd64 Packages
 *** 8.0+4.0.5.2+3.0.9~20170802.e702b5f9+-1+0~20210302.23+debian10~1.gbpff8707 100
        100 /var/lib/dpkg/status

Je déduis donc que le paquet cible à déployer est :

php8.0-memcache:8.0+4.0.5.2+3.0.9~20170802.e702b5f9+-7+0~20220117.28+debian10~1.gbp8ceec4

Dans cet exemple, il s’agit d’une simple mise à jour, bien sur cela peut être une mise à jour de sécurité ou encore un paquet à installer
comme nous l’avons vu précédemment.

Maintenant, je peux donc utiliser ce playbook, notez ici l’utilisation du signe  » =  » comme mentionné dans la documentation ANSIBLE,
Voici mon code YAML :    crx-awx/upgrade_package.yml

---

- hosts: all
  become: yes
  tasks:
    - name: Install the package
      apt:
        name: php8.0-memcache=8.0+4.0.5.2+3.0.9~20170802.e702b5f9+-7+0~20220117.28+debian10~1.gbp8ceec4
        update_cache: yes

Je peux donc le lancer via cette commande :

bba@crx-ans01 ~ $ ansible-playbook crx-awx/upgrade_package.yml --limit crx-webng04.crxcluster.lan

PLAY [all] **********************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************************************************************************************************************************************
ok: [crx-webng04.crxcluster.lan]

TASK [Install the package] ******************************************************************************************************************************************************************************************************************
[WARNING]: Updating cache and auto-installing missing dependency: python3-apt
changed: [crx-webng04.crxcluster.lan]

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
crx-webng04.crxcluster.lan : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

bba@crx-ans01 ~ $

Ensuite je contrôle que tout est bon :

# apt-cache policy php8.0-memcache
php8.0-memcache:
  Installed: 8.0+4.0.5.2+3.0.9~20170802.e702b5f9+-7+0~20220117.28+debian10~1.gbp8ceec4
  Candidate: 8.0+4.0.5.2+3.0.9~20170802.e702b5f9+-7+0~20220117.28+debian10~1.gbp8ceec4
  Version table:
 *** 8.0+4.0.5.2+3.0.9~20170802.e702b5f9+-7+0~20220117.28+debian10~1.gbp8ceec4 500
        500 https://packages.sury.org/php buster/main amd64 Packages
        100 /var/lib/dpkg/status

Ou je peux aussi le contrôler via cette commande ANSIBLE :

$ ansible crx-webng04.crxcluster.lan -a "apt-cache policy php8.0-memcache" | grep -- 'Installed\|Candidate'
  Installed: 8.0+4.0.5.2+3.0.9~20170802.e702b5f9+-7+0~20220117.28+debian10~1.gbp8ceec4
  Candidate: 8.0+4.0.5.2+3.0.9~20170802.e702b5f9+-7+0~20220117.28+debian10~1.gbp8ceec4
b

Et voilà j’ai pu mettre à jour simplement un paquet via un processus ANSIBLE.

 

A bientôt,

 

Automatisation de la gestion des utilisateurs et des mises à jour.

 

Ce petit article à pour but de présenter « une gestion des utilisateurs et des mises à jour système » ( ici sous LINUX/DEBIAN ),
avec un système centralisé appelé « ANSIBLE ».

ANSIBLE permet via le protocole SSH de gérer des configurations et d’automatiser des actions (comme des installations d’applications par exemple),
sur des serveurs « distants » ceci est aussi appelé « infrastructure as code ».

Prérequis :

ANSIBLE est installé sur ma machine locale et aussi sur mon serveur distant,
pour commencer j’ai créé le référentiel de mes machines  » /home/bba/hosts  » :

bba@crx-ans01 ~ $ cat hosts
[webserversdev]
crx-dev01.crxcluster.lan

Soit un serveur « crx-dev01.crxcluster.lan » qui se trouve dans un groupe appelé « webserversdev ».
On n’est pas obligé d’utiliser un FQDN, une IP suffit.

Voici ma configuration ANSIBLE :

bba@crx-ans01 ~ $ cat ansible.cfg
[defaults]
interpreter_python = auto_silent
inventory       =  /home/bba/hosts

bba@crx-ans01 ~ $ ansible-config dump --only-changed
DEFAULT_HOST_LIST(/home/bba/ansible.cfg) = [u'/home/bba/hosts']
INTERPRETER_PYTHON(/home/bba/ansible.cfg) = auto_silent

La commande « only-changed » m’assure que ANSIBLE a bien parsé mon fichier de configuration,
Et cela va m’afficher que les variables qui changent par rapport à la configuration globale ( /etc/ansible/ansible.cfg ).

Notez que :
-> interpreter_python = auto_silent   :   permet d’éviter les messages de ce type au lancement d’un playbook :

[WARNING]: Platform linux on host crx-dev01.crxcluster.lan is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.


1. Gestion des utilisateurs

L’objectif ici est de gérer mes utilisateurs avec ANSIBLE, les clés SSH sont dans ce dossier « ssh-keys ».
Je commence donc par créer ce dossier à la racine de mon projet :

ssh-keys-dep/

Ensuite j’ajoute la clé publique SSH pour chaque utilisateur que je vais déployer, ici mon utilisateur est « f4eyq » :

ssh-keys-dep/f4eyq.pub

Maintenant je vais créer un script ANSIBLE aussi appelé « playbook », pour utiliser ce dossier et créer / supprimer un utilisateur.
=> L’instruction « become: yes » va simplement exécuter les commandes en « root » :

bba@crx-ans01 ~ $ cat add-remote-user-ssh.yml
---
- name: Create new user
  hosts: all
  #  vars:
  #  - user_password: 'test'
  #  - user_login: 'f4eyq'

  gather_facts: no
  become: yes

  tasks:
    - name: Add a new user
      user:
          name: "{{ user_login }}"
          shell: /bin/bash

          password: "{{ user_password | password_hash('sha512') }}"
          groups: sudo

          #remove user :
          #state: absent
          #remove: yes

    - name: Deploy SSH Key
      authorized_key:
        user: "{{ user_login }}"
        key: "{{ lookup('file', 'ssh-keys-dep/{{ user_login }}.pub') }}"
        state: present
        exclusive: True

Je peux ensuite l’appeler pour créer mon utilisateur « f4eyq » avec mon mot de passe « test »,
cela va créer l’utilisateur le mettre dans le groupe que j’ai indiqué et cela va créer son home enfin déployer sa clé SSH :

bba@crx-ans01 ~ $ ansible-playbook add-remote-user-ssh.yml --limit webserversdev  --extra-vars "user_password=test user_login=f4eyq"

PLAY [Create new user] ****************************************************************************************************************************************************************************************

TASK [Add a new user] *****************************************************************************************************************************************************************************************
changed: [crx-dev01.crxcluster.lan]

TASK [Deploy SSH Key] *****************************************************************************************************************************************************************************************
changed: [crx-dev01.crxcluster.lan]

PLAY RECAP ****************************************************************************************************************************************************************************************************
crx-dev01.crxcluster.lan   : ok=2    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Si par le suite je dois mettre la clé à jour, j’édite simplement le fichier dans le dossier « ssh-keys-dep » et relance le playbook.
Enfin pour supprimer l’utilisateur je décommente ceci dans le playbook :

          #state: absent
          #remove: yes

Et je relance le tout.

Notez que :
-> le home de l’utilisateur est supprimé sur le serveur cible ainsi que toute existence sur le serveur ( passwd/ group ) donc faut bien contrôler vos commandes 🙂
-> pour changer le mot de passe de l’utilisateur rejouer simplement le playbook avec un nouveau mot de passe.

2. Gestion des mises à jour

Pour ce faire j’utilise le playbook suivant :

bba@crx-ans01 ~ $ cat debian_upgrade2.yml
---
 - name: Update servers OS
   hosts: all
   become: true
   tasks:
    - name: Update cache and packages
      register: updatesys
      apt:
        name: "*"
        state: latest
        update_cache: yes

    - name: Check status
      debug:
        msg: "{{updatesys.stdout_lines}}"
        #msg:  "{{updatesys.stdout_lines|last}}"

Ensuite pour lancer ma mise à jour uniquement sur mon serveur de dev qui dans le groupe « webserversdev »,
Je lance ceci :

bba@crx-ans01 ~ $ ansible-playbook debian_upgrade2.yml --limit webserversdev

PLAY [Update servers OS] **************************************************************************************************************************************************************************************

TASK [Gathering Facts] ****************************************************************************************************************************************************************************************
ok: [crx-dev01.crxcluster.lan]

TASK [Update cache and packages] ******************************************************************************************************************************************************************************
ok: [crx-dev01.crxcluster.lan]

TASK [Check status] *******************************************************************************************************************************************************************************************
ok: [crx-dev01.crxcluster.lan] => {
    "msg": "Building tag database..."
}

PLAY RECAP ****************************************************************************************************************************************************************************************************
crx-dev01.crxcluster.lan   : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Pour afficher le détail et pas juste la dernière ligne j’indique simplement :

msg: "{{updatesys.stdout_lines}}"
à la place de :
msg: "{{updatesys.stdout_lines|last}}"
bba@crx-ans01 ~ $ ansible-playbook debian_upgrade2.yml --limit webserversdev

PLAY [Update servers OS] **************************************************************************************************************************************************************************************

TASK [Gathering Facts] ****************************************************************************************************************************************************************************************
ok: [crx-dev01.crxcluster.lan]

TASK [Update cache and packages] ******************************************************************************************************************************************************************************
ok: [crx-dev01.crxcluster.lan]

TASK [Check status] *******************************************************************************************************************************************************************************************
ok: [crx-dev01.crxcluster.lan] => {
    "msg": [
        "Reading package lists...",
        "Building dependency tree...",
        "Reading state information...",
        "Reading extended state information...",
        "Initializing package states...",
        "Writing extended state information...",
        "Building tag database...",
        "No packages will be installed, upgraded, or removed.",
        "0 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.",
        "Need to get 0 B of archives. After unpacking 0 B will be used.",
        "Writing extended state information...",
        "Reading package lists...",
        "Building dependency tree...",
        "Reading state information...",
        "Reading extended state information...",
        "Initializing package states...",
        "Writing extended state information...",
        "Building tag database..."
    ]
}

PLAY RECAP ****************************************************************************************************************************************************************************************************
crx-dev01.crxcluster.lan   : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

A partir de là, il est possible via une CRON d’automatiser complétement les mises à jour de serveurs, si vous souhaitez un outil graphique,
ANSIBLE-AWX permet de gérer cela via une interface WEB.

Bonne automatisation à tous !