buildenv/README_en.md

<!-- LANGUAGE_LINKS_START -->
[🇩🇪 German](README_de.md) | <span style="color: grey;">🇬🇧 English</span> | [🇪🇸 Spanish](README_es.md) | [🇫🇷 French](README_fr.md) | [🇮🇹 Italian](README_it.md)
<!-- LANGUAGE_LINKS_END -->

This script serves as a tool to simplify the creation of a development environment and the build process for images running on different hardware platforms using Neutrino as the user interface. It automates some of the steps required to set up a consistent and functional development and build environment by pre-setting up the necessary dependencies and basic configurations, meta layers, and enabling custom settings. The script aims to provide a foundation upon which to build, experiment, and create, update, and maintain your own customized versions of Tuxbox-Neutrino images.

[![Version](https://img.shields.io/badge/version-0.5.10-blue.svg)](https://github.com/dbt1/tuxbox-explorer)

- [1. Preparation](#1-preparation)
	- [1.1 Install required host packages](#11-install-required-host-packages)
		- [1.1.1 Recommended additional packages for graphical support and analysis](#111-recommended-additional-packages-for-graphical-support-and-analysis)
	- [1.2 Prepare Git (if necessary)](#12-prepare-git-if-necessary)
	- [1.3 Clone init script](#13-clone-init-script)
	- [1.4 Run init script](#14-run-init-script)
	- [1.5 Structure of the build environment](#15-structure-of-the-build-environment)
- [2. Build image](#2-build-an-image)
	- [2.1 Select Box](#21-select-box)
	- [2.2 Start environment script](#22-start-environment-script)
	- [2.3 Create Image](#23-create-image)
- [3. Update](#3-update)
	- [3.1 Update Image](#31-update-image)
	- [3.2 Update package](#32-update-package)
	- [3.3 Update meta layer repositories](#33-update-meta-layer-repositories)
- [4. Custom adjustments](#4-customizations)
	- [4.1 Configuration](#41-configuration)
		- [4.1.1 Configuration files](#411-configuration-files)
		- [4.1.2 bblayers.conf](#412-bblayersconf)
		- [4.1.3 Reset configuration](#413-reset-configuration)
	- [4.3 Recipes](#43-recipes)
	- [4.4 Packages](#44-packages)
		- [4.4.1 Edit source code in the workspace (example)](#441-edit-source-code-in-the-workspace-example)
- [5. Force rebuild a single package](#5-force-rebuild-a-single-package)
- [6. Force complete image construction](#6-force-complete-image-construction)
- [7. License](#7-license)
- [8. Further information](#8-further-information)
## 1. Preparation

It is recommended at this point to use the Docker container provided for this purpose, as this already completes the essential steps so that you can get started with as few adjustments to your system as possible. [see docker-buildenv](https://github.com/tuxbox-neutrino/docker-buildenv). In this case you can start immediately [with the initialization](#14-run-init-script).

**NOTE:** [docker-buildenv](https://github.com/tuxbox-neutrino/docker-buildenv) completely replaces the [Tuxbox-Builder](https://sourceforge.net/projects/n4k/files/Tuxbox-Builder) VM. Their maintenance will no longer be continued.

Paths specified here are based on specifications generated by the init script. Some entries are represented as ```<Platzhalter>```, which need to be adjusted locally. [See diagram](#14-run-init-script)
### 1.1 Install required host packages

**Note:** If using other distributions see: [Yocto Project Quick Build](https://docs.yoctoproject.org/3.2.4/ref-manual/ref-system-requirements.html#supported-linux-distributions)

Debian 11

```bash
sudo apt-get install -y gawk wget git-core diffstat unzip texinfo gcc-multilib build-essential \
chrpath socat cpio python python3 python3-pip python3-pexpect xz-utils debianutils \
iputils-ping python3-git python3-jinja2 libegl1-mesa pylint3 xterm subversion locales-all \
libxml2-utils ninja-build default-jre clisp libcapstone4 libsdl2-dev doxygen
```

Debian 12

```bash
sudo apt-get install -y gawk wget git diffstat unzip texinfo gcc-multilib build-essential \
chrpath socat cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping \
python3-git python3-jinja2 libegl1-mesa pylint3 xterm subversion locales-all libxml2-utils \
ninja-build default-jre clisp libcapstone4 libsdl2-dev doxygen
```

**NOTE:** On Debian 10 (buster) use libcapstone3.
#### 1.1.1 Recommended additional packages for graphical support and analysis

```bash
sudo apt-get install -y gitk git-gui meld cppcheck clazy kdevelop
```
### 1.2 Prepare Git (if necessary)

The init script uses Git to clone the meta-layer repositories. If there is no Git configured yet, please create your global Git user data, otherwise this will be pointed out unnecessarily while the script is running.

```bash
git config --global user.email "you@example.com"
git config --global user.name "Dein Name"
```
### 1.3 Clone init script

```bash
git clone https://github.com/tuxbox-neutrino/buildenv.git && cd buildenv
```
### 1.4 Run init script

```bash
./init && cd poky-3.2.4
```
### 1.5 Structure of the build environment

After [Step 1.4](#14-run-init-script) this structure should have been created:

```
.buildenv
 ├── dist                          <-- Freigabeordner für http-Server (falls eingerichtet) http://localhost, http://localhost:8080 , benötigt für IPK-Feeds und Images
 │   └── {DISTRO_VERSION}          <-- hier liegen die erzeugten Images und Pakete (Symlinks zeigen auf die Deploy-Verzeichnisse innerhalb der Build-Unterverzeichnisse)
 :
 ├── init.sh                       <-- init-Script
 ├── local.conf.common.inc         <-- globale Benutzerkonfiguration, ist in die benutzerdefinierte Konfiguration inkluiert
 :
 ├── log                           <-- Ordner für Logs, enthält Logs für jede Ausführung des Init-Scripts
 :
 └── poky-{DISTRO_VERSION}         <-- Nach Schritt 1.4 befindest Du dich hier. Hier befindet sich der Buildsystem-Kern und die Meta-Layer
     │
     :
     └── build                     <-- Hier liegen die Build-Unterverzeichnisse, nach Schritt 2.2 befindest Du dich in einem dieser Build-Unterverzeichnisse
         ├── <machine x>           <-- Build-Unterverzeichnis für Maschinentyp x
         │   ├── conf              <-- Ordner für Layer und benutzerdefinierte Konfiguration
         │   │   └── bblayers.conf <-- Konfigurationsdatei für eingebundene Meta-Layer
         │   │   └── local.conf    <-- benutzerdefinierte Konfiguration für einen Maschinentyp
		 │   :
         │   ├── (tmp)             <-- Arbeitsverzeichnis, wird beim Bauen automatisch angelegt
         │   └── (workspace)       <-- Workspace, wird beim Ausführen von devtool angelegt
         :
         └── <machine y>           <-- weiteres Build-Unterverzeichnis für Maschinentyp y
```
## 2. Build an image

Make sure you are here as shown in [schema](#15-structure-of-the-build-environment):

```
poky-{DISTRO_VERSION}
```
### 2.1 Select box

View a list of available devices:

```bash
ls  build
```
### 2.2 Start environment script

Run the environment script once for the box you want from the list! You will then automatically be taken to the appropriate build subdirectory.

```bash
. ./oe-init-build-env build/<machine>
```

As long as you are now in the desired build subdirectory with the created environment within the open shell, you do not have to run this script again and can build [Step 2.3 ](#23-create-image) images or any packages.

**Note:** You can also create additional shells and thus build environments for other box types in parallel and, if necessary, switch to the appropriate terminal and have them built in parallel, if your system allows it.
### 2.3 Create image

```bash
bitbake neutrino-image
```

This may take a while. Some warning messages can be ignored. Error messages affecting the setscene tasks are not a problem, but errors during the build and package tasks terminate the process in most cases.  [In this case, please report the error or share your solution](https://forum.tuxbox-neutrino.org/forum/viewforum.php?f=77). Help is very welcome.

When everything is done, you should see a message similar to this:

```bash
"NOTE: Tasks Summary: Attempted 4568 tasks of which 4198 didn't need to be rerun and all succeeded."
```

<span style="color: green;">That's it...</span>

You can find results at:

```bash
buildenv/poky-{DISTRO_VERSION}/build/<machine>/tmp/deploy
```

or in the release directory:

```bash
buildenv/dist/<Image-Version>/<machine>/
```

If a web server is set up that points to the share directory:

```bash
http://localhost/{DISTRO_VERSION} oder mit Portnummer http://localhost:8080/{DISTRO_VERSION}
```
## 3. Update

Manual updates of the packages are not required. This is done automatically for every target called with Bitbake. This also applies to possible dependencies. If you want to have full control over certain package sources, you can store them in the designated workspace for each package, see [4.4 Packages](#44-packages).
If no updates are necessary, the builds are automatically skipped.
### 3.1 Update image

```bash
bitbake neutrino-image
```
### 3.2 Update package

```bash
bitbake <package>
```
### 3.3 Update meta layer repositories

Executing the init script with the ```--update``` parameter updates the included meta layers to the status of the remote repositories.

```bash
./init --update
```

If you have made changes to the meta layers, the init script update routines should temporarily stash uncommitted changes or rebase them on the local repository. Of course, you can manually update your local meta layers for meta neutrino and machine layer repositories. However, conflicts always have to be resolved manually.

**Note:** Configuration files remain essentially untouched, but possible variable names are migrated. New or changed settings will not be changed. Please check the configuration if necessary.
## 4. Customizations

### 4.1 Configuration

It is recommended to build for the first time without modified configuration files to get an idea of ​​how the build process works and to see the results as quickly as possible.
The setting options are very extensive and not really manageable for beginners. However, the Yocto Project in particular is very OpenEmbedded
comprehensively documented and provides the best source of information.
#### 4.1.1 Configuration files

> ~/buildenv/poky-3.2.4/build/```<machine>```/conf/local.conf

This file is located in the build directory of the respective machine type and is the actual custom configuration file originally intended for this by the build system. However, in this environment, this local.conf only contains a few lines and includes a global configuration. This file is valid **only** for the machine type it supports. Here you can therefore make additional entries that are only intended for the machine type. [See also diagram](#14-run-init-script)

> ~/buildenv/local.conf.common.inc

This file contains settings that apply to all machine types and is created from the template ```~/buildenv/local.conf.common.inc.sample``` when the init script is executed for the first time.

The ```./build/<machine>/conf/local.conf``` provided by the build system could be used as the primary configuration file for each machine type separately, as originally intended by the build system, but that would unnecessarily increase the maintenance effort. That's why ```~/buildenv/local.conf.common.inc``` is only included in ```./build/<machine>/conf/local.conf```,

**Note on** ```~/buildenv/local.conf.common.inc.sample```**:** This is just a template and should be left untouched to avoid possible conflicts when updating the build script repository and to see what may have changed.

After an update to the build script repository, new or changed options may have been added or removed that are not reflected in the included configuration file. This case should be taken into account in your own configuration and checked and adjusted if necessary.
#### 4.1.2 bblayers.conf

> ~/buildenv/poky-3.2.4/build/```<machine>```/conf/bblayers.conf

This file is usually customized when the init script is run for the first time and usually only needs to be customized if you want to add, remove or replace layers.
#### 4.1.3 Reset configuration

If you want to reset your machine configurations, please rename the conf directory (deletion is not recommended) and run the init script again.

```bash
~/mv ~/buildenv/poky-3.2.4/build/<machine>/conf ~/buildenv/poky-3.2.4/build/<machine>/conf.01
~/cd ~/buildenv
~/./init
```
### 4.3 Recipes

**Unless you are directly involved in the development of the Poky levels, do not change anything in the official Poky levels (meta layers)! This is strongly not recommended by the Yocto project, as you risk losing all your work when updating and creating incompatibilities or conflicts that can be difficult to maintain. The usual procedure to complete, extend or overwrite existing official recipes is to [use .bbappend](https://docs.yoctoproject.org/3.2.4/dev-manual/dev-manual-common-tasks.html#using-bbappend-files-in-your-layer) files.**

Alternatively, although not really recommended, you could copy copies of official recipes into your own meta layers and adapt them, as these are then usually preferred by the build system. In such a case, however, you are responsible for keeping these recipes up to date, which can unnecessarily increase the maintenance effort.

In principle, this also applies to recipes from your own meta layers such as meta-neutrino or the machine layers. But if you would like to [actively collaborate on the recipes](https://docs.yoctoproject.org/current/ref-manual/devtool-reference.html#modifying-an-existing-recipe), you are welcome to do so.
### 4.4 Packages

If you want to have full control over a package source code, for example to fix something or actively develop something, the source code you want to work on should be moved to the workspace. See: [Example of Neutrino](#441-edit-source-code-in-the-workspace-example)

See [devtool](https://docs.yoctoproject.org/current/ref-manual/devtool-reference.html) and especially [devtool modify](https://docs.yoctoproject.org/current/ref-manual/devtool-reference.html#modifying-an-existing-recipe). In the workspace you have the guarantee that the source code will not be touched by the build system. If you don't take this into account, it can happen, for example, that changed source code is repeatedly deleted or modified. Your own customizations can therefore be lost or become incompatible. In the local default configuration, [rm_work](https://docs.yoctoproject.org/ref-manual/classes.html#ref-classes-rm-work) is activated, which ensures that after each package has been built, the respective working directory is cleaned up, so that nothing remains except a few logs.
#### 4.4.1 Edit source code in the workspace (example)

Neutrino is used here as an example, but this approach essentially applies to all other packages.

```bash
~/buildenv/poky-3.2.4/build/hd61$ devtool modify neutrino
NOTE: Starting bitbake server...54cf81d24c147d888c"
...
workspace            = "3.2.4:13143ea85a1ab7703825c0673128c05845b96cb5"

Initialising tasks: 100% |###################################################################################################################################################################################################| Time: 0:00:01
Sstate summary: Wanted 0 Found 0 Missed 0 Current 10 (0% match, 100% complete)
NOTE: Executing Tasks
NOTE: Tasks Summary: Attempted 83 tasks of which 80 didn't need to be rerun and all succeeded.
INFO: Adding local source files to srctree...
INFO: Source tree extracted to /home/<user>/buildenv/poky-3.2.4/build/hd61/workspace/sources/neutrino
INFO: Recipe neutrino-mp now set up to build from /home/<user>/buildenv/poky-3.2.4/build/hd61/workspace/sources/neutrino
```

The source code for Neutrino is now located at ```/buildenv/poky-3.2.4/build/hd61/workspace/sources/neutrino```. You can then work on it there. This means that the build system no longer clones or automatically updates the Neutrino sources from the remote Git repo, but from now on only uses the local sources within the workspace, which you have to manage yourself. This is a Git repo created by devtool, into which you can integrate into the original remote repository, if this is not already the case.

If you do this now...

```bash
bitbake neutrino
```

...from now on Neutrino will only be built from the local repo in the workspace:

```bash
NOTE: Started PRServer with DBfile: /home/<user>/buildenv/poky-3.2.4/build/hd61/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 34211, PID: 56838
...
workspace            = "3.2.4:13143ea85a1ab7703825c0673128c05845b96cb5"

Initialising tasks: 100% |###################################################################################################################################################################################################| Time: 0:00:01
Sstate summary: Wanted 122 Found 116 Missed 6 Current 818 (95% match, 99% complete)
NOTE: Executing Tasks
NOTE: neutrino-mp: compiling from external source tree /home/<user>/buildenv/poky-3.2.4/build/hd61/workspace/sources/neutrino
NOTE: Tasks Summary: Attempted 2756 tasks of which 2741 didn't need to be rerun and all succeeded.
```

**Note!** In the special case of Neutrino, it is advisable to transfer not only its source code, but also the associated ```libstb-hal``` into the workspace.

```bash
devtool modify libstb-hal
```
## 5. Force rebuild a single package

In some cases it can happen that a target breaks off for whatever reason. But you shouldn't panic and delete the working folder and the expensive sstate cache. Cleanups can be carried out individually for each target without destroying an otherwise functioning system.

In particular, broken archive URLs can lead to termination. However, these errors are always displayed and you can check the URL. Often it's just the servers and they even work again after a few minutes.

To make sure whether the recipe in question actually has a problem, it makes sense to completely clean up the target in question and rebuild it. To achieve this, all associated package, build and cache data must be cleaned.

```bash
bitbake -c cleansstate <target>

```
then rebuild:

```bash
bitbake <target>
```
## 6. Force complete image construction

The init script provides the `--reset` option for this.

```bash
./init --reset
# Anweisungen befolgen

```

You can also achieve this manually by manually renaming the tmp directory in the respective build subdirectory. You can delete it later if you want to free up storage space or are sure that you no longer need the directory:

```bash
mv tmp tmp.01
```

Then have the image rebuilt:

```bash
bitbake neutrino-image
```

If you have **not** cleared the cache, the image should be built in a relatively short time. This is precisely why it is recommended to keep the cache. The directory where the cache is located is determined via the variable ```${SSTATE_DIR}``` and can be adjusted in the configuration.

This directory is quite valuable and only in rare cases is it necessary to delete this directory. Please note that the build process will take much longer after clearing the cache.
## 7. License

```
MIT License
```

## 8. Further information

Further information about the Yocto build system:

* https://docs.yoctoproject.org