buildenv/README_en.md

<!-- LANGUAGE_LINKS_START -->
[🇩🇪 German](README_de.md) | <span style="color: grey;">🇬🇧 English</span>
<!-- LANGUAGE_LINKS_END -->

This script serves as a tool to simplify setting up a development environment and the build process for images that run with Neutrino as the user interface on various hardware platforms. It automates several steps required to establish a consistent and functional development and build environment by preconfiguring necessary dependencies, basic configurations, and meta-layers, while allowing for custom settings. The script aims to provide a foundation on which you can build and experiment to create, update, and maintain your own customized versions of Tuxbox-Neutrino images.

* [1. Preparation](#1-preparation)

  * [1.1 Install Required Host Packages](#11-install-required-host-packages)

    * [1.1.1 Recommended Additional Packages for Graphical Support and Analysis](#111-recommended-additional-packages-for-graphical-support-and-analysis)
  * [1.2 Prepare Git (if necessary)](#12-prepare-git-if-necessary)
  * [1.3 Clone Init Script](#13-clone-init-script)
  * [1.4 Run Init Script](#14-run-init-script)
  * [1.5 Build Environment Structure](#15-build-environment-structure)
* [2. Building an Image](#2-building-an-image)

  * [2.1 Choose a Box](#21-choose-a-box)
  * [2.2 Start the Environment Script](#22-start-the-environment-script)
  * [2.3 Create the Image](#23-create-the-image)
* [3. Updates](#3-updates)

  * [3.1 Update Image](#31-update-image)
  * [3.2 Update Package](#32-update-package)
  * [3.3 Update Meta-Layer Repositories](#33-update-meta-layer-repositories)
* [4. Customizations](#4-customizations)

  * [4.1 Configuration](#41-configuration)

    * [4.1.1 Configuration Files](#411-configuration-files)
    * [4.1.2 bblayers.conf](#412-bblayersconf)
    * [4.1.3 Resetting Configuration](#413-resetting-configuration)
  * [4.3 Recipes](#43-recipes)
  * [4.4 Packages](#44-packages)

    * [4.4.1 Editing Source Code in the Workspace (Example)](#441-editing-source-code-in-the-workspace-example)
* [5. Forcing a Rebuild of a Single Package](#5-forcing-a-rebuild-of-a-single-package)
* [6. Forcing a Full Image Rebuild](#6-forcing-a-full-image-rebuild)
* [7. License](#7-license)
* [8. Further Information](#8-further-information)

## 1. Preparation

At this point, it is recommended to use the provided Docker container, as it already covers essential steps needed to get started with minimal changes to your system. [See docker-buildenv](https://github.com/tuxbox-neutrino/docker-buildenv). In this case, you can jump straight to [running the initialization](#14-run-init-script).

**NOTE:** [docker-buildenv](https://github.com/tuxbox-neutrino/docker-buildenv) fully replaces the [Tuxbox-Builder](https://sourceforge.net/projects/n4k/files/Tuxbox-Builder) VM, which is no longer maintained.

The paths shown here are based on those generated by the init script. Some entries appear as `<placeholder>` and need to be adjusted locally. [See schema](#14-run-init-script).

### 1.1 Install Required Host Packages

**Note:** If you are using other distributions, see: [Yocto Project Quick Build](https://docs.yoctoproject.org/3.2.4/ref-manual/ref-system-requirements.html#supported-linux-distributions)

**Debian 11**

```bash
sudo apt-get install -y gawk wget git-core diffstat unzip texinfo gcc-multilib build-essential \
  chrpath socat cpio python python3 python3-pip python3-pexpect xz-utils debianutils \
  iputils-ping python3-git python3-jinja2 libegl1-mesa pylint3 xterm subversion locales-all \
  libxml2-utils ninja-build default-jre clisp libcapstone4 libsdl2-dev doxygen
```

**Debian 12**

```bash
sudo apt-get install -y gawk wget git diffstat unzip texinfo gcc-multilib build-essential \
  chrpath socat cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping \
  python3-git python3-jinja2 libegl1-mesa pylint3 xterm subversion locales-all libxml2-utils \
  ninja-build default-jre clisp libcapstone4 libsdl2-dev doxygen
```

#### 1.1.1 Recommended Additional Packages for Graphical Support and Analysis

```bash
sudo apt-get install -y gitk git-gui meld cppcheck clazy kdevelop
```

### 1.2 Prepare Git (if necessary)

The init script uses Git to clone the meta-layer repositories. If you don’t have Git configured yet, please set up your global Git user data; otherwise, you’ll get unnecessary warnings while the script runs.

```bash
git config --global user.email "you@example.com"
git config --global user.name "Your Name"
```

### 1.3 Clone Init Script

```bash
git clone https://github.com/tuxbox-neutrino/buildenv.git && cd buildenv
```

### 1.4 Run Init Script

```bash
./init && cd poky-3.2.4
```

### 1.5 Build Environment Structure

After [step 1.4](#14-run-init-script), you should see a structure similar to this:

```
.buildenv
 ├── dist                          <-- Shared folder for HTTP server (if set up) http://localhost, http://localhost:8080, needed for IPK feeds and images
 │   └── {DISTRO_VERSION}          <-- Contains generated images and packages (symlinks to the deploy dirs inside build subdirectories)
 :
 ├── init.sh                       <-- init script
 ├── local.conf.common.inc         <-- global user configuration, included in custom config
 :
 ├── log                           <-- folder for logs, contains logs for each init script run
 :
 └── poky-{DISTRO_VERSION}         <-- After step 1.4 you’ll be here. Contains the build system core and meta-layers
     │
     :
     └── build                     <-- Build subdirectories; after step 2.2 you’ll be inside one of these
         ├── <machine x>           <-- Build dir for machine type x
         │   ├── conf              <-- Folder for layers and custom config
         │   │   └── bblayers.conf <-- Config for included meta-layers
         │   │   └── local.conf    <-- Custom config for one machine type
         │   :
         │   ├── (tmp)             <-- Work dir, auto-created during build
         │   └── (workspace)       <-- Workspace, created when running devtool
         :
         └── <machine y>           <-- Another build dir for machine type y
```

## 2. Building an Image

Make sure you are in the directory shown in the [schema](#15-build-environment-structure):

```
poky-{DISTRO_VERSION}
```

### 2.1 Choose a Box

List available devices:

```bash
ls build
```

### 2.2 Start the Environment Script

Run the environment script for the desired box from the list once! You will be placed into the corresponding build subdirectory.

```bash
. ./oe-init-build-env build/<machine>
```

From now on, as long as you remain in the generated environment within the open shell in the chosen build subdirectory, you don’t need to re-run this script and can go straight to [step 2.3](#23-create-the-image) to build images or any packages.

**Note:** You can also open additional shells and thus create parallel build environments for other box types, switching to the appropriate terminal as needed and even building in parallel, provided your system can handle it.

### 2.3 Create the Image

```bash
bitbake neutrino-image
```

This can take a while. Some warnings can be ignored. Errors related to setscene tasks are usually harmless, but build and package task errors typically abort the process. [Please report the error or share your solution](https://forum.tuxbox-neutrino.org/forum/viewforum.php?f=77). Help is always welcome.

When done, you should see a message like this:

```bash
NOTE: Tasks Summary: Attempted 4568 tasks of which 4198 didn't need to be rerun and all succeeded.
```

<span style="color: green;">That’s it...</span>

Results can be found under:

```bash
buildenv/poky-{DISTRO_VERSION}/build/<machine>/tmp/deploy
```

or in the shared directory:

```bash
buildenv/dist/<Image-Version>/<machine>/
```

If you have a web server pointing to the shared directory:

```bash
http://localhost/{DISTRO_VERSION} or with port number http://localhost:8080/{DISTRO_VERSION}
```

## 3. Updates

Manual package updates are not required. These are handled automatically on every bitbake invocation, including dependencies. If you want full control over certain package sources, you can place them in the workspace for each package; see [4.4 Packages](#44-packages).

If no updates are needed, builds are skipped automatically.

### 3.1 Update Image

```bash
bitbake neutrino-image
```

### 3.2 Update Package

```bash
bitbake <package>
```

### 3.3 Update Meta-Layer Repositories

Running the init script with the `--update` parameter updates the included meta-layers to the state of their remote repositories.

```bash
./init --update
```

If you have made changes to the meta-layers, the init script’s update routines should temporarily stash or rebase unstashed changes on the local repositories. Of course, you can manually update your local meta-layers for meta-neutrino and machine-layer repos, but you must resolve any conflicts manually.

**Note:** Configuration files remain mostly untouched, but variable names may be migrated. New or changed settings are not auto-updated. Please review your configuration if necessary.

## 4. Customizations

### 4.1 Configuration

It is recommended to build for the first time without modified configuration files to get a feel for the build process and see results quickly. The configuration options are extensive and not easy for beginners to grasp. However, OpenEmbedded and the Yocto Project are thoroughly documented and provide the best information source.

#### 4.1.1 Configuration Files

> \~/buildenv/poky-3.2.4/build/`<machine>`/conf/local.conf

This file, located in the build directory for each machine type, is the actual user configuration file intended by the build system. In this environment, it contains only a few lines and includes a global configuration. This file is **only** valid for its specific machine type. You can add entries here that apply only to that machine type. [See schema](#14-run-init-script)

> \~/buildenv/local.conf.common.inc

This file holds settings that apply to all machine types and is generated from the template `~/buildenv/local.conf.common.inc.sample` on the first run of the init script.

Although you could use the build system’s intended `./build/<machine>/conf/local.conf` as the primary config file for each machine separately, this would increase maintenance overhead. Hence, `~/buildenv/local.conf.common.inc` is only included in `./build/<machine>/conf/local.conf`.

**Note on** `~/buildenv/local.conf.common.inc.sample`**:** This is a template and should remain untouched to avoid conflicts when updating the build-script repository and to see what has changed.

After updating the build-script repository, new or changed options may have been added or removed and not merged into your included config file. You should consider these cases and review and adjust your configuration if necessary.

#### 4.1.2 bblayers.conf

> \~/buildenv/poky-3.2.4/build/`<machine>`/conf/bblayers.conf

This file is usually adjusted on the first run of the init script and typically only needs changes when you want to add, remove, or replace layers.

#### 4.1.3 Resetting Configuration

If you want to reset your machine configurations, rename (rather than delete) the conf directory and rerun the init script.

```bash
mv ~/buildenv/poky-3.2.4/build/<machine>/conf ~/buildenv/poky-3.2.4/build/<machine>/conf.backup
cd ~/buildenv
./init
```

### 4.3 Recipes

**Unless you are directly contributing to Poky layers, do not modify the official Poky layers (meta-layers)!** The Yocto Project strongly discourages this practice, as you risk losing your work during updates and creating hard-to-maintain conflicts.

The usual way to extend or override existing recipes is with `.bbappend` files. [Learn more](https://docs.yoctoproject.org/3.2.4/dev-manual/dev-manual-common-tasks.html#using-bbappend-files-in-your-layer).

Alternatively (though also not recommended), you could copy official recipes into your own meta-layers and modify them, in which case you are responsible for keeping them up to date, increasing maintenance overhead.

The same principle applies to your own meta-layers like meta-neutrino or machine layers. If you want to actively work on recipes, feel free to [contribute](https://docs.yoctoproject.org/current/ref-manual/devtool-reference.html#modifying-an-existing-recipe).

### 4.4 Packages

If you want full control over a package’s source code (e.g., to fix or actively develop something), move the source you want to work on into the workspace. See the Neutrino example below.

See [devtool](https://docs.yoctoproject.org/current/ref-manual/devtool-reference.html) and especially [devtool modify](https://docs.yoctoproject.org/current/ref-manual/devtool-reference.html#modifying-an-existing-recipe). In the workspace, you’re guaranteed that the build system won’t touch the source. If you ignore this, you might find your changes repeatedly cleaned or overridden, losing or rendering them incompatible. In the default local config, [rm\_work](https://docs.yoctoproject.org/ref-manual/classes.html#ref-classes-rm-work) is enabled, which cleans each package’s work directory after build, leaving only logs.

#### 4.4.1 Editing Source Code in the Workspace (Example)

This example uses Neutrino, but the procedure applies to most packages.

```bash
~/buildenv/poky-3.2.4/build/hd61$ devtool modify neutrino
NOTE: Starting bitbake server...54cf81d24c147d888c"
...
workspace            = "3.2.4:13143ea85a1ab7703825c0673128c05845b96cb5"

Initialising tasks: 100% |########################################################################################################| Time: 0:00:01
Sstate summary: Wanted 0 Found 0 Missed 0 Current 10 (0% match, 100% complete)
NOTE: Executing Tasks
NOTE: Tasks Summary: Attempted 83 tasks of which 80 didn't need to be rerun and all succeeded.
INFO: Adding local source files to srctree...
INFO: Source tree extracted to /home/<user>/buildenv/poky-3.2.4/build/hd61/workspace/sources/neutrino
INFO: Recipe neutrino-mp now set up to build from /home/<user>/buildenv/poky-3.2.4/build/hd61/workspace/sources/neutrino
```

The Neutrino source code is now in `/buildenv/poky-3.2.4/build/hd61/workspace/sources/neutrino`. You can work on it there. From now on, the build system will not clone Neutrino from the remote Git repo but will use only your local workspace sources, managed by you.

When you run:

```bash
bitbake neutrino
```

... Neutrino will build from the local workspace repo:

```bash
NOTE: Started PRServer with DBfile: /home/<user>/buildenv/poky-3.2.4/build/hd61/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 34211, PID: 56838
...
workspace            = "3.2.4:13143ea85a1ab7703825c0673128c05845b96cb5"

Initialising tasks: 100% |########################################################################################################| Time: 0:00:01
Sstate summary: Wanted 122 Found 116 Missed 6 Current 818 (95% match, 99% complete)
NOTE: Executing Tasks
NOTE: neutrino-mp: compiling from external source tree /home/<user>/buildenv/poky-3.2.4/build/hd61/workspace/sources/neutrino
NOTE: Tasks Summary: Attempted 2756 tasks of which 2741 didn't need to be rerun and all succeeded.
```

**Tip:** For Neutrino, it’s also recommended to transfer the associated `libstb-hal` to the workspace:

```bash
devtool modify libstb-hal
```

## 5. Forcing a Rebuild of a Single Package

Sometimes a target fails for unknown reasons. Avoid panicking and wiping your workspace or sstate cache. You can clean up a specific target without breaking everything else.

Broken archive URLs often cause failures. These errors are shown, and you can verify the URL. They often resolve after a few minutes.

To ensure a recipe actually has an issue, clean and rebuild the entire target’s data:

```bash
bitbake -c cleansstate <target>
```

Then rebuild:

```bash
bitbake <target>
```

## 6. Forcing a Full Image Rebuild

The init script provides the `--reset` option:

```bash
./init --reset
# Follow instructions
```

Manually, rename the tmp directory in the respective build subdirectory. You can delete it later to free space if you’re sure you don’t need it:

```bash
mv tmp tmp.backup
```

Then rebuild the image:

```bash
bitbake neutrino-image
```

If you haven’t deleted the cache, the image should finish relatively quickly. That’s why retaining the cache is recommended. The cache directory is defined by `${SSTATE_DIR}` and can be customized in the configuration.

This directory is valuable, and deleting it is rarely needed. Note that builds take much longer after deleting the cache.

## 7. License

```
MIT License
```

## 8. Further Information

More information on the Yocto build system:

* [https://docs.yoctoproject.org](https://docs.yoctoproject.org)